All projects matching “ai-crawlers”

Agent-tier preview of the AI-visibility check: live pass/fail scan of the actual website — robots.txt AI-crawler blocks (GPTBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more), llms.txt presence, Schema.org structured data, sitemap presence — plus a deterministic grade. No fix plan, no prioritization, no evidence rows (those are in the $0.50 full check). Any domain, any country.

Website audit bundle for local business websites and lead qualification: performance (browser-measured), security headers and TLS, SEO basics, accessibility and AI-crawler visibility of one public page, with section scores, grades and prioritized findings, in one call.

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

Given a company website domain, returns a compact company card: the business name and the page element it came from, a generic role contact address (info@, sales@, support@ ...) only if the site publishes one, never a personal address, and the names of up to 10 technology platforms the site runs. Same scrape and evidence check as /scrape, without the evidence detail, at half the price. Free prompt