All projects matching “ai crawlers”

AI crawler visibility audit: is your website visible to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) so ChatGPT, Claude and Perplexity can read and cite it? Fetched like an AI crawler, without JavaScript: robots.txt, llms.txt, sitemap, soft 404, JSON-LD, and which crawlers are blocked at the edge. $0.01 per check, one payment per check: every rule, a model verdict, a fix with evid

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

Agent-tier preview of the AI-visibility check: live pass/fail scan of the actual website — robots.txt AI-crawler blocks (GPTBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more), llms.txt presence, Schema.org structured data, sitemap presence — plus a deterministic grade. No fix plan, no prioritization, no evidence rows (those are in the $0.50 full check). Any domain, any country.

Detect whether a User-Agent belongs to a known bot, crawler, AI agent or HTTP tool, and if so return its name and category (search, social, seo, ai, monitoring, tool, …).

Website audit bundle for local business websites and lead qualification: performance (browser-measured), security headers and TLS, SEO basics, accessibility and AI-crawler visibility of one public page, with section scores, grades and prioritized findings, in one call.

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Apify Store is the largest marketplace of tools for AI agents. Extract data from any website and automate your workflows with thousands of scrapers, crawlers, and automation tools called Actors.

AI and search visibility check for any live URL. Loads the page as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot, CCBot and a browser, and flags JSON, empty or blocked pages served to crawlers. Checks robots.txt per bot, sitemap, llms.txt, noindex, text visible without JavaScript, tags, and how many pages Google shows. Returns the top 3 fixes in plain words.

Web page to Markdown, $0.001 USDC per call: Fetch a public web page and return LLM-ready Markdown: title, meta description, canonical, language, main content (scripts/nav/footer stripped, headings/lists/links kept), first 50 absolute links, word count. Honours robots.txt (disallowed → 403, not charged). No JS rendering; non-HTML → 415. max_chars 500-50000 (default 20000). No params = runs the exam