All projects matching “crawler”

AI crawler visibility audit: is your website visible to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) so ChatGPT, Claude and Perplexity can read and cite it? Fetched like an AI crawler, without JavaScript: robots.txt, llms.txt, sitemap, soft 404, JSON-LD, and which crawlers are blocked at the edge. $0.01 per check, one payment per check: every rule, a model verdict, a fix with evid

Read the main content of a page your own crawler cannot reach, as clean markdown. Send a URL and receive the main content of the page as clean markdown, stripped of navigation, advertising and boilerplate. The page is fetched through a French residential IP with a real Chromium browser, so JavaScript-rendered content is fully resol…

Apify Store is the largest marketplace of tools for AI agents. Extract data from any website and automate your workflows with thousands of scrapers, crawlers, and automation tools called Actors.

Extract and classify all hyperlinks from a web page — internal vs external using eTLD+1 domain matching. Returns up to 200 internal and 200 external links with counts. No web crawler setup needed. Accepts USDC payments on Base and Solana

Partner tier of the dns.pizza bot observatory: every network (ASN), named crawler, TLS (JA4) fingerprint and firewall rule seen in the last 7 days, refreshed hourly. The public tier is free at https://dns.pizza/bots.

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

AI and search visibility check for any live URL. Loads the page as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot, CCBot and a browser, and flags JSON, empty or blocked pages served to crawlers. Checks robots.txt per bot, sitemap, llms.txt, noindex, text visible without JavaScript, tags, and how many pages Google shows. Returns the top 3 fixes in plain words.

Agent-tier preview of the AI-visibility check: live pass/fail scan of the actual website — robots.txt AI-crawler blocks (GPTBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more), llms.txt presence, Schema.org structured data, sitemap presence — plus a deterministic grade. No fix plan, no prioritization, no evidence rows (those are in the $0.50 full check). Any domain, any country.

Use before spending tokens, money or crawler capacity fetching an unfamiliar web resource. Resolves redirects (manually, bounded, re-validated per hop) and returns deterministic HTTP/resource facts — final URL, status, content type and length, response time, a bounded HTML title/canonical peek, and a lightweight generic-bot robots.txt indication — so an agent can decide whether a normal web fetch

Detect whether a User-Agent belongs to a known bot, crawler, AI agent or HTTP tool, and if so return its name and category (search, social, seo, ai, monitoring, tool, …).

Website audit bundle for local business websites and lead qualification: performance (browser-measured), security headers and TLS, SEO basics, accessibility and AI-crawler visibility of one public page, with section scores, grades and prioritized findings, in one call.

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Web scraping API for AI agents: read any live web page as clean markdown, text or HTML. Renders JavaScript in our own headless browser, then strips navigation, ads and boilerplate. Use it when a plain HTTP GET returns empty or garbled HTML: single-page apps, dynamic feeds, or content that only appears after scripts run. Also returns the page's HTTP status. Page text is flagged untrusted so an agen

Real-time web search for AI agents. Search the web for current and up-to-date information, news, facts, research, documentation, and sources beyond model training data. Returns ranked structured JSON results with title, URL, description, and position for web grounding, RAG retrieval, research, and fact-finding. Send query, optional limit, and optional freshness as URL parameters. Keyless x402 pay-

Starts a website migration comparison: checks page existence, content and visual regression between a source site and its migrated copy, then produces a report with screenshots. Use it to verify a migration or redesign before go-live. Priced on the pages it compares (the urls list, or the pages discovery finds, at most the optional maxPages ceiling), no page limit: 0.02 USD per page for the first