All projects matching “robots.txt”

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as give

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

URL preflight - status, redirect chain, key headers, response time, robots.txt rules for the path and sitemap location

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

AI and search visibility check for any live URL. Loads the page as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot, CCBot and a browser, and flags JSON, empty or blocked pages served to crawlers. Checks robots.txt per bot, sitemap, llms.txt, noindex, text visible without JavaScript, tags, and how many pages Google shows. Returns the top 3 fixes in plain words.

On-page SEO audit of one URL: title and meta description length, canonical, robots meta and X-Robots-Tag, robots.txt allow for user-agent *, sitemap.xml with URL count, lang, hreflang, h1-h3 outline, word count, images without alt, internal/external links, Open Graph, Twitter card, JSON-LD schema.org types, favicon, viewport, mixed content; each check pass/warn/fail with a fix and a 0-100 score.

Agent-tier preview of the AI-visibility check: live pass/fail scan of the actual website — robots.txt AI-crawler blocks (GPTBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more), llms.txt presence, Schema.org structured data, sitemap presence — plus a deterministic grade. No fix plan, no prioritization, no evidence rows (those are in the $0.50 full check). Any domain, any country.

Read a public static HTTPS webpage as clean text with headings, links, timestamp and content hash. Use to ground an answer or ingest documentation. Respects robots.txt; no JavaScript or login.

Use before spending tokens, money or crawler capacity fetching an unfamiliar web resource. Resolves redirects (manually, bounded, re-validated per hop) and returns deterministic HTTP/resource facts — final URL, status, content type and length, response time, a bounded HTML title/canonical peek, and a lightweight generic-bot robots.txt indication — so an agent can decide whether a normal web fetch

AI crawler visibility audit: is your website visible to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) so ChatGPT, Claude and Perplexity can read and cite it? Fetched like an AI crawler, without JavaScript: robots.txt, llms.txt, sitemap, soft 404, JSON-LD, and which crawlers are blocked at the edge. $0.01 per check, one payment per check: every rule, a model verdict, a fix with evid

Check a URL before fetching it: is it alive, final URL after redirects, HTTP status, content type, HTML title, and whether robots.txt allows your user-agent. Link checker and fetch preflight for agents.

Markdown only: fetch a public web page (robots.txt respected) and return clean Markdown plus title/word count. Cheaper than /v1/extract when you do not need structured product/table JSON.

Geo web fetch for AI agents. Fetch any public web page from a residential IP in a chosen country (190+ countries, optional city or US state) as clean markdown, text or HTML, with the product price/currency/availability extracted automatically when the page has product data. $0.005 per page, only charged when the page loads. Respects robots.txt; no CAPTCHA solving; sign-in, payment and email servic

Read a public web page as it is seen from up to 3 countries (residential and datacenter vantage points), returned as clean text with title and language, plus whether the countries see the same content. Respects robots.txt and Content-Signal. Not charged if the page cannot be read.

Check a public website's security hygiene before your agent links to it, recommends it, or sends data to it. One URL in, deterministic JSON out: HTTPS and redirect chain, HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, security.txt, robots.txt, sitemap.xml, plus plain-language notes. Private, localhost and raw-IP targets are rejected; error responses are no

PDFs, JavaScript pages and ordinary HTML as clean Markdown. Pays a page's x402 toll for you. Follows robots.txt.

Paid equivalent of /api/aeo/scan — scanner for AI search visibility (AEO, GEO) and agent readiness. Triple score + AI Identity Card + top issues covering JSON-LD Schema.org, meta tags, llms.txt, robots.txt, sitemap, and agent-discovery signals for ChatGPT, Perplexity, Google AI Overviews, Bing Copilot. $0.01 in USDC (Base or Solana) — Bazaar-indexable variant; the free version at /api/aeo/scan sta

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Use when you need to know what a website is built with (CMS, frameworks, analytics, CDN, hosting), who hosts its email and DNS, and how good its security headers are. Up to 3 domains per call; fetches each homepage once, follows robots.txt.

Fetch a public web page and convert it to clean Markdown (title, description, headings, absolute links). LLM-free, deterministic, honours robots.txt. ?url=

Web page to Markdown, $0.001 USDC per call: Fetch a public web page and return LLM-ready Markdown: title, meta description, canonical, language, main content (scripts/nav/footer stripped, headings/lists/links kept), first 50 absolute links, word count. Honours robots.txt (disallowed → 403, not charged). No JS rendering; non-HTML → 415. max_chars 500-50000 (default 20000). No params = runs the exam

Read any public web page as clean Markdown for LLMs: main content only (no nav, ads, scripts), absolute links, title/description/lang metadata. Respects robots.txt; only successful reads are charged.

Same as the free /v1/extract but without the 30 requests per day limit: fetch one public static HTML or JSON page and return structured data (text, attr, list, table, price, events, page, links, json) as JSON, CSV or ICS. Honours robots.txt. No JavaScript rendering. Failed calls (4xx/5xx) are not settled.

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the U

All links on a page, absolute and deduplicated, split into internal and external.