All projects matching “sitemap”

Give this API a link to any website's sitemap.xml file and it will fetch and parse it, returning every URL found inside. If the sitemap links to other sitemaps (called sitemap indexes), it will follow and extract those too. Each URL is returned with any extra information available, such as when the page was last updated and how often it changes. Useful for SEO audits, content inventories, broken l

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the U

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

URL preflight - status, redirect chain, key headers, response time, robots.txt rules for the path and sitemap location

XML tier (deterministic, no LLM): verifies a claimed value lives at a named path in a public XML document, by independent double re-fetch. Use it for feeds and sitemaps; for JSON use /verify-json, for HTML use /verify. Paths address elements and @attributes (urlset.url[0].loc). String-exact; ambiguous or volatile values return unverifiable, never a guess. First 100 checks per client are free: rese

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

AI and search visibility check for any live URL. Loads the page as Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot, CCBot and a browser, and flags JSON, empty or blocked pages served to crawlers. Checks robots.txt per bot, sitemap, llms.txt, noindex, text visible without JavaScript, tags, and how many pages Google shows. Returns the top 3 fixes in plain words.

Website growth signals — Read one public website and return the observable growth signals: title and meta, CTAs, forms, chat and analytics tools, tech stack, structured data, blog freshness from the sitemap, robots. Facts only, never scores. $0.02 per call.

Agent-tier preview of the AI-visibility check: live pass/fail scan of the actual website — robots.txt AI-crawler blocks (GPTBot, ClaudeBot, PerplexityBot, Google-Extended + 10 more), llms.txt presence, Schema.org structured data, sitemap presence — plus a deterministic grade. No fix plan, no prioritization, no evidence rows (those are in the $0.50 full check). Any domain, any country.

AI crawler visibility audit: is your website visible to AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) so ChatGPT, Claude and Perplexity can read and cite it? Fetched like an AI crawler, without JavaScript: robots.txt, llms.txt, sitemap, soft 404, JSON-LD, and which crawlers are blocked at the edge. $0.01 per check, one payment per check: every rule, a model verdict, a fix with evid

Paid equivalent of /api/aeo/scan — scanner for AI search visibility (AEO, GEO) and agent readiness. Triple score + AI Identity Card + top issues covering JSON-LD Schema.org, meta tags, llms.txt, robots.txt, sitemap, and agent-discovery signals for ChatGPT, Perplexity, Google AI Overviews, Bing Copilot. $0.01 in USDC (Base or Solana) — Bazaar-indexable variant; the free version at /api/aeo/scan sta

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Inspect a public HTTP or HTTPS URL and return a structured site passport. Includes pulse fields, registrable domain, robots and sitemap discovery, feeds, OpenAPI hints, and page metadata. Uses at most eight external HTTP requests and does not execute JavaScript. Pass url as an absolute public HTTP(S) URL. Keep fresh=false to reuse cache; set fresh=true only when a new upstream fetch is required.

Starts a website migration comparison: checks page existence, content and visual regression between a source site and its migrated copy, then produces a report with screenshots. Use it to verify a migration or redesign before go-live. Priced on the pages it compares (the urls list, or the pages discovery finds, at most the optional maxPages ceiling), no page limit: 0.02 USD per page for the first

Paid audiobook narration economics advisor (cycle 207, x402 $0.001) - GET ?title=<BOOK_TITLE>&author=<AUTHOR>&minutes=<N> OR ?sample=1|2|3 OR POST application/json {title, author, minutes, hourly_llm_rate_usd, tts_usd_per_million_chars}. Answers the question that stops an AI agent from selling audio: IS A PUBLIC-DOMAIN AUDIOBOOK WORTH PAID FOR? Checks public-domain status (no account, no key: Gute