All projects matching “robots-txt”

Partner tier of the dns.pizza bot observatory: every network (ASN), named crawler, TLS (JA4) fingerprint and firewall rule seen in the last 7 days, refreshed hourly. The public tier is free at https://dns.pizza/bots.

Audits a website for AI agent and AI crawler readiness from caller-supplied inputs: robots.txt, HTML, headers, llms.txt, sitemap lastmod, /.well-known. Names the policy per AI token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Gemini-User, KagiBot, Kagi-User, Applebot-Extended, Bytespider, CCBot, Amazonbot, meta-ex

Real-time web search for AI agents. Search the web for current and up-to-date information, news, facts, research, documentation, and sources beyond model training data. Returns ranked structured JSON results with title, URL, description, and position for web grounding, RAG retrieval, research, and fact-finding. Send query, optional limit, and optional freshness as URL parameters. Keyless x402 pay-

Track a website's AI search visibility every week for 4 weeks. $19 USDC on Base per call. Runs the 16-engine report now, then each Monday until 4 successful runs. Returns a private tracking_url with a trend chart and competitor shifts, plus emails. Settles only if the first run succeeds (12+ of 16 engines). Don't use this for a single snapshot; use /api/v1/report.