CrawlWitness

Use when an agent is about to crawl, train on or ingest a website into RAG: checks the domain's AI consent and licensing signals (robots.txt per AI bot, RSL license, IETF AIPREF, TDMRep, noai meta tags, llms.txt, C2PA), returns a 0-100 score and a training stance, and a signed EAS receipt anchored on Base proving what was checked and when. Errors: 400 invalid domain; 502 site unreachable (not charged).

Use when an agent is about to crawl, train on or ingest a website into RAG: checks the domain's AI consent and licensing signals (robots.txt per AI bot, RSL license, IETF AIPREF, TDMRep, noai meta tags, llms.txt, C2PA), returns a 0-100 score and a training stance, and a signed EAS receipt anchored on Base proving what was checked and when. Errors: 400 invalid domain; 502 site unreachable (not charged).

AI consent and licensing check for a web domain before crawling, training or RAG ingestion: robots.txt rules per AI bot, RSL license, IETF AIPREF, TDMRep, noai meta tags, llms.txt and C2PA, combined into a 0-100 score and a training stance, with a signed EAS receipt anchored on Base that proves what was checked and when. Errors: 400 invalid domain; 502 site unreachable (not charged).

Use when a crawler or dataset builder needs AI-consent verdicts for many domains at once: yes/no per domain for one AI bot and use (train, rag, search), up to 100 domains per payment, with one signed receipt for the whole batch. Errors: 400 invalid input (not charged); 503 if too many domains are still being analysed (not charged, retry).

Use when a crawler, scraper or RAG pipeline must decide whether it may use a website for AI: yes/no verdict for a domain, AI bot and use (train, rag, search) from robots.txt, IETF AIPREF, TDMRep, noai and RSL licenses, with a signed receipt. Errors: 400 invalid domain or unknown bot (not charged).