All projects matching “url redirect tracer”

Trace an HTTP redirect chain to its final destination without downloading page bodies. Returns every hop (URL + status), the final URL, and safety warnings — link-safety grounding for agents before they visit or cite a URL. Clean JSON.

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.