All projects matching “follow redirect chain”

URL safety verdict: follows the redirect chain, detects homograph and punycode hostname tricks, and cross-references the destination host against the URLhaus threat feed. Returns safe, suspicious or malicious with reasons. Single-URL scope; domain-wide grading is /sitegrade and registration facts are /domain-intel.

Send an HTTP request to any URL from our servers and receive the full response — including the status code, all response headers, and the body content. Supports GET, POST, PUT, PATCH, DELETE, and HEAD. Automatically follows up to 3 redirects. Useful for testing endpoints, verifying servers are responding correctly, debugging redirect chains, or fetching raw data without needing a local setup.

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.