All projects on Base matching “server header check”

Audit a URL's HTTP security headers over a single body-free (HEAD) request. Grades Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy and Permissions-Policy, flags information-leak headers (Server, X-Powered-By), and returns the parsed values plus advisory warnings. Clean JSON for security and pentest automation.

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.