All projects matching “user-agent”

Detect whether a User-Agent belongs to a known bot, crawler, AI agent or HTTP tool, and if so return its name and category (search, social, seo, ai, monitoring, tool, …).

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

Check a URL before fetching it: is it alive, final URL after redirects, HTTP status, content type, HTML title, and whether robots.txt allows your user-agent. Link checker and fetch preflight for agents.

On-page SEO audit of one URL: title and meta description length, canonical, robots meta and X-Robots-Tag, robots.txt allow for user-agent *, sitemap.xml with URL count, lang, hreflang, h1-h3 outline, word count, images without alt, internal/external links, Open Graph, Twitter card, JSON-LD schema.org types, favicon, viewport, mixed content; each check pass/warn/fail with a fix and a 0-100 score.

Overwing guardrail: check text before you send, post, or act on it. Content moderation, PII detection, confidential leak detection, toxicity, self-harm and severity in ~150 ms. Returns a verdict plus one recommended_action (block / redact / review / allow) with calibrated confidence. Pass context (recipient, channel, owns_contact_info) with the outbound-message set. $0.002 USDC, no account.