All projects on Base matching “crawling”

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as give

Fetch a site's robots.txt and evaluate crawl permissions: given a URL (plus optional extra paths) and a user-agent, return whether each path is allowed or disallowed, the rule that matched, the user-agent group, the crawl-delay and any declared sitemaps. Implements the Robots Exclusion Protocol (RFC 9309) with longest-match-wins, Allow-over-Disallow tie-breaking and * / $ wildcards.

Prüft am Objekt parameter-url den Befund deklarierte Crawl-Verzögerung.

Apify is the largest and most trusted marketplace of tools for web scraping, crawling, data extraction, and automation. Use these tools, called Actors, to extract structured data from social media, e-commerce sites, search engines, maps, travel sites, or any other website. Turn the open web into a real-time data source for your research, lead generation, and monitoring.

Audit text for malicious AI-skill patterns. Requires `content`. Nothing to audit — crawling, or discovering this endpoint cold? POST /audit/registry runs the same scanner over the newest servers in the official MCP registry: no input required, $0.005.

Multi-page research brief with cited sources. Crawl one URL and follow its links, or pass a list of URLs, and get a synthesized answer to your question as JSON with a sources array, optionally shaped by your JSON Schema. Use it to compare several pages, gather a list across a site, summarise a topic from multiple sources, or answer a question that spans documents. Up to 8 pages per call, typically

Web search for AI agents over ASI's own crawled English index: title, URL, snippet, source, date and score as JSON.

DeAI artifact gateway: reuse (package_check). Deterministic JSON with byte-exact quotes for $0.0100; no crawl, no bot walls.

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the U

Page to JSON: scrape and extract any public web page URL into clean structured data JSON. Web scraper / HTML-to-JSON extractor / readability parser for AI agents and LLMs: returns title, author, site name, published date, main article text (plain text), word count, up to 200 links and an optional short summary. Web scraping, content extraction, metadata extraction, article parsing, crawl, RAG inge

Automated /llms.txt & Agent Documentation Context Compiler (/v1/llms-generator) — Crawls any documentation or SaaS site and compiles a standards-compliant /llms.txt manifest and JSON-LD schema bundle. (https://llms-generator.agentdiet.dev/v1/llms-generator)

All links on a page, absolute and deduplicated, split into internal and external.

Domain intelligence in one call: RDAP registration (registrar, age, expiry, status), live DNS (A/AAAA/MX/NS/TXT), the mail and DNS providers behind them, the SaaS vendors the domain's TXT verification tokens reveal, SPF/DMARC posture, and risk signals like newly-registered or no-registrar-lock.