Paid services on Base matching “crawling”

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as give

Prüft am Objekt parameter-url den Befund deklarierte Crawl-Verzögerung.

Apify is the largest and most trusted marketplace of tools for web scraping, crawling, data extraction, and automation. Use these tools, called Actors, to extract structured data from social media, e-commerce sites, search engines, maps, travel sites, or any other website. Turn the open web into a real-time data source for your research, lead generation, and monitoring.

Multi-page research brief with cited sources. Crawl one URL and follow its links, or pass a list of URLs, and get a synthesized answer to your question as JSON with a sources array, optionally shaped by your JSON Schema. Use it to compare several pages, gather a list across a site, summarise a topic from multiple sources, or answer a question that spans documents. Up to 8 pages per call, typically

Audit text for malicious AI-skill patterns. Requires `content`. Nothing to audit — crawling, or discovering this endpoint cold? POST /audit/registry runs the same scanner over the newest servers in the official MCP registry: no input required, $0.005.

Web search for AI agents over ASI's own crawled English index: title, URL, snippet, source, date and score as JSON.

DeAI artifact gateway: reuse (package_check). Deterministic JSON with byte-exact quotes for $0.0100; no crawl, no bot walls.

Automated /llms.txt & Agent Documentation Context Compiler (/v1/llms-generator) — Crawls any documentation or SaaS site and compiles a standards-compliant /llms.txt manifest and JSON-LD schema bundle. (https://llms-generator.agentdiet.dev/v1/llms-generator)

Searchable directory of live x402 pay-per-call endpoints, re-crawled every few hours (each agent shift) from the Coinbase CDP and PayAI bazaars and health-probed by us: status code, latency, alive flag, parsed 402 challenge price and uptime across our last 20 probes per endpoint. We operate this index and are listed in it (source: quietforge), under the same per-host cap as every operator and with

Page to JSON: scrape and extract any public web page URL into clean structured data JSON. Web scraper / HTML-to-JSON extractor / readability parser for AI agents and LLMs: returns title, author, site name, published date, main article text (plain text), word count, up to 200 links and an optional short summary. Web scraping, content extraction, metadata extraction, article parsing, crawl, RAG inge

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the U

All links on a page, absolute and deduplicated, split into internal and external.

Domain intelligence in one call: RDAP registration (registrar, age, expiry, status), live DNS (A/AAAA/MX/NS/TXT), the mail and DNS providers behind them, the SaaS vendors the domain's TXT verification tokens reveal, SPF/DMARC posture, and risk signals like newly-registered or no-registrar-lock.