All projects matching “url”

Use when you need the list of pages a site publishes (to crawl it, find recent posts, or audit SEO): give a site URL (the sitemap is discovered from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) or a sitemap URL directly. Returns the page URLs with lastmod, changefreq and priority; for a sitemap index, returns its child sitemaps and (expand=true, default) the U

Call when qualifying a short company website list. Processes 2-5 URL slots for one fixed payment and returns evidenced profiles in input order, with per-URL failures. Batch deadline 18 seconds; each target at most 8.5 seconds. Settles only if at least half are delivered and no target was left unstarted by the deadline; otherwise returns all items uncharged. No contact values or outreach.

Cheap robots.txt preflight check before you call the Minifetch /extract and /run/seo-page-audit endpoints: returns whether Minifetch may fetch a target URL (allow/deny) plus its crawl-delay. If blocked and the site is not yours try our external proxy routes. If blocked and the site is yours or to set a custom crawl-delay read https://minifetch.com/tutorials/unblock-minifetch Checks the URL as give

Unblocker for AI agents: send {url} and get the page back as clean Markdown, with title, description, language, final URL and HTTP status. Gets through Cloudflare and other common bot protection and can render JavaScript-heavy pages. Set format to html or text for those instead. A page that can't be fetched (blocked, error status or empty) is not charged. No API key, proxy or account: pay per call

Returns the page's readable text plus an Ed25519-signed record (final URL after redirects, fetch time, status, sha256 of the body and of the text) that verifies offline. Charged only for 2xx text pages; refused URLs cost nothing.

Scrapes any webpage and pulls title, description, canonical URL, OpenGraph + Twitter card metadata, headings, and outbound links from a single URL. Server-side rendering; body content rendered as text, raw HTML, or clean markdown, with optional link extraction. Cheerio-based with no headless browser, so it is fast and cheap, ideal for static pages and SSR sites. For JS-heavy SPAs that need a real

Capture a PNG screenshot of a single CSS-selected element on an http(s) URL; returns a JSON envelope with the PNG base64-encoded under image_base64, plus selector, final_url, title, viewport, and size.

URL safety verdict: follows the redirect chain, detects homograph and punycode hostname tricks, and cross-references the destination host against the URLhaus threat feed. Returns safe, suspicious or malicious with reasons. Single-URL scope; domain-wide grading is /sitegrade and registration facts are /domain-intel.

Opens each URL in a cloud browser and returns one tab per URL, in input order, with the title, content type, extracted text, the accessibility tree and a link map. Results over the inline size limit return a preview per tab plus a download reference. Each call is one request.

Read any public web page as clean Markdown: main-content extraction (navigation, footers, scripts and hidden elements removed), headings, lists, tables, code blocks, links (absolute, deduplicated), images, plus title, description, canonical URL, language, author and publish date, and a content hash + fetch time as evidence. JSON and plain-text URLs are returned as-is. No JavaScript rendering. $0.0

SEO and AEO website audit. Returns an audit ID and status URL while processing, then a report URL when completed. Check status approximately every 30 seconds.

Get structured metadata for any public web page in one call: title, meta description, canonical URL, Open Graph and Twitter card tags, favicons, JSON-LD schema types, language and the heading outline. Use it to build link previews and social cards, classify or dedupe URLs, and run SEO and AI-readability audits - far cheaper than fetching and parsing the whole page.

Request an https URL and report whether it is reachable, its status, every redirect hop with timings, key and security headers (HSTS, CSP and others), and the TLS certificate with its issuer, the names it covers and the days until it expires. An unreachable URL is a result too, with the reason.

Use this before an AI agent pays an x402 endpoint: give the endpoint URL, we read its own 402 quote, pull every payTo (all networks) and screen each against OFAC, UN, EU and UK lists. Signed per-payee and overall verdict: deny on an OFAC match, review on a UN/EU/UK match or an unavailable list, clean otherwise. $0.01 in USDC on Base or Solana. A bad, unreachable or non-x402 URL returns 400, 422 or

The Phoblographer content lookup. Include the content URL or slug in the request using the content_url key

Extract clean, boilerplate-free readable text from one known public HTTP(S) URL for LLM research or RAG ingestion; private, local, authenticated, paywalled, or blocked resources are not supported. USE WHEN: agent already knows the URL and needs clean page text. DO NOT USE WHEN: agent needs web search, multi-page research, comparison, structured web workflow, or change detection. USE web-intelligen

Get structured metadata for any public web page in one call: title, meta description, canonical URL, Open Graph and Twitter card tags, favicons, JSON-LD schema types, language and the heading outline. Use it to build link previews and social cards, classify or dedupe URLs, and run SEO and AI-readability audits - far cheaper than fetching and parsing the whole page.

On-page SEO audit of one URL: title and meta description length, canonical, robots meta and X-Robots-Tag, robots.txt allow for user-agent *, sitemap.xml with URL count, lang, hreflang, h1-h3 outline, word count, images without alt, internal/external links, Open Graph, Twitter card, JSON-LD schema.org types, favicon, viewport, mixed content; each check pass/warn/fail with a fix and a 0-100 score.

Real-time web search for AI agents. Search the web for current and up-to-date information, news, facts, research, documentation, and sources beyond model training data. Returns ranked structured JSON results with title, URL, description, and position for web grounding, RAG retrieval, research, and fact-finding. Send query, optional limit, and optional freshness as URL parameters. Keyless x402 pay-

Builder tool for agents: FREE GET /product/bazaar-check.json?url= → indexed yes/nein + issues_found count (teaser). PAID GET/POST /product/bazaar-check/report.json?url= ($0.05 USDC Base) → 402 header compliance, extensions.bazaar schema validation, CDP quality fields, concrete Fix-JSON. Fail-soft. English-first. Brand: geld machen.

SWARM Read Page - Retrieve the content of a URL as clean, LLM-ready text (headings with #, lists with -; menus, scripts, styles and markup removed). Use when an agent needs to read a web page, article or docs page without paying model tokens for raw HTML: on 15 public pages we measured, the text was a median 97% smaller than the HTML (about 67,000 fewer tokens per page). Requires `url`; HTML or te

Verify a claim or fact check a statement against public web sources: send a question and up to 5 URLs, get cited evidence - passages that support, contradict or fail to settle it. Claim verification with a citation for every excerpt: source URL, retrieval time, content hash. Never charges when nothing is retrieved. Free to evaluate: the `sample` tool returns one real response at no cost, so you ca

Multi-page research brief with cited sources. Crawl one URL and follow its links, or pass a list of URLs, and get a synthesized answer to your question as JSON with a sources array, optionally shaped by your JSON Schema. Use it to compare several pages, gather a list across a site, summarise a topic from multiple sources, or answer a question that spans documents. Up to 8 pages per call, typically

Analyze any image URL using GPT-4o-mini vision. Returns structured analysis based on the mode: describe (full description), ocr (text extraction), chart (data/trend extraction), ui (interface analysis), identify (object/subject ID), or qa (answer a specific question about the image). Input must be a publicly accessible image URL (JPEG, PNG, GIF, WebP). $0.261/call.

Fetch one URL from a residential IP and get the raw response as JSON: status, final URL, headers and body (text or base64), up to 10 MB. GET, HEAD or POST, your own headers, an optional country. No JavaScript rendering. You pay for the megabytes delivered; a fetch that fails is not charged. Pay per call in stablecoins.

Extract clean, LLM-ready Markdown or plain text from up to 5 public webpage URLs. Removes page boilerplate and returns structured per-URL results for AI agents, RAG ingestion, research and summarization.

Twitter (X) video & image downloader — pass a public tweet / X post URL, get direct CDN media links: .mp4 video URLs with thumbnails plus every image in the post, ready to stream, embed, re-host, or download. No scraping or browser automation.

Fetch one public HTTP(S) URL for research, extraction, or agent workflows. Returns the final URL, upstream status, content type, and decoded content. Private, local, metadata, and reserved destinations are blocked; redirects are revalidated and response size and duration are bounded.

Request an https URL and report whether it is reachable, its status, every redirect hop with timings, key and security headers (HSTS, CSP and others), and the TLS certificate with its issuer, the names it covers and the days until it expires. An unreachable URL is a result too, with the reason.

Full enforcement report: every matching record with title, date, agency, court, case number, case type, official source URL and courtpls.com page URL, plus per-month and per-case-type breakdowns and the basis each record matched on.