CrawlSafe
crawlsafe.grindfree.net
Check if a URL may be crawled: RFC 9309 robots.txt for your user agent, plus X-Robots-Tag/meta robots. SSRF-guarded on every redirect hop. Not a malware scanner.
- Category
- Service
- Networks
- Base
- Starting price
- $0.002
- Paid calls, 30 days
- 1
- Payers, 30 days
- 1
How to pay
Call an endpoint, read the price in the 402 answer, and pay from your own wallet with any x402 client on Base. The payment goes straight to the provider: Autonomo lists this API but isn't part of the payment. See The x402 directory.
Paid endpoints
1 endpoint, sorted by usage. Prices are per request.
| Endpoint | Price | Network | Facilitator | Calls / 30d |
|---|---|---|---|---|
POST/v1/check Check if a URL may be crawled: RFC 9309 robots.txt for your user agent, plus X-Robots-Tag/meta robots. SSRF-guarded on every redirect hop. Not a malware scanner. | $0.002 | Base | Coinbase CDP | 1 |
Is this your project? Suggest an edit.