Crawl Policy Oracle

The question it answers

"For this URL, what do the site's robots.txt, llms.txt and ai.txt say, and may AI crawlers such as GPTBot, ClaudeBot or CCBot fetch this path?"

You get a one-word recommendation (allow, caution, avoid), the sentence behind it, and a per-bot allowed / blocked / unknown list for that exact path.

Use it when

• An agent is about to fetch a page and should check the site's stated crawl rules first.
• You want to know which named AI bots a site blocks.

Do not use it for

• Legal advice. It reads robots.txt; it does not read the site's terms of service.
• Crawl-delay scheduling. The result does not report Crawl-delay.
• A history of changes. Use crawl-policy-history for that.

Example call

There is no public HTTP route yet, so there is no curl example. Locally (CLI):

npx tsx src/cli.ts check https://example.com/docs agent_fetch GPTBot

Or the MCP tool may_i_crawl with { "url": "https://example.com/docs", "ua_hint": "GPTBot" }.

Params: url (required), purpose (optional, echoed back), ua_hint (optional; checks * plus that one bot instead of the default list of 14 AI bots).

Example response (trimmed; shape follows the code; values are for a site with no robots.txt)

{
  "tool": "crawl-policy-oracle",
  "version": "0.2.1",
  "origin": "https://example.com",
  "path": "/docs",
  "robots": {
    "url": "https://example.com/robots.txt",
    "ok": false, "status": 404, "statusUnknown": false,
    "ruleGroups": 0, "sitemaps": [],
    "pathChecks": [
      { "bot": "*", "statusUnknown": false, "allowed": true, "matchedAgent": "(none)" },
      { "bot": "GPTBot", "statusUnknown": false, "allowed": true, "matchedAgent": "(none)" }
    ],
    "summary": { "starAllowed": true, "blockedAiBots": [], "allowedAiBots": ["GPTBot"], "unknownAiBots": [] },
    "fetched": false
  },
  "llmsTxt": { "url": "https://example.com/llms.txt", "ok": false, "status": 404, "parsed": null },
  "aiTxt": { "url": "https://example.com/.well-known/ai.txt", "ok": false, "status": 404, "bytes": 0 },
  "securityTxt": { "url": "https://example.com/.well-known/security.txt", "ok": false, "status": 404 },
  "recommendation": "caution",
  "recommendationDetail": "No robots.txt (404) — technically unrestricted for robots, but stay polite and verify ToS.",
  "purpose": "agent_fetch", "uaHint": "GPTBot", "cached": false
}

allowed: null with statusUnknown: true means robots.txt could not be fetched (and it was not a 404). Treat that as not allowed.

Data sources

Only the target site's own robots.txt, llms.txt, /.well-known/ai.txt and /.well-known/security.txt. No third-party data. Fetches use a polite User-Agent with at least 150 ms between requests.

Freshness and caching

Results are cached on disk with no expiry, keyed by host, path and bot list. A site that changes its robots.txt can still show the old answer. A cache TTL is on the fix list. Failed lookups are not cached.

Honest limits

• recommendation comes from keywords in the explanation sentence. A site with no robots.txt, or with an llms.txt, reads caution, not allow.
• Known bug in this version: rules ending in $ (for example Disallow: /*.pdf$) can be missed when the path repeats the ending (/a.pdf.pdf), which wrongly reports "allowed". A patch exists but is not applied here.
• Group matching is by substring, so a robots group named ai also catches anthropic-ai.
• Max 5 MB per fetched file, 20 s timeout, 5 redirects.
• Not legal advice.

Errors are not charged

Payment settles only on a successful answer. Any response with status 400 or higher (bad input, not found, upstream failure) is never settled.