# Crawl Policy Oracle

> "For this URL, what do the site's robots.txt, llms.txt and ai.txt say, and may AI crawlers such as GPTBot, ClaudeBot or CCBot fetch this path?"

- Status: MCP package (hosted access coming soon)

## The question it answers

"For this URL, what do the site's robots.txt, llms.txt and ai.txt say, and may AI crawlers such as GPTBot, ClaudeBot or CCBot fetch this path?"

You get a one-word recommendation (`allow`, `caution`, `avoid`), the sentence behind it, and a per-bot allowed / blocked / unknown list for that exact path.

## Use it when

- An agent is about to fetch a page and should check the site's stated crawl rules first.
- You want to know which named AI bots a site blocks.

## Do not use it for

- Legal advice. It reads robots.txt; it does not read the site's terms of service.
- Crawl-delay scheduling. The result does not report `Crawl-delay`.
- A history of changes. Use `crawl-policy-history` for that.

## Example call

There is no public HTTP route yet, so there is no curl example. Locally (CLI):
```bash
npx tsx src/cli.ts check https://example.com/docs agent_fetch GPTBot
```
Or the MCP tool `may_i_crawl` with `{ "url": "https://example.com/docs", "ua_hint": "GPTBot" }`.

Params: `url` (required), `purpose` (optional, echoed back), `ua_hint` (optional; checks `*` plus that one bot instead of the default list of 14 AI bots).

## Example response (trimmed; shape follows the code; values are for a site with no robots.txt)

```json
{
  "tool": "crawl-policy-oracle",
  "version": "0.2.1",
  "origin": "https://example.com",
  "path": "/docs",
  "robots": {
    "url": "https://example.com/robots.txt",
    "ok": false, "status": 404, "statusUnknown": false,
    "ruleGroups": 0, "sitemaps": [],
    "pathChecks": [
      { "bot": "*", "statusUnknown": false, "allowed": true, "matchedAgent": "(none)" },
      { "bot": "GPTBot", "statusUnknown": false, "allowed": true, "matchedAgent": "(none)" }
    ],
    "summary": { "starAllowed": true, "blockedAiBots": [], "allowedAiBots": ["GPTBot"], "unknownAiBots": [] },
    "fetched": false
  },
  "llmsTxt": { "url": "https://example.com/llms.txt", "ok": false, "status": 404, "parsed": null },
  "aiTxt": { "url": "https://example.com/.well-known/ai.txt", "ok": false, "status": 404, "bytes": 0 },
  "securityTxt": { "url": "https://example.com/.well-known/security.txt", "ok": false, "status": 404 },
  "recommendation": "caution",
  "recommendationDetail": "No robots.txt (404) — technically unrestricted for robots, but stay polite and verify ToS.",
  "purpose": "agent_fetch", "uaHint": "GPTBot", "cached": false
}
```
`allowed: null` with `statusUnknown: true` means robots.txt could not be fetched (and it was not a 404). Treat that as **not allowed**.

## Data sources

Only the target site's own `robots.txt`, `llms.txt`, `/.well-known/ai.txt` and `/.well-known/security.txt`. No third-party data. Fetches use a polite User-Agent with at least 150 ms between requests.

## Freshness and caching

Results are cached on disk with **no expiry**, keyed by host, path and bot list. A site that changes its robots.txt can still show the old answer. A cache TTL is on the fix list. Failed lookups are not cached.

## Honest limits

- `recommendation` comes from keywords in the explanation sentence. A site with no robots.txt, or with an llms.txt, reads `caution`, not `allow`.
- Known bug in this version: rules ending in `$` (for example `Disallow: /*.pdf$`) can be missed when the path repeats the ending (`/a.pdf.pdf`), which wrongly reports "allowed". A patch exists but is not applied here.
- Group matching is by substring, so a robots group named `ai` also catches `anthropic-ai`.
- Max 5 MB per fetched file, 20 s timeout, 5 redirects.
- Not legal advice.