# Crawl Policy History

> "What is this host's crawl and AI-training policy right now (robots.txt, llms.txt, ai.txt, TDM headers), and has it changed between the snapshots we have on record?"

- Status: MCP package (hosted access coming soon)

## The question it answers

"What is this host's crawl and AI-training policy right now (robots.txt, llms.txt, ai.txt, TDM headers), and has it changed between the snapshots we have on record?"

## Use it when

- You want a dated, hashed record of a host's policy files and AI-bot rules that you can compare over time.
- You want a list of Internet Archive captures of a host's robots.txt to look at yourself.

## Do not use it when

- You need a long history on day one. History only exists for snapshots **we have taken**. It starts when you first call it, so schedule repeated calls. Wayback captures are returned as links only; they are not turned into history.
- You only need today's answer for one URL. Use `crawl-policy-oracle`.
- You need legal advice.

## Example call

There is no public HTTP route yet, so no curl example. Locally:
```bash
npx tsx src/cli.ts snapshot example.com
npx tsx src/cli.ts diff example.com
```
MCP tools: `snapshot_policy {host, bots?}`, `list_policy_history {host}`, `diff_policy_history {host, bots?}`, `backfill_wayback_robots {host, limit?}`, `snapshot_and_diff {host, backfill?, backfillLimit?}`.

## Example response (diff; trimmed; shape follows the code)

```json
{
  "host": "example.org",
  "snapshotCount": 3,
  "events": [
    { "bot": "GPTBot", "at": "2026-11-02T09:00:00.000Z", "from": "allowed", "to": "blocked", "snapshotId": "2026-11-02T09-00-00-000Z_849ecc40589f" },
    { "bot": "ClaudeBot", "at": "2026-11-02T09:00:00.000Z", "from": "unknown", "to": "blocked", "snapshotId": "2026-11-02T09-00-00-000Z_849ecc40589f",
      "note": "status unknown — not a confirmed allow/block change" }
  ],
  "docHashChanges": [
    { "at": "2026-11-02T09:00:00.000Z", "kind": "robots.txt", "fromSha": "ab12...", "toSha": "cd34..." }
  ],
  "latest": { "host": "example.org", "source": "live", "bots": [ { "bot": "GPTBot", "blockedAtRoot": true } ], "tdm": { "noai": false, "noimageai": false } }
}
```
`unknown` means robots.txt could not be read at that time. It is never reported as "allowed". Bot status is checked for the root path `/` only.

## Data sources and licence

- The host's own public policy files and homepage headers.
- Internet Archive CDX index (robots.txt captures): returned as links and metadata only. Archive content is not our data and is not resold. Check the Internet Archive's terms before using it further.

## Freshness

Each snapshot is live at the time of the call. Stored snapshots are never refreshed or deleted by the tool.

## Honest limits

- Snapshots are stored on the machine running the tool, so history is only as good as how often you call it.
- Known bug in this version: a robots.txt that returns 404 (no rules) is recorded as `unknown` for every bot instead of "allowed". A patch exists but is not applied here.
- Known bug: robots rules ending in `$` can be matched wrongly (same matcher as the oracle).
- `noai` / `noimageai` are true if that word appears anywhere in the homepage HTML or the `X-Robots-Tag` header, so false positives are possible.
- Does not run JavaScript; meta tags come from raw HTML only.
- 5 MB per file, 20 s timeout. Not legal advice.