Crawl Policy History

The question it answers

"What is this host's crawl and AI-training policy right now (robots.txt, llms.txt, ai.txt, TDM headers), and has it changed between the snapshots we have on record?"

Use it when

• You want a dated, hashed record of a host's policy files and AI-bot rules that you can compare over time.
• You want a list of Internet Archive captures of a host's robots.txt to look at yourself.

Do not use it when

• You need a long history on day one. History only exists for snapshots we have taken. It starts when you first call it, so schedule repeated calls. Wayback captures are returned as links only; they are not turned into history.
• You only need today's answer for one URL. Use crawl-policy-oracle.
• You need legal advice.

Example call

There is no public HTTP route yet, so no curl example. Locally:

npx tsx src/cli.ts snapshot example.com
npx tsx src/cli.ts diff example.com

MCP tools: snapshot_policy {host, bots?}, list_policy_history {host}, diff_policy_history {host, bots?}, backfill_wayback_robots {host, limit?}, snapshot_and_diff {host, backfill?, backfillLimit?}.

Example response (diff; trimmed; shape follows the code)

{
  "host": "example.org",
  "snapshotCount": 3,
  "events": [
    { "bot": "GPTBot", "at": "2026-11-02T09:00:00.000Z", "from": "allowed", "to": "blocked", "snapshotId": "2026-11-02T09-00-00-000Z_849ecc40589f" },
    { "bot": "ClaudeBot", "at": "2026-11-02T09:00:00.000Z", "from": "unknown", "to": "blocked", "snapshotId": "2026-11-02T09-00-00-000Z_849ecc40589f",
      "note": "status unknown — not a confirmed allow/block change" }
  ],
  "docHashChanges": [
    { "at": "2026-11-02T09:00:00.000Z", "kind": "robots.txt", "fromSha": "ab12...", "toSha": "cd34..." }
  ],
  "latest": { "host": "example.org", "source": "live", "bots": [ { "bot": "GPTBot", "blockedAtRoot": true } ], "tdm": { "noai": false, "noimageai": false } }
}

unknown means robots.txt could not be read at that time. It is never reported as "allowed". Bot status is checked for the root path / only.

Data sources and licence

• The host's own public policy files and homepage headers.
• Internet Archive CDX index (robots.txt captures): returned as links and metadata only. Archive content is not our data and is not resold. Check the Internet Archive's terms before using it further.

Freshness

Each snapshot is live at the time of the call. Stored snapshots are never refreshed or deleted by the tool.

Honest limits

• Snapshots are stored on the machine running the tool, so history is only as good as how often you call it.
• Known bug in this version: a robots.txt that returns 404 (no rules) is recorded as unknown for every bot instead of "allowed". A patch exists but is not applied here.
• Known bug: robots rules ending in $ can be matched wrongly (same matcher as the oracle).
• noai / noimageai are true if that word appears anywhere in the homepage HTML or the X-Robots-Tag header, so false positives are possible.
• Does not run JavaScript; meta tags come from raw HTML only.
• 5 MB per file, 20 s timeout. Not legal advice.

Errors are not charged

Payment settles only on a successful answer. Any response with status 400 or higher (bad input, not found, upstream failure) is never settled.