Signal
Origin serve-ability for agents
Keeping your origin open and cheap to serve, by edge-caching the near-static machine surfaces and allowlisting verified bots while rate-limiting only unverified spoofers, so a verified agent always gets a fast 200 instead of a block.
Per-platform status
Where a platform documents relevant behavior, its status and source are shown. Undocumented platforms show "unknown" - honest unknowns, never invented.
| Platform | Status | Notes | Source |
|---|---|---|---|
| ChatGPT | Partial | OpenAI publishes verified crawler IP files so genuine agents can be allowlisted. Edge caching and rate limiting are not documented. | Source (opens in new tab) · verified 2026-09-18 |
| Gemini / Google AI Mode | Partial | Google publishes IP ranges for allowlisting and notes Storebot-Google only supports caching in certain conditions. There is no edge doctrine for machine surfaces. | Source (opens in new tab) · verified 2026-03-20 |
| Perplexity | Unknown | Not documented by this platform. | - |
| Copilot | Unknown | Not documented by this platform. | - |
| Claude | Partial | Anthropic publishes a source-IP list so merchants can allowlist genuine Claude bots. Caching and rate limiting are the merchant's side and are not documented. | Source (opens in new tab) · verified 2026-04-07 |
Evidence and sources
This signal is typed hypothesis (Low confidence).
- Cloudflare Radar 2025 Year in Review (opens in new tab) · 2025-12-15
- Cloudflare, Cache revalidation (opens in new tab)
- nginx, ngx_http_proxy_module (opens in new tab)
- Wikimedia Diff, Mueller, Danis and Lavagetto (opens in new tab) · 2025-04-01
- 404 Media, Jason Koebler (opens in new tab) · 2024-07-24
- Read the Docs, Eric Holscher (opens in new tab) · 2024-07-25
How to act on this
Edge-cache the near-static machine surfaces (llms.txt, Markdown mirrors, feeds, mostly-static product pages) with stale-while-revalidate, and allowlist verified bots while rate-limiting only unverified spoofers.
Related checklist items
Handbook chapters