Signal
Agent infrastructure reachability
Whether a crawler you allow in robots.txt actually receives your full page - a 200 over HTTPS, not a WAF challenge, rate-limit, or geoblock - rather than being silently turned away.
The consequence for selection is still inferred, but the cause is now documented: a major CDN blocks agent-class traffic by default in stated conditions.
Per-platform status
Where a platform documents relevant behavior, its status and source are shown. Undocumented platforms show "unknown" - honest unknowns, never invented.
| Platform | Status | Notes | Source |
|---|---|---|---|
| ChatGPT | Partial | OpenAI publishes verified crawler IP files and the feed requires publicly accessible HTTP or HTTPS product URLs. What happens when a crawler meets a WAF challenge is not documented. | Source (opens in new tab) · verified 2026-09-18 |
| Gemini / Google AI Mode | Partial | Google publishes crawler IP ranges and reverse-DNS verification so you can allowlist genuine crawlers. It does not document the outcome of a WAF challenge, rate limit or geoblock. | Source (opens in new tab) · verified 2026-03-20 |
| Perplexity | Unknown | Not documented by this platform. | - |
| Copilot | Unknown | Not documented by this platform. | - |
| Claude | Partial | The web fetch tool documents url_not_accessible on HTTP errors and too_many_requests on rate limits. WAF-challenge handling is not covered. | Source (opens in new tab) · verified 2026-09-18 |
Evidence and sources
This signal is typed hypothesis (Medium confidence).
How to act on this
Fetch your own product URL with each vendor's published user-agent and IP range, confirm a 200 (not a challenge), serve everything over HTTPS with no mixed content, and re-test after any WAF or CDN change.
Related checklist items
Handbook chapters
Notes
From 2026-09-15 Cloudflare blocks its Agent and Training classes by default on ad-bearing pages for domains newly onboarding, and judges multi-purpose crawlers by every behaviour they carry, so a customer who blocked training can lose Googlebot, Applebot and Bingbot with it. Verified status no longer means allowed by default. Check what your own CDN classifies as agent traffic before assuming an allow in robots.txt is enough.