Signal
Crawl directives (robots.txt + sitemap)
Whether your robots.txt and XML sitemap let AI search and citation crawlers find and reach your product URLs at all.
Per-platform status
Where a platform documents relevant behavior, its status and source are shown. Undocumented platforms show "unknown" - honest unknowns, never invented.
| Platform | Status | Notes | Source |
|---|---|---|---|
| ChatGPT | Supported | OpenAI states that disallowing OAI-SearchBot means you will not be shown in ChatGPT search answers; the search crawler honors robots.txt. | Source (opens in new tab) · verified 2026-07-08 |
| Gemini / Google AI Mode | Unknown | Not documented by this platform. | - |
| Perplexity | Unknown | Not documented by this platform. | - |
| Copilot | Unknown | Not documented by this platform. | - |
| Claude | Unknown | Not documented by this platform. | - |
Evidence and sources
This signal is typed spec-fact (Very high confidence).
- Overview of OpenAI Crawlers (opens in new tab) · 2026-07-08
- Google's common crawlers (opens in new tab) · 2026-07-08
- Rankly UCP/ACP validator (opens in new tab) · 2026-07-08
How to act on this
Allow the named search/citation crawlers in robots.txt, decide training tokens (GPTBot, Google-Extended) deliberately, and keep an accessible current XML sitemap of live product URLs.
Related checklist items
- Allow the AI search and citation crawlers in robots.txt (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Readiness
- Set the AI-training tokens (GPTBot, Google-Extended) as a deliberate, documented decision, separate from search access. Readiness
- Don't rely on robots.txt to control the user-initiated fetchers (ChatGPT-User, Perplexity-User). Readiness
- Keep an accessible, current XML sitemap. Readiness
Handbook chapters