Signal
Crawl directives (robots.txt + sitemap)
Whether your robots.txt and XML sitemap let AI search and citation crawlers find and reach your product URLs at all.
Per-platform status
Where a platform documents relevant behavior, its status and source are shown. Undocumented platforms show "unknown" - honest unknowns, never invented.
| Platform | Status | Notes | Source |
|---|---|---|---|
| ChatGPT | Supported | OAI-SearchBot and GPTBot respect robots.txt, and disallowing opts a site out with roughly 24 hours of propagation. XML sitemaps are not mentioned. | Source (opens in new tab) · verified 2026-09-18 |
| Gemini / Google AI Mode | Supported | robots.txt controls access per user agent. Storebot-Google crawls Shopping surfaces and Googlebot feeds AI Mode, so directives govern whether product URLs are reachable at all. | Source (opens in new tab) · verified 2026-07-14 |
| Perplexity | Supported | robots.txt controls PerplexityBot, which surfaces and links sites in search results. Perplexity-User, the user-triggered fetcher, generally ignores robots.txt. | Source (opens in new tab) · verified 2026-09-18 |
| Copilot | Unknown | Not documented by this platform. | - |
| Claude | Supported | ClaudeBot, Claude-User and Claude-SearchBot all honor robots.txt, plus the non-standard Crawl-delay. XML sitemaps are not addressed. | Source (opens in new tab) · verified 2026-04-07 |
Evidence and sources
This signal is typed spec-fact (Very high confidence).
How to act on this
Allow the named search/citation crawlers in robots.txt, decide training tokens (GPTBot, Google-Extended) deliberately, and keep an accessible current XML sitemap of live product URLs.
Related checklist items
- Allow the AI search and citation crawlers in robots.txt (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Readiness
- Set the AI-training tokens (GPTBot, Google-Extended) as a deliberate, documented decision, separate from search access. Readiness
- Don't rely on robots.txt to control the user-initiated fetchers (ChatGPT-User, Perplexity-User). Readiness
- Keep an accessible, current XML sitemap. Readiness
Handbook chapters
Notes
Cloudflare's Content Signals extension adds an optional fourth field beside search and ai-train, written as use=immediate, use=reference or use=full, which states what a crawler may keep and reshare rather than whether it may fetch. These are declarative preferences, not enforcement, and no major engine has committed to honouring them.