# Crawl directives (robots.txt + sitemap)

> Whether your robots.txt and XML sitemap let AI search and citation crawlers find and reach your product URLs at all.

<!-- Canonical: https://agentmint.net/signals/crawl-directives/ · Signal in the AgentMint.net Signals Database · Evidence: spec-fact (Very high confidence) · Last verified 2026-07-08 · Evidence types are load-bearing: spec-fact | reported | hypothesis (no measured). -->

- **Affects (funnel stage):** Discovery
- **Evidence:** spec-fact
- **Confidence:** Very high
- **Impact:** 3/3
- **Status:** active
- **Last verified:** 2026-07-08

## Per-platform status

Undocumented platforms show "unknown" - honest unknowns, never invented.

| Platform | Status | Notes | Source |
| --- | --- | --- | --- |
| ChatGPT | supported | OpenAI states that disallowing OAI-SearchBot means you will not be shown in ChatGPT search answers; the search crawler honors robots.txt. | [link](https://developers.openai.com/api/docs/bots) (verified 2026-07-08) |
| Gemini / Google AI Mode | unknown | Not documented by this platform. | - |
| Perplexity | unknown | Not documented by this platform. | - |
| Copilot | unknown | Not documented by this platform. | - |
| Claude | unknown | Not documented by this platform. | - |

## Evidence and sources

- [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (2026-07-08)
- [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (2026-07-08)
- [Rankly UCP/ACP validator](https://tryrankly.com/ucp-validator) (2026-07-08)

## How to act on this

Allow the named search/citation crawlers in robots.txt, decide training tokens (GPTBot, Google-Extended) deliberately, and keep an accessible current XML sitemap of live product URLs.

### Related checklist items
- [Allow the AI search and citation crawlers in robots.txt (OAI-SearchBot, Claude-SearchBot, PerplexityBot).](https://agentmint.net/tools/agentic-readiness-checklist/#a-bot-access-01) (Readiness)
- [Set the AI-training tokens (GPTBot, Google-Extended) as a deliberate, documented decision, separate from search access.](https://agentmint.net/tools/agentic-readiness-checklist/#a-bot-access-02) (Readiness)
- [Don't rely on robots.txt to control the user-initiated fetchers (ChatGPT-User, Perplexity-User).](https://agentmint.net/tools/agentic-readiness-checklist/#a-bot-access-03) (Readiness)
- [Keep an accessible, current XML sitemap.](https://agentmint.net/tools/agentic-readiness-checklist/#a-monitoring-04) (Readiness)

### Handbook chapters
- [Serving Agent Traffic Without Breaking Your Origin](https://agentmint.net/handbook/serving-agent-traffic/)

### Related signals
- [Agent infrastructure reachability](https://agentmint.net/signals/agent-infra-reachability/)
- [Origin serve-ability for agents](https://agentmint.net/signals/origin-serveability/)
- [Agent-readiness monitoring and ownership](https://agentmint.net/signals/agent-monitoring/)