Key takeaways
- AI shopping agents read two things about you: your structured product feed and your live pages. robots.txt is what decides whether they can read the second one.
- Whether to allow the AI crawlers is a selection decision, not a bandwidth one: block them and you can remove the page-level detail an agent would otherwise read about your products.
- The commerce allow-list is a specific set of real user-agent tokens, each owned by a named engine: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), Google-Extended (Google), PerplexityBot and Perplexity-User (Perplexity).
- Crawler access and your product feed are two separate pipes. Opening the crawlers does not replace a feed, and a feed does not make the crawlers unnecessary.
- llms.txt is an emerging, proposed convention worth shipping as low-cost insurance, but there is no confirmed evidence yet that major shopping agents read it when choosing products.
What each AI crawler actually does
Before you can decide who to let in, it helps to know that the AI crawlers do two different jobs, and the difference is what matters for a store. Some fetch and index your pages so an engine can surface and cite you later; others are a live fetch triggered the moment a user's request reaches your URL. We will not rehearse robots.txt grammar here (the syntax is owned by each engine's own documentation, which we link) because the useful question for a merchant is not how to write a directive but which of these agents you want reading your catalog and why.
Take the three engines that dominate agentic shopping today.
OpenAI documents several distinct agents, including: GPTBot, which crawls content to train its models; OAI-SearchBot, which surfaces and links sites inside ChatGPT search; and ChatGPT-User, the fetch triggered when a ChatGPT user's own request reaches your page.Spec-factOpenAI, Bots documentation That split is the whole reason "should I allow AI crawlers?" has no single yes/no answer: GPTBot is mostly about training, while OAI-SearchBot and ChatGPT-User are the ones that decide whether a shopper sees you inside ChatGPT.
Google-Extended is not itself a crawler but a robots.txt control token that governs whether content Google has already crawled may be used to improve Gemini Apps and Vertex AI generative models, and toggling it does not change how a site ranks in Google Search.Spec-factGoogle, Google's common crawlers For a store, that means the Google-Extended decision is purely an AI-usage decision, cleanly separable from your SEO.
Perplexity documents PerplexityBot, which indexes pages so they can be surfaced and cited in answers, and Perplexity-User, the fetch triggered by a user's live query.Spec-factPerplexity, Bots documentationAnthropic documents three bots: ClaudeBot, which collects web content that can contribute to model training; Claude-User, the fetch triggered when someone asks Claude a question; and Claude-SearchBot, which crawls pages to improve the quality of Claude's search results.Spec-factAnthropic, Claude Help Center Claude-SearchBot is the one that matters for selection: it is Anthropic's equivalent of OAI-SearchBot and PerplexityBot, the crawler behind whether Claude surfaces and cites a page at all.
Our own site ships the same allow-list for these plus Anthropic's ClaudeBot, Claude-SearchBot and Claude-User (an older anthropic-ai token still shows up in some third-party lists, but it is not part of Anthropic's current documented set above), Apple's Applebot-Extended, Amazonbot, and Common Crawl's CCBot. You can read the exact file at agentmint.net/robots.txt, because a publication whose purpose is to be read by agents should practise what it preaches.
Each bot-documentation link above was re-verified on 2026-07-08 and the per-token descriptions still match the operators' docs; the dated, token-by-token reference for every operator lives in the AI shopping crawlers reference.
The allow-list decision
For a store whose goal is to be found, quoted, and bought through AI agents, the sensible default is to allow the shopping and answer crawlers, and to treat blocking as the exception that needs a reason. The tokens above are your allow-list; the decision is which, if any, you deliberately close.
The one nuance worth spending thought on is training versus surfacing. Because OpenAI separates GPTBot (training) from OAI-SearchBot and ChatGPT-User (surfacing and live fetch), a merchant who is wary of feeding model training can, in principle, decline the training crawler while still letting the surfacing agents in. For most stores that middle ground is the pragmatic choice: you keep every path that puts you in front of a shopper and give up only the training use you were never compensated for.Hypothesis (our analysis)
What you lose by blocking
Blocking the AI crawlers rarely makes a store vanish outright, which is exactly why the cost is easy to miss. The loss is quieter: closing the crawlers removes the supplemental page data an agent reads beyond your structured feed: the fuller description copy, specification details, review context, and availability nuance that live on your product page but never make it into a feed row.Hypothesis (our analysis) An agent that can only see your feed is choosing from a thinner picture of your product than one that can also read the page.
Crawler access vs your feed: two separate pipes
It is tempting to think a good product feed makes crawler access optional, or that open crawlers make a feed redundant. Neither is true, because they are two different pipes into an agent. Your product feed is the structured, merchant-controlled channel (titles, price, availability, GTINs) delivered to an engine's commerce surface. Crawler access is the unstructured channel: the live page an agent can read for everything the feed leaves out.
The Google-Extended case makes the separation concrete. Because it is a usage-control token rather than a crawler, opening or closing it changes only whether your already-indexed content feeds Google's generative models. It touches neither your Merchant Center feed nor your search ranking. The lesson generalizes: audit the feed pipe and the crawler pipe independently, because a gap in one is invisible from the other. If your feed itself is thin, that is a separate fix: start with making your product feed AI-readable and the product schema agents trust.
llms.txt for stores
llms.txt is a proposed convention, published at llmstxt.org, for a root-level /llms.txt Markdown file that offers a large language model a curated, easy-to-parse map of a site's most important content.Spec-factllms.txt proposal, llmstxt.org The idea is to hand an agent a clean index instead of making it infer structure from your navigation. Our site ships a generated /llms.txt as a worked example; the format spec lives at llmstxt.org and we link out to it rather than reproduce it.
Here is the honest part, because this is where the hype outruns the evidence. There is no confirmed public evidence yet that ChatGPT, Gemini, or Perplexity read llms.txt when choosing products, so treat it as low-cost insurance for a future many expect, not as a ranking lever that moves selection today.Hypothesis (our analysis) Shipping one costs almost nothing and can only help if adoption arrives; building a strategy around it would be betting on behavior no engine has documented. One engine has now answered directly, and the answer is negative. Google's guide to optimizing for generative AI features, updated in June 2026, states that llms.txt files are not needed to appear in Google Search and that Google Search ignores them, so maintaining one neither helps nor harms a site's visibility or rankings there.Spec-factGoogle, Optimizing your website for generative AI features on Google Search Re-checked 2026-07-08: we found no documentation from OpenAI, Perplexity, Anthropic, or Microsoft that any shopping agent reads /llms.txt when selecting products.
Diagnose a store that's been blocked
If your products are missing from an agent's results, an accidentally-closed crawler is one of the first things to rule out: a stray Disallow, a bot-manager rule, or a CDN returning 403s to these user agents. Check that the surfacing crawlers for your target engine (OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot for Perplexity) can actually reach a live product URL before you go looking for subtler causes.
That crawler check is only the first branch of a larger decision tree, though; missing products are far more often a data-gap problem than a blocking one. Walk the full diagnosis in order in why you're not in ChatGPT shopping, and return to how AI shopping agents choose products for the selection signals that decide the outcome once the crawler can see you.