# AI Crawlers, robots.txt & llms.txt for Stores

> Blocking AI crawlers in robots.txt deletes the supplemental page data agents read beyond your feed. Here's the commerce allow-list (GPTBot, OAI-SearchBot, Google-Extended, PerplexityBot) and what llms.txt adds.

<!-- Blocking AI crawlers in robots.txt hides the page data agents read beyond your feed. Here's the allow-list and what llms.txt adds. -->
<!-- Canonical: https://agentmint.net/ai-crawlers-robots-llms-txt/ · By Abdalsalaam Halawa · Published 2026-07-07 -->
<!-- Evidence tags below are load-bearing: <Claim type="spec-fact|reported|hypothesis"> marks how each factual claim is sourced. -->

<KeyTakeaways
  items={[
    "AI shopping agents read two things about you: your structured product feed and your live pages. robots.txt is what decides whether they can read the second one.",
    "Whether to allow the AI crawlers is a selection decision, not a bandwidth one: block them and you can remove the page-level detail an agent would otherwise read about your products.",
    "The commerce allow-list is a specific set of real user-agent tokens, each owned by a named engine: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), Google-Extended (Google), PerplexityBot and Perplexity-User (Perplexity).",
    "Crawler access and your product feed are two separate pipes. Opening the crawlers does not replace a feed, and a feed does not make the crawlers unnecessary.",
    "llms.txt is an emerging, proposed convention worth shipping as low-cost insurance, but there is no confirmed evidence yet that major shopping agents read it when choosing products.",
  ]}
/>

## What each AI crawler actually does

Before you can decide who to let in, it helps to know that the AI crawlers do two different jobs, and the difference is what matters for a store. Some fetch and index your pages so an engine can *surface and cite* you later; others are a *live fetch* triggered the moment a user's request reaches your URL. We will not rehearse robots.txt grammar here (the syntax is owned by each engine's own documentation, which we link) because the useful question for a merchant is not how to write a directive but which of these agents you want reading your catalog and why.

Take the three engines that dominate agentic shopping today.

<Claim type="spec-fact" source="OpenAI, Bots documentation" href="https://developers.openai.com/api/docs/bots">OpenAI documents several distinct agents, including: GPTBot, which crawls content to train its models; OAI-SearchBot, which surfaces and links sites inside ChatGPT search; and ChatGPT-User, the fetch triggered when a ChatGPT user's own request reaches your page.</Claim> That split is the whole reason "should I allow AI crawlers?" has no single yes/no answer: GPTBot is mostly about training, while OAI-SearchBot and ChatGPT-User are the ones that decide whether a shopper sees you inside ChatGPT.

<Claim type="spec-fact" source="Google, Google's common crawlers" href="https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers">Google-Extended is not itself a crawler but a robots.txt control token that governs whether content Google has already crawled may be used to improve Gemini Apps and Vertex AI generative models, and toggling it does not change how a site ranks in Google Search.</Claim> For a store, that means the Google-Extended decision is purely an AI-usage decision, cleanly separable from your SEO.

<Claim type="spec-fact" source="Perplexity, Bots documentation" href="https://docs.perplexity.ai/guides/bots">Perplexity documents PerplexityBot, which indexes pages so they can be surfaced and cited in answers, and Perplexity-User, the fetch triggered by a user's live query.</Claim>

<Claim type="spec-fact" source="Anthropic, Claude Help Center" href="https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler">Anthropic documents three bots: ClaudeBot, which collects web content that can contribute to model training; Claude-User, the fetch triggered when someone asks Claude a question; and Claude-SearchBot, which crawls pages to improve the quality of Claude's search results.</Claim> Claude-SearchBot is the one that matters for selection: it is Anthropic's equivalent of OAI-SearchBot and PerplexityBot, the crawler behind whether Claude surfaces and cites a page at all.

Our own site ships the same allow-list for these plus Anthropic's ClaudeBot, Claude-SearchBot and Claude-User (an older `anthropic-ai` token still shows up in some third-party lists, but it is not part of Anthropic's current documented set above), Apple's Applebot-Extended, Amazonbot, and Common Crawl's CCBot. You can read the exact file at [agentmint.net/robots.txt](/robots.txt), because a publication whose purpose is to be read by agents should practise what it preaches.

Each bot-documentation link above was re-verified on 2026-07-08 and the per-token descriptions still match the operators' docs; the dated, token-by-token reference for every operator lives in [the AI shopping crawlers reference](/reference/ai-shopping-crawlers/).

## The allow-list decision

For a store whose goal is to be found, quoted, and bought through AI agents, the sensible default is to allow the shopping and answer crawlers, and to treat *blocking* as the exception that needs a reason. The tokens above are your allow-list; the decision is which, if any, you deliberately close.

The one nuance worth spending thought on is training versus surfacing. Because OpenAI separates GPTBot (training) from OAI-SearchBot and ChatGPT-User (surfacing and live fetch), a merchant who is wary of feeding model training can, in principle, decline the training crawler while still letting the surfacing agents in. <Claim type="hypothesis">For most stores that middle ground is the pragmatic choice: you keep every path that puts you in front of a shopper and give up only the training use you were never compensated for.</Claim>

<Callout variant="warning" title="The block you didn't mean to ship">
  The most common way stores lock out AI agents is not a line in robots.txt at all. It's a web application firewall, a CDN "bot manager," or a "block AI bots" plugin quietly rejecting these user agents at the edge. If being selected by agents matters to you, audit those layers too: a crawler that gets a 403 is just as blocked as one you disallowed on purpose.
</Callout>

## What you lose by blocking

Blocking the AI crawlers rarely makes a store vanish outright, which is exactly why the cost is easy to miss. The loss is quieter: <Claim type="hypothesis">closing the crawlers removes the supplemental page data an agent reads beyond your structured feed: the fuller description copy, specification details, review context, and availability nuance that live on your product page but never make it into a feed row.</Claim> An agent that can only see your feed is choosing from a thinner picture of your product than one that can also read the page.

<Illustrative>
  Two stores sell the same weighted sleep mask and both submit a clean feed, so both are eligible to appear. Store A allows the crawlers, so when a shopper asks an agent for "a breathable organic sleep mask for side sleepers," the agent can read the product page's fabric notes, the side-sleeper fit guidance, and forty reviews mentioning breathability. Store B blocks the crawlers, so the agent sees only the feed's title, price, and availability. When the query hinges on detail that lives on the page, Store A is the one with something to match against. This is a worked illustration of the trade-off, not a measured result.
</Illustrative>

## Crawler access vs your feed: two separate pipes

It is tempting to think a good product feed makes crawler access optional, or that open crawlers make a feed redundant. Neither is true, because they are two different pipes into an agent. Your [product feed](/make-product-feed-ai-readable/) is the structured, merchant-controlled channel (titles, price, availability, GTINs) delivered to an engine's commerce surface. Crawler access is the unstructured channel: the live page an agent can read for everything the feed leaves out.

The Google-Extended case makes the separation concrete. Because it is a usage-control token rather than a crawler, opening or closing it changes only whether your already-indexed content feeds Google's generative models. It touches neither your Merchant Center feed nor your search ranking. The lesson generalizes: audit the feed pipe and the crawler pipe independently, because a gap in one is invisible from the other. If your feed itself is thin, that is a separate fix: start with [making your product feed AI-readable](/make-product-feed-ai-readable/) and [the product schema agents trust](/product-schema-for-ai-shopping/).

## llms.txt for stores

<Claim type="spec-fact" source="llms.txt proposal, llmstxt.org" href="https://llmstxt.org">llms.txt is a proposed convention, published at llmstxt.org, for a root-level `/llms.txt` Markdown file that offers a large language model a curated, easy-to-parse map of a site's most important content.</Claim> The idea is to hand an agent a clean index instead of making it infer structure from your navigation. Our site ships a generated [/llms.txt](/llms.txt) as a worked example; the format spec lives at llmstxt.org and we link out to it rather than reproduce it.

Here is the honest part, because this is where the hype outruns the evidence. <Claim type="hypothesis">There is no confirmed public evidence yet that ChatGPT, Gemini, or Perplexity read llms.txt when choosing products, so treat it as low-cost insurance for a future many expect, not as a ranking lever that moves selection today.</Claim> Shipping one costs almost nothing and can only help if adoption arrives; building a strategy around it would be betting on behavior no engine has documented. One engine has now answered directly, and the answer is negative. <Claim type="spec-fact" source="Google, Optimizing your website for generative AI features on Google Search" href="https://developers.google.com/search/docs/fundamentals/ai-optimization-guide">Google's guide to optimizing for generative AI features, updated in June 2026, states that llms.txt files are not needed to appear in Google Search and that Google Search ignores them, so maintaining one neither helps nor harms a site's visibility or rankings there.</Claim> Re-checked 2026-07-08: we found no documentation from OpenAI, Perplexity, Anthropic, or Microsoft that any shopping agent reads `/llms.txt` when selecting products.

## Diagnose a store that's been blocked

If your products are missing from an agent's results, an accidentally-closed crawler is one of the first things to rule out: a stray `Disallow`, a bot-manager rule, or a CDN returning 403s to these user agents. Check that the surfacing crawlers for your target engine (OAI-SearchBot and ChatGPT-User for ChatGPT, PerplexityBot for [Perplexity](/perplexity-shopping-optimization/)) can actually reach a live product URL before you go looking for subtler causes.

That crawler check is only the first branch of a larger decision tree, though; missing products are far more often a data-gap problem than a blocking one. Walk the full diagnosis in order in [why you're not in ChatGPT shopping](/store-not-in-chatgpt-shopping/), and return to [how AI shopping agents choose products](/how-ai-agents-choose-products/) for the selection signals that decide the outcome once the crawler can see you.