AI Shopping Crawler Reference: Every User-Agent Token

Key takeaways

• This page is a dated reference, not analysis: every token below is named in an operator's own documentation, fetched and verified on 2026-07-08, with the exact purpose and robots.txt behavior each operator states.
• Four platforms get full detail because this site already covers them elsewhere: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Google (Googlebot, Storebot-Google, Google-Extended, Google-Agent), Anthropic (ClaudeBot, Claude-User, Claude-SearchBot), and Perplexity (PerplexityBot, Perplexity-User).
• Storebot-Google is the actual Google Shopping crawler and the single most on-topic entry here; Google-Extended only controls Gemini Apps and Vertex AI grounding, not AI Mode inclusion, a distinction Google's own docs draw explicitly.
• No Microsoft-owned source documents a distinct Copilot crawler: Copilot's shopping-relevant answers run on Bing's existing index through bingbot, and the 'CopilotBot' token some third-party lists cite has no primary source.
• Several widely assumed tokens do not exist in any operator's documentation: no distinct crawler for Rakuten, Walmart, or TikTok Shop, and no token literally named for Amazon's Rufus. Treat any source claiming otherwise as unverified.

This page is the exhaustive, dated reference: the exact user-agent tokens named in each operator's own documentation, what each one does, and whether it honors robots.txt. For the concepts behind these tokens (what robots.txt and llms.txt are, why blocking a crawler is a selection decision) see AI crawlers, robots.txt and llms.txt for stores. For the strategy of which agent engines to prioritize, see the AI shopping platforms hub. Every entry below was fetched and verified on 2026-07-08; check the "Last verified" logic on platform pages for how often these are expected to move.

The reference table

Fifteen to eighteen rows can't hold every token every operator has ever shipped, so this table prioritizes shopping relevance: the crawlers that decide whether a store's pages get read, cited, or trained on by an agent a shopper might use. A few less central tokens (OpenAI's ad-safety bot, two of Meta's five) are covered in prose below the table instead of as rows.

Token	Operator	Purpose	Robots.txt-controllable
GPTBot	OpenAI	Trains OpenAI's generative foundation models	Yes
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT search citations	Yes
ChatGPT-User	OpenAI	Live fetch when a ChatGPT user's request reaches a page	May not apply (user-triggered)
Googlebot	Google	Standard Search, Shopping tab, Images, Video, and News indexing	Yes
Storebot-Google	Google	The dedicated Google Shopping crawler, across all Shopping surfaces	Yes
Google-Extended	Google	Opt-out control for Gemini Apps and Vertex AI grounding only	Yes (control token, no separate crawl)
Google-Agent	Google	Powers agents on Google's own infrastructure acting on a user's request (for example, Project Mariner)	Generally no (user-triggered)
ClaudeBot	Anthropic	Collects web content that can contribute to training	Yes
Claude-User	Anthropic	Live fetch when someone asks Claude a question	Yes
Claude-SearchBot	Anthropic	Improves Claude's search results; decides if Claude cites a page	Yes
PerplexityBot	Perplexity	Surfaces and links sites in Perplexity search results	Yes
Perplexity-User	Perplexity	Live fetch when a user asks Perplexity a question	Generally no (user-triggered)
Amazonbot	Amazon	Improves Amazon products and services; may train Amazon AI models	Yes (site's robots.txt file cached up to 30 days)
Amzn-SearchBot	Amazon	Improves search experiences in Amazon products; not used for AI training	Yes (falls back to other search-bot rules if unnamed)
Amzn-User	Amazon	Supports user actions, such as Alexa queries needing live data	Not stated by source
Meta-WebIndexer	Meta	Improves Meta AI search result quality	Not stated by source
Meta-ExternalAgent	Meta	Trains foundation AI models and indexes content for products	Yes (up to 24-hour propagation)
Meta-ExternalFetcher	Meta	User-requested fetch supporting agentic AI capabilities	May bypass (user-triggered)

OpenAI's shopping-relevant crawlers

OpenAI documents GPTBot as the crawler used to train its generative foundation models, and its own guidance is explicit that disallowing GPTBot means a site's content should not be used in that training, not that the site is opting out of search visibilitySpec-factOpenAI, Bots documentation. That distinction matters because OpenAI splits training from surfacing across separate tokens. OAI-SearchBot is the one that indexes pages for citation, and OpenAI states plainly that a site opted out of OAI-SearchBot "will not be shown in ChatGPT search answers"Spec-factOpenAI, Bots documentation, which makes it the token to protect if being cited in ChatGPT search matters more than the training question.

ChatGPT-User is a different kind of token entirely. ChatGPT-User is the fetch OpenAI triggers the moment an actual ChatGPT user's own request reaches a page, and because a person, not a schedule, initiated it, OpenAI notes that "robots.txt rules may not apply"Spec-factOpenAI, Bots documentation. A fourth, narrower token rounds out the set: OAI-AdsBot validates that a ChatGPT ad's landing page is safe to send traffic to, and OpenAI states it is not used for AI trainingSpec-factOpenAI, Bots documentation. All four are verifiable by IP through OpenAI's published JSON files (searchbot.json, gptbot.json, chatgpt-user.json, and adsbot.json at openai.com), and OpenAI states that robots.txt changes take about 24 hours to propagate across its crawlers.

Google's shopping-relevant crawlers

Google's set is the widest of the four, and it includes the single most on-topic entry on this whole page. Googlebot is Google's standard crawler, covering Search, the Shopping tab, Images, Video, and News indexing, and it honors robots.txtSpec-factGoogle, Google's common crawlers. Storebot-Google is the crawler Google names specifically for Shopping: Google's own documentation states it can "affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping)," and it honors robots.txtSpec-factGoogle, Google's common crawlers. If a merchant reads only one row in this reference, this is the one worth reading twice.

Google-Extended gets confused with a general AI opt-out more often than any other token here, and Google's own wording draws the boundary precisely. Google-Extended does not have a separate HTTP request user agent string of its own; it is a robots.txt-only control token scoped to whether already-crawled content may train or ground Gemini Apps and Vertex AI, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search"Spec-factGoogle, Google's common crawlers. What Google's text does not say is equally important: it scopes Google-Extended to Gemini Apps and Vertex AI grounding specifically, not to AI Mode inclusion, so a merchant should not treat toggling it as a lever over AI Mode results.

Google-Agent is the newest addition, introduced by Google's own account around March 2026. Google-Agent is used by agents hosted on Google's own infrastructure to navigate the web and perform actions on a user's request, and Google names Project Mariner as an exampleSpec-factGoogle, Google-Agent. Google groups Google-Agent among its user-triggered fetchers, the category of crawlers that generally does not honor robots.txt because a specific user request triggered the fetchSpec-factGoogle, Google's user-triggered fetchers. Project Mariner has been described elsewhere as capable of multi-step shopping and product-comparison tasks, and it is reasonable to expect Google-Agent traffic to include that kind of browsing, but this is our inference: Google's own token description does not use the word "shopping," so do not treat that specific link as confirmed by the primary source.Hypothesis (our analysis)

Anthropic's (Claude) shopping-relevant crawlers

Anthropic documents three bots on a single support page, and the split maps directly onto the training-versus-surfacing pattern seen at OpenAI. ClaudeBot collects web content that can contribute to model training, and it honors robots.txtSpec-factAnthropic, Claude Help Center. Claude-User is the live fetch behind an actual conversation: Anthropic states that "when individuals ask questions to Claude, it may access websites using a Claude-User agent," and this token also honors robots.txtSpec-factAnthropic, Claude Help Center.

Claude-SearchBot is the one that decides whether Claude cites a page at all. Anthropic describes Claude-SearchBot as a crawler that "navigates the web to improve search result quality for users," and it warns that blocking it "prevents our system from indexing your content for search optimization"Spec-factAnthropic, Claude Help Center. Two smaller notes worth carrying into a robots.txt audit: Anthropic's current documentation makes no mention of the older anthropic-ai token some third-party lists still carry, and Anthropic supports the non-standard Crawl-delay directive alongside IP verification at claude.com/crawling/bots.jsonSpec-factAnthropic, Claude Help Center.

Perplexity's shopping-relevant crawlers

PerplexityBot is described as "designed to surface and link websites in search results on Perplexity," and Perplexity states that "PerplexityBot only crawls content in compliance with robots.txt"Spec-factPerplexity, Crawlers documentation. Perplexity-User "supports user actions within Perplexity," visiting a page when a user's own question requires it, and Perplexity states that "this fetcher generally ignores robots.txt rules" because the fetch follows a specific user requestSpec-factPerplexity, Crawlers documentation.

Perplexity's help center also states that a prior feature letting users prompt it to summarize robots.txt-blocked URLs anyway "has been disabled to prevent misuse," a dated clarification of how strictly Perplexity now enforces the blockReportedPerplexity, Help Center (2026-07-08).

Amazon's crawlers

Amazon documents three bots, not one, each with a distinct job, all covered by one general robots.txt policy. Amazon's crawlers honor the Robots Exclusion Protocol, and Amazon may rely on a cached copy of a site's robots.txt file up to 30 days old if a fresh fetch failsSpec-factAmazon, Amazonbot documentation. Amazonbot is used to improve Amazon's products and services and may be used to train Amazon AI modelsSpec-factAmazon, Amazonbot documentation. Amzn-SearchBot is used to improve search experiences in Amazon's products and services, and Amazon states it "does not crawl content for generative AI model training"; it carries one additional fallback rule beyond the general policy, that if a robots.txt file does not name it explicitly, it follows whatever rules apply to Amazon's other search botsSpec-factAmazon, Amazonbot documentation. Amzn-User supports user actions, such as responding to Alexa queries that require up-to-date information, and does not crawl for AI trainingSpec-factAmazon, Amazonbot documentation.

Third-party analysis (for example, dejan.ai) has linked Amzn-SearchBot to Rufus, Amazon's shopping assistant, but the word "Rufus" does not appear anywhere on Amazon's own documentation page. Treat that specific link as third-party speculation, not a fact confirmed by the primary source. Amazon's crawlers also honor page-level noarchive (exclude from AI training), noindex/none, and rel=nofollow directivesSpec-factAmazon, Amazonbot documentation.

Meta's crawlers

Meta documents five bots, and only three carry enough shopping relevance for the table above. Meta-WebIndexer "navigates the web to improve Meta AI search result quality for users"Spec-factMeta, Crawler documentation. Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly," and it respects robots.txt with up to 24-hour propagationSpec-factMeta, Crawler documentation. Meta-ExternalFetcher "fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities," and Meta states it "may bypass robots.txt because it performs fetches that were requested by the user"Spec-factMeta, Crawler documentation.

The other two are worth knowing but not full rows. facebookexternalhit generates link previews when content is shared inside Meta's apps, and Meta-ExternalAds "crawls the web for use cases such as improving advertising and other business-related products"Spec-factMeta, Crawler documentation. Meta's documentation names no shopping-specific crawler token: Facebook and Instagram Shops and Meta AI's shopping features are not tied to a distinct token in this sourceSpec-factMeta, Crawler documentation, a confirmed null result rather than a gap in our research.

Apple and DuckDuckGo, briefly

Two more operators publish a documented crawler relevant to AI answers, though neither carries enough shopping-specific weight for a full table row. Applebot powers Siri, Spotlight, and Safari search, honors robots.txt, and Apple states that "if robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions"Spec-factApple, Applebot documentation. Applebot-Extended is a training-opt-out-only token with no separate HTTP user agent of its own; Apple states it does not crawl webpages, and is only used to determine how to use the data Applebot already crawled, so pages that disallow it can still appear in Spotlight, Siri, and Safari searchSpec-factApple, Applebot documentation.

DuckAssistBot "crawls pages in real-time for our AI-assisted answers, which prominently cite their sources," is not used for AI training, and an opt-out via robots.txt takes about 72 hours to applySpec-factDuckDuckGo, DuckAssistBot.

Confirmed absent: tokens that do not exist

A negative finding, stated plainly, is worth more than a guess: it stops someone else from fabricating a token. Four are worth recording here.

• Rakuten. No documented, distinct AI-shopping crawler token exists anywhere in Rakuten's own materials.
• Walmart. No documented, distinct crawler token for Walmart's AI shopping features exists anywhere in Walmart's own materials.
• TikTok Shop. No official ByteDance or TikTok developer documentation for any crawler exists. The widely reported Bytespider token is sourced only to third parties (for example, Cloudflare Radar and DataDome), never to a ByteDance-owned page, so treat any Bytespider-to-training-data claim as unconfirmed. TikTok's own robots.txt currently blocks a long list of AI bots by nameSpec-factTikTok, robots.txt, which is itself the one verifiable, on-the-record fact in this space.
• Amazon's "Rufus" as a distinct token. As covered above, no crawler named for Rufus exists; Amazon's own documentation never uses the name.

Two robots.txt snippets

Standard User-agent / Allow / Disallow syntax, no vendor-specific extensions beyond Anthropic's optional Crawl-delay. Adjust to your own catalog and legal posture before shipping either.

A permissive allow-list, letting every documented, shopping-relevant bot in this reference read the site:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-AdsBot
Allow: /

User-agent: Googlebot
Allow: /
User-agent: Storebot-Google
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Google-Agent
Allow: /

User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /

User-agent: Amazonbot
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /

User-agent: Meta-WebIndexer
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /

User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /

User-agent: DuckAssistBot
Allow: /

A training-data-only block, keeping every search, citation, and user-fetch bot open while closing only the tokens each operator documents as training crawlers:

User-agent: *
Allow: /

# Training crawlers: closed
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /

# Search, citation, and live user-fetch bots: left open
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Storebot-Google
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Amzn-SearchBot
Allow: /
User-agent: Amzn-User
Allow: /
User-agent: Meta-WebIndexer
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /
User-agent: Applebot
Allow: /
User-agent: DuckAssistBot
Allow: /

Key takeaways

• Storebot-Google, OAI-SearchBot, Claude-SearchBot, and PerplexityBot are the four tokens that most directly decide whether a store gets surfaced or cited; treat them as the ones to protect first.
• GPTBot, ClaudeBot, Amazonbot, Meta-ExternalAgent, Google-Extended, and Applebot-Extended are training-scoped opt-outs a merchant can close without losing surfacing or citation.
• User-triggered fetchers (ChatGPT-User, Perplexity-User, Google-Agent, Meta-ExternalFetcher) generally are not bound by robots.txt because a real person's request triggered the fetch, so blocking them rarely works as intended.
• No primary source documents a Copilot-specific crawler, a Rakuten or Walmart AI-shopping crawler, or a token literally named for Amazon's Rufus. Treat any claim to the contrary as unverified until an operator publishes one.
• This is a reference snapshot from 2026-07-08. Crawler rosters change: recheck the primary sources linked throughout before copying a robots.txt block verbatim into production.