# AI Shopping Crawler Reference: Every User-Agent Token > Documented AI shopping crawler tokens: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); Googlebot, Storebot-Google, Google-Extended (Google); ClaudeBot, Claude-User, Claude-SearchBot (Anthropic); PerplexityBot, Perplexity-User (Perplexity). This page is the exhaustive, dated reference: the exact user-agent tokens named in each operator's own documentation, what each one does, and whether it honors robots.txt. For the concepts behind these tokens (what robots.txt and llms.txt are, why blocking a crawler is a selection decision) see [AI crawlers, robots.txt and llms.txt for stores](/ai-crawlers-robots-llms-txt/). For the strategy of which agent engines to prioritize, see the [AI shopping platforms hub](/platforms/). Every entry below was fetched and verified on 2026-07-08; check the "Last verified" logic on [platform pages](/platforms/) for how often these are expected to move. ## The reference table Fifteen to eighteen rows can't hold every token every operator has ever shipped, so this table prioritizes shopping relevance: the crawlers that decide whether a store's pages get read, cited, or trained on by an agent a shopper might use. A few less central tokens (OpenAI's ad-safety bot, two of Meta's five) are covered in prose below the table instead of as rows. ## OpenAI's shopping-relevant crawlers OpenAI documents GPTBot as the crawler used to train its generative foundation models, and its own guidance is explicit that disallowing GPTBot means a site's content should not be used in that training, not that the site is opting out of search visibility. That distinction matters because OpenAI splits training from surfacing across separate tokens. OAI-SearchBot is the one that indexes pages for citation, and OpenAI states plainly that a site opted out of OAI-SearchBot "will not be shown in ChatGPT search answers", which makes it the token to protect if being cited in ChatGPT search matters more than the training question. ChatGPT-User is a different kind of token entirely. ChatGPT-User is the fetch OpenAI triggers the moment an actual ChatGPT user's own request reaches a page, and because a person, not a schedule, initiated it, OpenAI notes that "robots.txt rules may not apply". A fourth, narrower token rounds out the set: OAI-AdsBot validates that a ChatGPT ad's landing page is safe to send traffic to, and OpenAI states it is not used for AI training. All four are verifiable by IP through OpenAI's published JSON files (searchbot.json, gptbot.json, chatgpt-user.json, and adsbot.json at openai.com), and OpenAI states that robots.txt changes take about 24 hours to propagate across its crawlers. ## Google's shopping-relevant crawlers Google's set is the widest of the four, and it includes the single most on-topic entry on this whole page. Googlebot is Google's standard crawler, covering Search, the Shopping tab, Images, Video, and News indexing, and it honors robots.txt. Storebot-Google is the crawler Google names specifically for Shopping: Google's own documentation states it can "affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping)," and it honors robots.txt. If a merchant reads only one row in this reference, this is the one worth reading twice. Google-Extended gets confused with a general AI opt-out more often than any other token here, and Google's own wording draws the boundary precisely. Google-Extended does not have a separate HTTP request user agent string of its own; it is a robots.txt-only control token scoped to whether already-crawled content may train or ground Gemini Apps and Vertex AI, and Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". What Google's text does not say is equally important: it scopes Google-Extended to Gemini Apps and Vertex AI grounding specifically, not to AI Mode inclusion, so a merchant should not treat toggling it as a lever over AI Mode results. Google-Agent is the newest addition, introduced by Google's own account around March 2026. Google-Agent is used by agents hosted on Google's own infrastructure to navigate the web and perform actions on a user's request, and Google names Project Mariner as an example. Google groups Google-Agent among its user-triggered fetchers, the category of crawlers that generally does not honor robots.txt because a specific user request triggered the fetch. Project Mariner has been described elsewhere as capable of multi-step shopping and product-comparison tasks, and it is reasonable to expect Google-Agent traffic to include that kind of browsing, but this is our inference: Google's own token description does not use the word "shopping," so do not treat that specific link as confirmed by the primary source. ## Anthropic's (Claude) shopping-relevant crawlers Anthropic documents three bots on a single support page, and the split maps directly onto the training-versus-surfacing pattern seen at OpenAI. ClaudeBot collects web content that can contribute to model training, and it honors robots.txt. Claude-User is the live fetch behind an actual conversation: Anthropic states that "when individuals ask questions to Claude, it may access websites using a Claude-User agent," and this token also honors robots.txt. Claude-SearchBot is the one that decides whether Claude cites a page at all. Anthropic describes Claude-SearchBot as a crawler that "navigates the web to improve search result quality for users," and it warns that blocking it "prevents our system from indexing your content for search optimization". Two smaller notes worth carrying into a robots.txt audit: Anthropic's current documentation makes no mention of the older `anthropic-ai` token some third-party lists still carry, and Anthropic supports the non-standard `Crawl-delay` directive alongside IP verification at claude.com/crawling/bots.json. ## Perplexity's shopping-relevant crawlers PerplexityBot is described as "designed to surface and link websites in search results on Perplexity," and Perplexity states that "PerplexityBot only crawls content in compliance with robots.txt". Perplexity-User "supports user actions within Perplexity," visiting a page when a user's own question requires it, and Perplexity states that "this fetcher generally ignores robots.txt rules" because the fetch follows a specific user request. Perplexity's help center also states that a prior feature letting users prompt it to summarize robots.txt-blocked URLs anyway "has been disabled to prevent misuse," a dated clarification of how strictly Perplexity now enforces the block. ## Amazon's crawlers Amazon documents three bots, not one, each with a distinct job, all covered by one general robots.txt policy. Amazon's crawlers honor the Robots Exclusion Protocol, and Amazon may rely on a cached copy of a site's robots.txt file up to 30 days old if a fresh fetch fails. Amazonbot is used to improve Amazon's products and services and may be used to train Amazon AI models. Amzn-SearchBot is used to improve search experiences in Amazon's products and services, and Amazon states it "does not crawl content for generative AI model training"; it carries one additional fallback rule beyond the general policy, that if a robots.txt file does not name it explicitly, it follows whatever rules apply to Amazon's other search bots. Amzn-User supports user actions, such as responding to Alexa queries that require up-to-date information, and does not crawl for AI training. Third-party analysis (for example, dejan.ai) has linked Amzn-SearchBot to Rufus, Amazon's shopping assistant, but the word "Rufus" does not appear anywhere on Amazon's own documentation page. Treat that specific link as third-party speculation, not a fact confirmed by the primary source. Amazon's crawlers also honor page-level `noarchive` (exclude from AI training), `noindex`/`none`, and `rel=nofollow` directives. ## Meta's crawlers Meta documents five bots, and only three carry enough shopping relevance for the table above. Meta-WebIndexer "navigates the web to improve Meta AI search result quality for users". Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly," and it respects robots.txt with up to 24-hour propagation. Meta-ExternalFetcher "fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities," and Meta states it "may bypass robots.txt because it performs fetches that were requested by the user". The other two are worth knowing but not full rows. facebookexternalhit generates link previews when content is shared inside Meta's apps, and Meta-ExternalAds "crawls the web for use cases such as improving advertising and other business-related products". Meta's documentation names no shopping-specific crawler token: Facebook and Instagram Shops and Meta AI's shopping features are not tied to a distinct token in this source, a confirmed null result rather than a gap in our research. ## Apple and DuckDuckGo, briefly Two more operators publish a documented crawler relevant to AI answers, though neither carries enough shopping-specific weight for a full table row. Applebot powers Siri, Spotlight, and Safari search, honors robots.txt, and Apple states that "if robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions". Applebot-Extended is a training-opt-out-only token with no separate HTTP user agent of its own; Apple states it does not crawl webpages, and is only used to determine how to use the data Applebot already crawled, so pages that disallow it can still appear in Spotlight, Siri, and Safari search. DuckAssistBot "crawls pages in real-time for our AI-assisted answers, which prominently cite their sources," is not used for AI training, and an opt-out via robots.txt takes about 72 hours to apply. Microsoft is named in this site's own mission, and the honest answer here is a null result worth stating plainly. Bing's own webmaster documentation lists only bingbot, adidxbot, BingPreview, MicrosoftPreview, and BingVideoPreview among its crawlers. Microsoft's Copilot Studio guidance for generative AI on public websites likewise names no distinct Copilot crawler. Copilot's shopping-relevant answers run on Bing's existing index through bingbot; there is no separate token to allow or block specifically for Copilot. Some third-party lists cite a "CopilotBot," but no Microsoft-owned source names one, so treat that token as unconfirmed rather than real. ## Confirmed absent: tokens that do not exist A negative finding, stated plainly, is worth more than a guess: it stops someone else from fabricating a token. Four are worth recording here. - **Rakuten.** No documented, distinct AI-shopping crawler token exists anywhere in Rakuten's own materials. - **Walmart.** No documented, distinct crawler token for Walmart's AI shopping features exists anywhere in Walmart's own materials. - **TikTok Shop.** No official ByteDance or TikTok developer documentation for any crawler exists. The widely reported `Bytespider` token is sourced only to third parties (for example, Cloudflare Radar and DataDome), never to a ByteDance-owned page, so treat any `Bytespider`-to-training-data claim as unconfirmed. TikTok's own robots.txt currently blocks a long list of AI bots by name, which is itself the one verifiable, on-the-record fact in this space. - **Amazon's "Rufus" as a distinct token.** As covered above, no crawler named for Rufus exists; Amazon's own documentation never uses the name. ## Two robots.txt snippets Standard `User-agent` / `Allow` / `Disallow` syntax, no vendor-specific extensions beyond Anthropic's optional `Crawl-delay`. Adjust to your own catalog and legal posture before shipping either. **A permissive allow-list**, letting every documented, shopping-relevant bot in this reference read the site: ``` User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-AdsBot Allow: / User-agent: Googlebot Allow: / User-agent: Storebot-Google Allow: / User-agent: Google-Extended Allow: / User-agent: Google-Agent Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Amazonbot Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / User-agent: Meta-WebIndexer Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / User-agent: DuckAssistBot Allow: / ``` **A training-data-only block**, keeping every search, citation, and user-fetch bot open while closing only the tokens each operator documents as training crawlers: ``` User-agent: * Allow: / # Training crawlers: closed User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / # Search, citation, and live user-fetch bots: left open User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Googlebot Allow: / User-agent: Storebot-Google Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / User-agent: Meta-WebIndexer Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: Applebot Allow: / User-agent: DuckAssistBot Allow: / ```