You keep an origin healthy while staying open to AI agents by caching the cheap, near-static surfaces agents read most and reserving your compute for the offer data that has to be fresh, not by blocking the agents that would have selected you.
Key takeaways
- A live, user-triggered agent fetch hits your origin at question time while a shopper waits, so the cost of being open is real, but the answer is caching, not blocking.
- Retail is one of the sectors AI crawlers hit hardest, and crawl-to-refer ratios can be extremely lopsided, so plan for the fetch load instead of being surprised by the bill.
- Edge-cache the near-static machine surfaces (llms.txt, Markdown mirrors, feeds, mostly-static product pages) and keep price and availability on a short leash so an agent never quotes a stale offer.
- Access control (throttling unverified spoofers) is not content variation (serving different content by user-agent, which is cloaking); allowlist the verified agents and rate-limit only the fakes.
Why agent traffic strains an origin differently
A scheduled search crawler and a live shopping agent load your server on opposite economics. A classic index crawler visits on its own timetable, so its cost is smooth and predictable: fetch once, and the engine can send you many shoppers off that one visit. A live agent fetch is the reverse. It fires the moment a real person asks a question, it is tied to that person waiting for an answer, and nothing guarantees the work sends a visitor back.
That second mode is growing fast. Cloudflare recorded "user action" crawling, meaning bots that simulate a human action rather than run a scheduled index, growing more than 15x across 2025, and more than 21x from the start of January to early December.ReportedCloudflare Radar 2025 Year in Review (2025-12-15)
The return on that traffic can be lopsided. Cloudflare reported that Anthropic's crawlers had the highest crawl-to-refer ratios it saw, reaching as high as 500,000 fetches for every visitor referred back, while Perplexity's were the lowest.ReportedCloudflare Radar 2025 Year in Review (2025-12-15) Read that as a range across engines, not a fixed toll: the point for a merchant is that "being crawled" and "being sent shoppers" are not the same size, so you should budget for the fetch load on its own terms.
Retail sits squarely in the blast radius. Retail and computer software were consistently the two sectors that drew the most AI crawler traffic, together just over 40% of all AI crawler activity Cloudflare observed.ReportedCloudflare Radar 2025 Year in Review (2025-12-15) If you sell online, you are in one of the sectors AI crawlers hit hardest, so this is your bill to plan for.
One caveat before you quote any of these numbers: Cloudflare Radar reports Cloudflare's own network traffic, which is a large first-party sample rather than a census of the whole internet. Treat the direction and shape of the trend as the signal, and hold the exact percentages loosely.
The cost is not hypothetical. 404 Media reported that Anthropic's ClaudeBot hit iFixit nearly a million times in a single day, thousands of requests a minute for several hours, after which iFixit blocked it in robots.txt.Reported404 Media, Jason Koebler (2024-07-24) Read the Docs reported a single crawler downloading 73 TB of zipped HTML in one month (roughly 10 TB of it in a single day) at a bandwidth cost of over $5,000; after it blocked the abusive crawlers, downloaded-file bandwidth fell about 75% (from roughly 800 GB/day to roughly 200 GB/day), which it estimated at about $50 a day, or $1,500 a month, in avoided cost.ReportedRead the Docs, Eric Holscher (2024-07-25)
Both operators reached for a block because the traffic was unmanaged, not because the agents were unwelcome in principle. This chapter is about the option they did not have time to build: making the surfaces agents read cheap enough that an open door stops hurting.
Scheduled crawl vs live user fetch
Every AI engine loads you in one of two modes, and the difference decides what you can safely cache. The first is the scheduled index crawl: a bot walks your pages on its own schedule to build or refresh an index, so its requests are periodic, predictable, and easy to serve from cache. The second is the live user fetch: the request that fires the instant a shopper's question reaches your URL, while the person waits for the agent to answer.
For selection, the live fetch is the one that counts, because a shopper is mid-decision when it happens. It is also the harder mode to reason about, since it can arrive at any moment and sometimes carries a unique query string. The reassuring part is that most of what a live fetch actually reads is still your ordinary, slow-changing pages, which are exactly the ones you can cache.
Which AI crawler user-agent token belongs to which mode is a table we already maintain, so we will not restate it here. See AI crawlers, robots.txt and llms.txt for the per-engine tokens (which are index crawlers, which are live user fetches) and how to verify each one.
What is cheap to serve, and what is not
Split your surfaces into two buckets before you touch a cache setting.
Cheap to serve are the near-static machine surfaces: your llms.txt, any Markdown mirrors of your pages, RSS or Atom feeds, and the mostly-static parts of a product page (the description, the specifications, the care instructions). These change slowly and look identical to every requester, so you can generate them once and serve the same bytes to thousands of fetches. They are also the dense, fact-heavy surfaces covered in information density for agents, which is the other half of this decision: pack them with verifiable facts, then make them cheap to serve.
Expensive, and not safely cacheable for long, is the live offer data: current price, stock status, and the delivery window for this shopper. This is the information that changes often and that an agent most needs to have correct.
The trap is the long tail. Wikimedia reported that at least 65% of its most resource-consuming traffic came from bots even though bots were only about 35% of pageviews, because crawlers bulk-read many low-popularity pages that human readers rarely open and that therefore miss the cache.ReportedWikimedia Diff, Mueller, Danis and Lavagetto (2025-04-01) It also reported that bandwidth for multimedia downloads grew 50% since January 2024, driven largely by automated scrapers rather than human readers.ReportedWikimedia Diff, Mueller, Danis and Lavagetto (2025-04-01) The lesson for a store: the requests that hurt are usually not the homepage, which is always warm in cache, but the thousands of deep product and variant URLs a crawler will read once and a shopper almost never will. Cache those on purpose, or they find your origin every time.
Edge-caching the machine surfaces
This is the part you hand to a developer, and it is deliberately short, because the caching directives themselves are documented by the CDN and web-server vendors and we link straight to them. The behavior you want is simple to state: serve the cached copy instantly, and refresh it in the background. Two mechanisms give you that.
Cloudflare serves stale content while it revalidates only if your origin's Cache-Control header includes stale-while-revalidate; when that is set, the first request after the cached copy expires triggers an asynchronous background revalidation and immediately receives the stale content marked with an UPDATING status, instead of blocking on the origin.Spec-factCloudflare, Cache revalidation On nginx, proxy_cache_use_stale with the updating parameter serves a stale cached response while that entry is being refreshed, which minimizes upstream requests (it is off by default); since version 1.11.10 nginx can also honor a stale-while-revalidate directive from the upstream, at lower priority than the proxy_cache_use_stale settings.Spec-factnginx, ngx_http_proxy_moduleThese are two separate implementations, not one feature: nginx's serve-stale-while-refreshing is not identical to Cloudflare's asynchronous UPDATING flow, so test the one you actually run rather than assuming they behave the same. If you want the edge to hold a copy longer than a browser does, Cloudflare's cache-control guidance is to send max-age together with stale-while-revalidate from the origin and set a separate Edge Cache TTL in a Cache Rule, and it warns against combining s-maxage with stale-while-revalidate.Spec-factCloudflare, Cache-Control
A minimal starting point for one near-static surface:
The directives and their behavior above are documented and exact; the specific numbers (one hour fresh, then up to one day of stale-while-revalidate) are an illustrative editorial choice, not a spec requirement, so set them to match how often that surface really changes.Hypothesis (our analysis)For anything past this, follow the vendor docs rather than a recipe from us. The two Cloudflare pages and the nginx page linked above are the source of truth, and they move.
The freshness tradeoff: never serve price and availability long-stale
Stale-while-revalidate is a gift for low-churn surfaces and a liability for offer data. Our stance is a hard line: cache descriptions, specs, and llms.txt generously, but keep price and availability on a short TTL, or bypass the cache for them entirely, so an agent can never read and then quote an offer you no longer honor.Hypothesis (our analysis)
The reason is selection, not correctness alone. An agent that surfaces a price you have since raised, or shows "in stock" for something you have sold out of, hands the shopper a bad checkout, and that mismatch is exactly the kind of thing an engine learns to stop trusting you for. The same freshness discipline you apply to your product feed has to apply to the offer fields on the live page, because an agent may read either one. When your page and your feed disagree on price, you have given the agent a reason to doubt both.
Allowlist verified bots; rate-limit only unverified spoofers
Two different levers get confused here, and keeping them apart is the whole point of this section.
Access control decides who gets in and how fast: allow, deny, or throttle a request based on the verified identity of the client. Content variation decides what a given client sees: serving one version of a page to one user-agent and a different version to another. That second lever, when you swap the actual content by user-agent to manipulate what an engine indexes, is cloaking, and it is covered in information density for agents. Throttling a request is not content variation: rate-limiting changes how much a client can fetch, never what the page says, so it is not cloaking. Say that plainly to anyone who worries the two are the same thing.
Our recommended stance is narrow. Allowlist the verified shopping and answer agents and let them through, and reserve rate limits for traffic you cannot verify: clients that claim to be a known agent but fail that engine's published verification method. Rate-limiting the verified agents that select you would be self-defeating.Hypothesis (our analysis) How to verify each engine's bots (their published IP ranges and reverse-DNS checks) is in AI crawlers, robots.txt and llms.txt.
For context on how far operators under real pressure have gone, look at what some open-source projects reached for. Anubis is an MIT-licensed reverse proxy by Xe Iaso that issues a SHA-256 proof-of-work challenge to browser-like clients (those whose user-agent contains "Mozilla") before the request reaches the backend, and its documented adopters include GNOME's GitLab, the Linux kernel mailing-list archives, FFmpeg, Wine, UNESCO, FreeCAD, ScummVM, and Duke.ReportedWikipedia, Anubis (software) (accessed 2026-07-08) That is a response to scraping abuse by projects without a budget to absorb it, and it is context here, not our advice. For a store that wants to be selected, a proof-of-work wall that challenges every browser-like client is the opposite of the goal: the aim is to make the door cheap to hold open, not to close it.
What to measure
Four numbers tell you whether you are affording openness or quietly paying for it.
- Cache-hit ratio on the machine surfaces. Measure it per surface (llms.txt, Markdown mirrors, feeds, product pages) at your CDN or in your access logs. A low hit rate on a near-static surface means you are serving from the origin what you should be serving from the edge.
- Origin egress and bandwidth, tracked before and after you add caching. This is the bill the Read the Docs and Wikimedia numbers describe, and the clearest sign your caching is working.
- Crawl-to-refer ratio per agent. Compare requests from each engine's bots against the visits or conversions you can attribute to that engine. A very lopsided ratio is not a reason to block; it tells you which agents are worth the most caching effort and which you are subsidizing.
- 403s (and any other blocks) returned to verified agents. This is the block you did not mean to ship: a WAF rule, a bot manager, or a plugin quietly rejecting an agent you wanted. Grep your access logs for these tokens and confirm they get 200s on a live product URL.
If verified agents are being blocked and your products are missing from an engine's results, start the full diagnosis in why your store is not in ChatGPT shopping, then step back to how AI shopping agents choose products for the selection signals that decide the outcome once your origin is healthy and open.