# Edge worker for machine surfaces > Serve llms.txt and .md mirrors from a Cloudflare Worker two ways: the CDN path (cf.cacheEverything) honors the origin's stale-while-revalidate; the Cache API does not, so it revalidates in the background via ctx.waitUntil. This blueprint gives you two ready-to-deploy Cloudflare Worker configurations for caching your machine surfaces (llms.txt, .md mirrors, RSS or Atom feeds) at the edge, and the choice between them turns on one fact: the Workers Cache API does not honor `stale-while-revalidate`, so only the CDN path gives you true directive-driven stale serving. ## Why there are two variants A Worker is a small piece of code that runs at Cloudflare's edge in front of your origin, so it is the natural place to cache low-churn machine surfaces and keep agent refetches off your origin. The question is how you get stale-while-revalidate behavior, where the edge returns the cached copy instantly and refreshes it out of band, which is exactly the pattern that protects an origin from crawler bursts without ever making an agent wait. A Workers module exports a default object with an `async fetch(request, env, ctx)` handler, and `ctx.waitUntil(promise)` extends the Worker's lifetime so it can finish background work after the response has already been returned to the client. The load-bearing constraint is what these two Worker APIs do with the `stale-while-revalidate` directive. The Workers Cache API (`caches.default`) does not honor the `stale-while-revalidate` or `stale-if-error` directives. A Worker that caches through that API therefore cannot get automatic stale serving from a header; once its own freshness window lapses, the next `cache.match` is simply a miss. To get real directive-driven stale serving you have to go through Cloudflare's CDN cache instead. Cloudflare's CDN serves stale content during revalidation only when the origin's `Cache-Control` includes `stale-while-revalidate`; the first request after the fresh window triggers an asynchronous background revalidation and immediately returns the stale response with an `UPDATING` cache status. So the split is clean: Variant A rides the CDN path and lets the origin's header drive real SWR; Variant B stays inside the Cache API and rebuilds the stale-then-refresh behavior by hand. ## Variant A: CDN path (preferred for llms.txt and .md mirrors) ```js export default { async fetch(request, env, ctx) { // Origin sends: Cache-Control: max-age=3600, stale-while-revalidate=86400 // cacheEverything makes Cloudflare's CDN cache honor those directives: // serve stale + async-revalidate (UPDATING) after the fresh window. // The TTLs live in the ORIGIN headers (illustrative: 1h fresh / 24h stale). return fetch(request, { cf: { cacheEverything: true } }); } }; ``` The whole mechanism lives in one line and one origin header. `cf: { cacheEverything: true }` tells Cloudflare to run this response through the CDN cache, and the CDN cache is the layer that reads `stale-while-revalidate` and produces the `UPDATING` flow described above. Because the TTLs live in the origin's `Cache-Control` and not in the Worker, you tune freshness per surface by changing a header, not by redeploying code. We prefer this variant for llms.txt and .md mirrors because stale serving there is nearly free: those files change rarely, and an agent that reads a link index or a Markdown mirror one revalidation behind loses nothing. Variant A is also the smaller surface to maintain, since the freshness policy is data (a header) rather than logic. Note the header shape. Variant A pairs `max-age` with `stale-while-revalidate` rather than using `s-maxage`. Cloudflare's cache-control guidance notes that `s-maxage` implies `proxy-revalidate`, so a shared cache must not serve stale without revalidating, and it advises against combining `s-maxage` with `stale-while-revalidate`. That is why the SWR path in Variant A is expressed with `max-age`, while the manual path in Variant B (which has no SWR to combine with) uses `s-maxage` on its own. ## Variant B: Cache API, manual refresh (self-contained) ```js export default { async fetch(request, env, ctx) { if (request.method !== "GET") return fetch(request); const cache = caches.default; const cacheKey = new Request(request.url, request); const cached = await cache.match(cacheKey); if (cached) { // HIT: serve now, revalidate in the background. NOTE: the Cache API does // NOT honor stale-while-revalidate, so we orchestrate the refresh manually. ctx.waitUntil(refresh(cache, cacheKey, request)); return cached; } const response = await fetch(request); const toStore = new Response(response.body, response); toStore.headers.set("Cache-Control", "s-maxage=3600"); // illustrative TTL ctx.waitUntil(cache.put(cacheKey, toStore.clone())); return toStore; } }; async function refresh(cache, cacheKey, request) { const response = await fetch(request); const toStore = new Response(response.body, response); toStore.headers.set("Cache-Control", "s-maxage=3600"); // illustrative TTL await cache.put(cacheKey, toStore); } ``` This variant reconstructs the stale-then-refresh pattern that the Cache API will not give you for free. On a hit it returns the cached copy right away and schedules `refresh()` with `ctx.waitUntil`, so the fetch to your origin happens after the agent already has its response. The `waitUntil` handle is what keeps that background fetch from being cut off when the response returns. Two documented Cache API behaviors shape the code. `cache.match` and `cache.put` respect `Cache-Control` including `s-maxage`, and `cache.put` refuses to store responses to non-GET requests, `206` partial responses, responses with `Vary: *`, or responses carrying a `Set-Cookie` header. The `if (request.method !== "GET")` guard exists precisely because non-GET requests cannot be cached, so there is no reason to route them through the cache lookup. And because the Cache API ignores `stale-while-revalidate` entirely, the code never sets it; `s-maxage` alone defines the freshness window, and the manual `refresh` replaces the SWR the API will not honor. Once the `s-maxage` window passes, `cache.match` returns a miss, and that request pays the full origin round trip while the fresh copy is stored. Variant B therefore approximates stale-while-revalidate rather than reproducing it: it revalidates in the background on hits, but it does not serve stale past expiry the way Cloudflare's CDN `UPDATING` flow does. If a warm-refresh-on-every-request guarantee matters, prefer Variant A. ## The reverse-proxy equivalent (nginx) If you cache on your own reverse proxy instead of a Worker, the same shape of behavior has a documented switch. ```nginx # Reverse-proxy equivalent of Variant A's stale-while-revalidate behavior: # serve the stale entry while one request refreshes it upstream, instead of # stampeding the origin. proxy_cache_valid sets the fresh window (illustrative). proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=machine:10m inactive=24h; server { location ~ \.(txt|md)$ { proxy_pass http://origin; proxy_cache machine; proxy_cache_valid 200 1h; # illustrative fresh window proxy_cache_use_stale updating; # serve stale while it refreshes } } ``` In nginx, `proxy_cache_use_stale` with the `updating` parameter serves a stale cached response while that entry is being refreshed, which minimizes upstream requests (the default is `off`); since version 1.11.10 nginx can also serve stale directly from an upstream's `stale-while-revalidate` or `stale-if-error` extensions, at lower priority than the `proxy_cache_use_stale` parameters. This produces the same practical outcome as Variant A, serve the cached copy now and refresh it out of band, but it is a distinct mechanism from Cloudflare's asynchronous `UPDATING` status rather than an identical one, so treat them as analogous rather than interchangeable. ## Freshness: what you must not serve stale A store puts llms.txt and every .md mirror behind Variant A with a 1 hour fresh window and a 24 hour stale window, and puts the product-detail pages that carry live price and availability on a 60 second TTL with no stale serving. The link index and Markdown copies are cheap to serve slightly behind; the offer facts an agent quotes are not. This is an illustration of the tradeoff, not a measured configuration. Every TTL in this blueprint (1 hour fresh, 24 hours stale, `s-maxage=3600`, `proxy_cache_valid 200 1h`) is an illustrative editorial choice, not a spec requirement. The directives and their behavior are documented; the specific seconds are yours to set per surface. Stale serving is safe for low-churn surfaces like llms.txt and Markdown mirrors, but price and availability are the opposite: keep them on a short TTL or bypass the cache entirely, because an agent that reads a long-stale offer may quote a price or a stock state you have already changed. ## Which to use, and where this sits Reach for Variant A on llms.txt, .md mirrors, and feeds, where directive-driven SWR is free and the config is one line plus one origin header. Reach for Variant B when you need caching logic to live inside the Worker, for example when the freshness window depends on request shape rather than a static header, and accept that it approximates SWR rather than reproducing it. On a non-Cloudflare stack, the nginx block above is the equivalent lever. This is the serving half of making machine surfaces fast for agents. The chapter [serving agent traffic](/handbook/serving-agent-traffic/) covers the surrounding judgment, which crawlers to expect and how caching protects an origin from refetch bursts, and [AI crawlers, robots.txt and llms.txt for stores](/ai-crawlers-robots-llms-txt/) covers the access-policy side. For what these Workers should be caching, see the companion blueprints: [llms.txt for e-commerce catalogs](/blueprints/llms-txt-for-ecommerce-catalogs/), the [product Markdown mirror](/blueprints/product-markdown-mirror/), and [token-efficient product JSON-LD](/blueprints/token-efficient-product-json-ld/). For the format itself, see the [llms.txt](/glossary/#llms-txt) glossary entry.