Edge worker for machine surfaces

Version 1.0 · Last verified 2026-07-08

This blueprint gives you two ready-to-deploy Cloudflare Worker configurations for caching your machine surfaces (llms.txt, .md mirrors, RSS or Atom feeds) at the edge, and the choice between them turns on one fact: the Workers Cache API does not honor stale-while-revalidate, so only the CDN path gives you true directive-driven stale serving.

Key takeaways

• Variant A (CDN path) calls fetch with cf.cacheEverything so Cloudflare honors the origin's stale-while-revalidate directive: it serves stale immediately and revalidates in the background. Prefer it for llms.txt and .md mirrors.
• Variant B (Cache API) is self-contained, but the Cache API does not honor stale-while-revalidate, so it has to orchestrate the background refresh by hand with ctx.waitUntil.
• The reverse-proxy equivalent is nginx proxy_cache_use_stale updating, which serves a stale entry while one request refreshes it upstream.
• Every TTL here is illustrative, not a spec requirement. Keep price and availability on a short TTL so an agent never quotes a stale offer.

Why there are two variants

A Worker is a small piece of code that runs at Cloudflare's edge in front of your origin, so it is the natural place to cache low-churn machine surfaces and keep agent refetches off your origin. The question is how you get stale-while-revalidate behavior, where the edge returns the cached copy instantly and refreshes it out of band, which is exactly the pattern that protects an origin from crawler bursts without ever making an agent wait.

A Workers module exports a default object with an async fetch(request, env, ctx) handler, and ctx.waitUntil(promise) extends the Worker's lifetime so it can finish background work after the response has already been returned to the client.Spec-factCloudflare Workers, fetch handler

The load-bearing constraint is what these two Worker APIs do with the stale-while-revalidate directive. The Workers Cache API (caches.default) does not honor the stale-while-revalidate or stale-if-error directives.Spec-factCloudflare Workers, Cache API A Worker that caches through that API therefore cannot get automatic stale serving from a header; once its own freshness window lapses, the next cache.match is simply a miss. To get real directive-driven stale serving you have to go through Cloudflare's CDN cache instead. Cloudflare's CDN serves stale content during revalidation only when the origin's Cache-Control includes stale-while-revalidate; the first request after the fresh window triggers an asynchronous background revalidation and immediately returns the stale response with an UPDATING cache status.Spec-factCloudflare, Cache revalidation

So the split is clean: Variant A rides the CDN path and lets the origin's header drive real SWR; Variant B stays inside the Cache API and rebuilds the stale-then-refresh behavior by hand.

Variant A: CDN path (preferred for llms.txt and .md mirrors)

export default {
  async fetch(request, env, ctx) {
    // Origin sends:  Cache-Control: max-age=3600, stale-while-revalidate=86400
    // cacheEverything makes Cloudflare's CDN cache honor those directives:
    // serve stale + async-revalidate (UPDATING) after the fresh window.
    // The TTLs live in the ORIGIN headers (illustrative: 1h fresh / 24h stale).
    return fetch(request, { cf: { cacheEverything: true } });
  }
};

The whole mechanism lives in one line and one origin header. cf: { cacheEverything: true } tells Cloudflare to run this response through the CDN cache, and the CDN cache is the layer that reads stale-while-revalidate and produces the UPDATING flow described above. Because the TTLs live in the origin's Cache-Control and not in the Worker, you tune freshness per surface by changing a header, not by redeploying code.

We prefer this variant for llms.txt and .md mirrors because stale serving there is nearly free: those files change rarely, and an agent that reads a link index or a Markdown mirror one revalidation behind loses nothing. Variant A is also the smaller surface to maintain, since the freshness policy is data (a header) rather than logic.Hypothesis (our analysis)

Note the header shape. Variant A pairs max-age with stale-while-revalidate rather than using s-maxage. Cloudflare's cache-control guidance notes that s-maxage implies proxy-revalidate, so a shared cache must not serve stale without revalidating, and it advises against combining s-maxage with stale-while-revalidate.Spec-factCloudflare, Cache-Control That is why the SWR path in Variant A is expressed with max-age, while the manual path in Variant B (which has no SWR to combine with) uses s-maxage on its own.

Variant B: Cache API, manual refresh (self-contained)

export default {
  async fetch(request, env, ctx) {
    if (request.method !== "GET") return fetch(request);
    const cache = caches.default;
    const cacheKey = new Request(request.url, request);
    const cached = await cache.match(cacheKey);
    if (cached) {
      // HIT: serve now, revalidate in the background. NOTE: the Cache API does
      // NOT honor stale-while-revalidate, so we orchestrate the refresh manually.
      ctx.waitUntil(refresh(cache, cacheKey, request));
      return cached;
    }
    const response = await fetch(request);
    const toStore = new Response(response.body, response);
    toStore.headers.set("Cache-Control", "s-maxage=3600"); // illustrative TTL
    ctx.waitUntil(cache.put(cacheKey, toStore.clone()));
    return toStore;
  }
};
async function refresh(cache, cacheKey, request) {
  const response = await fetch(request);
  const toStore = new Response(response.body, response);
  toStore.headers.set("Cache-Control", "s-maxage=3600"); // illustrative TTL
  await cache.put(cacheKey, toStore);
}

This variant reconstructs the stale-then-refresh pattern that the Cache API will not give you for free. On a hit it returns the cached copy right away and schedules refresh() with ctx.waitUntil, so the fetch to your origin happens after the agent already has its response. The waitUntil handle is what keeps that background fetch from being cut off when the response returns.

Two documented Cache API behaviors shape the code. cache.match and cache.put respect Cache-Control including s-maxage, and cache.put refuses to store responses to non-GET requests, 206 partial responses, responses with Vary: *, or responses carrying a Set-Cookie header.Spec-factCloudflare Workers, Cache API The if (request.method !== "GET") guard exists precisely because non-GET requests cannot be cached, so there is no reason to route them through the cache lookup. And because the Cache API ignores stale-while-revalidate entirely, the code never sets it; s-maxage alone defines the freshness window, and the manual refresh replaces the SWR the API will not honor.

The reverse-proxy equivalent (nginx)

If you cache on your own reverse proxy instead of a Worker, the same shape of behavior has a documented switch.

# Reverse-proxy equivalent of Variant A's stale-while-revalidate behavior:
# serve the stale entry while one request refreshes it upstream, instead of
# stampeding the origin. proxy_cache_valid sets the fresh window (illustrative).
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=machine:10m inactive=24h;

server {
  location ~ \.(txt|md)$ {
    proxy_pass http://origin;
    proxy_cache machine;
    proxy_cache_valid 200 1h;         # illustrative fresh window
    proxy_cache_use_stale updating;   # serve stale while it refreshes
  }
}

In nginx, proxy_cache_use_stale with the updating parameter serves a stale cached response while that entry is being refreshed, which minimizes upstream requests (the default is off); since version 1.11.10 nginx can also serve stale directly from an upstream's stale-while-revalidate or stale-if-error extensions, at lower priority than the proxy_cache_use_stale parameters.Spec-factnginx, ngx_http_proxy_module This produces the same practical outcome as Variant A, serve the cached copy now and refresh it out of band, but it is a distinct mechanism from Cloudflare's asynchronous UPDATING status rather than an identical one, so treat them as analogous rather than interchangeable.Hypothesis (our analysis)

Freshness: what you must not serve stale

Every TTL in this blueprint (1 hour fresh, 24 hours stale, s-maxage=3600, proxy_cache_valid 200 1h) is an illustrative editorial choice, not a spec requirement. The directives and their behavior are documented; the specific seconds are yours to set per surface.Hypothesis (our analysis) Stale serving is safe for low-churn surfaces like llms.txt and Markdown mirrors, but price and availability are the opposite: keep them on a short TTL or bypass the cache entirely, because an agent that reads a long-stale offer may quote a price or a stock state you have already changed.Hypothesis (our analysis)

Which to use, and where this sits

Reach for Variant A on llms.txt, .md mirrors, and feeds, where directive-driven SWR is free and the config is one line plus one origin header. Reach for Variant B when you need caching logic to live inside the Worker, for example when the freshness window depends on request shape rather than a static header, and accept that it approximates SWR rather than reproducing it. On a non-Cloudflare stack, the nginx block above is the equivalent lever.

This is the serving half of making machine surfaces fast for agents. The chapter serving agent traffic covers the surrounding judgment, which crawlers to expect and how caching protects an origin from refetch bursts, and AI crawlers, robots.txt and llms.txt for stores covers the access-policy side. For what these Workers should be caching, see the companion blueprints: llms.txt for e-commerce catalogs, the product Markdown mirror, and token-efficient product JSON-LD. For the format itself, see the llms.txt glossary entry.

Version 1.0

• 2026-07-08, v1.0: Initial publication.