Purging Two CDNs Without a Split-Brain Cache

Support reports that the new release “works for some people”. Provider A is serving the freshly deployed HTML; provider B is still handing out the previous document, and which one a user gets depends on nothing more principled than which PoP their resolver picked. That is a split-brain cache, and it is almost always a purge fan-out that partially failed and exited zero anyway.

The Symptom, Precisely

The tell is not a broken page — it is two correct pages that disagree. A user on provider B loads last week’s HTML, which references last week’s hashed bundles. Those bundles still resolve, because fingerprinted files are never deleted at deploy time, so nothing 404s and nothing appears in your error tracker. The user simply sees an older application, indefinitely, until provider B’s HTML happens to expire.

Split-brain symptom A user routed to CDN A receives the new HTML build and its new hashed bundle, while a user routed to CDN B receives the previous build because that provider's purge call failed, and each user then requests a different set of fingerprinted assets. One deploy, two answers, no error anywhere User in Berlin resolves CDN A User in Lima resolves CDN B CDN A edge purge returned 200 serves build 2f9c CDN B edge purge 429, retry lost serves build 9ab1 Berlin gets the new build requests app.a1b2c3d4.js 200 from the shield Lima gets the old build requests app.7e5f2c91.js 200 only while retained Nothing 404s, so no alert fires — the older build is simply served forever
Both branches return HTTP 200. The divergence is invisible to error monitoring and visible only in a build identifier.

Reproduce it deliberately before you trust any fix. Pin one PoP address per provider and compare the build identifier your template embeds:

HOST=www.example.com
A_POP=203.0.113.10
B_POP=198.51.100.20

for pair in "cdn-a:$A_POP" "cdn-b:$B_POP"; do
  name=${pair%%:*}; ip=${pair##*:}
  printf '%-6s ' "$name"
  curl -s --resolve "$HOST:443:$ip" "https://$HOST/" \
    | grep -o 'data-deploy-id="[0-9a-f]*"' | head -1
done

# cdn-a  data-deploy-id="2f9c41a7"
# cdn-b  data-deploy-id="9ab13d05"   <-- previous build, still being served

If your HTML carries no build identifier, add one before anything else. A data-deploy-id attribute on <html>, or a <meta name="build" content="…">, costs nothing and turns “some users see an old page” from a rumour into a one-line assertion. Without it, the closest proxy is the first hashed script tag in the document, which works but is harder to read in logs.

The response headers narrow it further. Ask each provider what it thinks the object’s state is:

curl -sI --resolve "$HOST:443:$A_POP" "https://$HOST/" \
  | grep -iE 'cf-cache-status|age|cache-control'
# cf-cache-status: MISS
# age: 0
# cache-control: no-cache

curl -sI --resolve "$HOST:443:$B_POP" "https://$HOST/" \
  | grep -iE 'x-cache|x-served-by|age|cache-control'
# x-cache: HIT, HIT
# x-served-by: cache-lim1234-LIM, cache-mia4567-MIA
# age: 3412
# cache-control: no-cache

An age of 3412 on an object whose purge you believe succeeded five minutes ago is the whole diagnosis. The object was never evicted; the edge is counting up from the original fetch. Note also that x-cache: HIT, HIT names two tiers — the second is the shield, which tells you the stale copy is upstream of the edge and a lower-tier purge alone will not fix it.

Why the Fan-Out Half-Fails

Purge calls to different providers fail in uncorrelated ways, which is exactly what makes partial failure the common case rather than the rare one.

  • Rate limiting. A matrix deploy running the same job for several environments fires every provider’s purge in the same second. One provider’s limiter trips, the other’s does not.
  • Ambiguous timeouts. A request that times out after the provider accepted it is indistinguishable from one that never landed. A pipeline that treats a timeout as failure and stops, without retrying, leaves the other provider purged.
  • Credential expiry. Tokens rotate on different schedules per provider. The branch with the expired token returns 401 in under 50 ms while the healthy branch returns 200, and a script using && chaining stops at the wrong place.
  • Divergent URL lists. Provider A is purged by URL and provider B by surrogate key. If the key is not attached to every HTML response — a Fastly Surrogate-Key header missing from one route’s origin response — that route silently never purges. This is why the key-tagging strategy in the Fastly instant purge guide insists on asserting the header at the origin, not just at deploy time.
  • A shell that ignores exit codes. curl -s without --fail, or a for loop whose body’s exit status is discarded, reports success for an HTTP 500 body.

The Outcome Matrix

Enumerate the combinations and the correct action falls out. There are only four interesting rows.

CDN A result CDN B result What users experience Correct action
2xx 2xx Consistent — everyone on the new build Proceed; run the reconciliation check anyway
2xx 429 / 5xx after retries Split brain until B’s HTML TTL expires Fail the deploy, re-run the whole fan-out
Timeout, state unknown 2xx Possibly consistent, possibly not Re-run: the purge is idempotent, so a duplicate call is free
2xx 2xx on an incomplete URL list One route stale on B, all others fine Fix the list, re-run; add a per-route assertion
Fan-out result matrix A four-row grid pairing the result from each provider with the resulting user experience: two successes are consistent, a failure on either side produces a split brain, an unknown timeout is resolved by re-running, and an incomplete URL list leaves one route stale. Every combination of fan-out results and what it means CDN A CDN B Outcome for users 2xx 2xx Consistent, new build 2xx 429 after retries Split brain, block release timeout, unknown 2xx Unknown, re-run the call 2xx 2xx, short URL list One route stale on B
Only the first row is a finished deploy. Every other row is an action item, not a warning to be logged and forgotten.

Notice that three of four rows end in “run it again”. That is the design constraint: because purging an already-fresh URL costs nothing on every major provider, the cheapest correct policy is to re-run the entire fan-out rather than reason about which branch needs repair.

Treat the Fan-Out as All-or-Nothing

Cache purges cannot be rolled back — you cannot un-evict an object — so the transaction you are enforcing is not atomicity in the database sense. It is a simpler contract: the deploy is not complete until every branch has reported success, and the pipeline may not advance until it has. In practice that means three rules.

First, run every branch to completion before evaluating any of them. A set -e script that aborts at the first failure leaves the remaining providers unpurged and produces a worse split than the one it was reacting to. Collect all results, then decide.

Second, make the aggregate exit code the only thing downstream steps look at. No “purge warnings” in the log that a human is supposed to notice.

Third, retry the whole fan-out rather than the failed branch. Repeating a successful purge is a no-op; repeating only the failed branch means your retry path is a second, less-tested code path.

#!/usr/bin/env bash
# scripts/fanout-guard.sh — run every branch, then judge them together
set -uo pipefail   # deliberately NOT -e: a failing branch must not abort the loop

DEPLOY_ID="$1"
declare -A RESULT

purge_cloudflare() {
  curl -s -o /tmp/cf.out -w '%{http_code}' -X POST \
    "https://api.cloudflare.com/client/v4/zones/${CF_ZONE_ID}/purge_cache" \
    -H "Authorization: Bearer ${CF_API_TOKEN}" \
    -H "Content-Type: application/json" \
    --data '{"files":["https://www.example.com/","https://www.example.com/pricing/"]}'
}

purge_fastly() {
  curl -s -o /tmp/fastly.out -w '%{http_code}' -X POST \
    "https://api.fastly.com/service/${FASTLY_SERVICE_ID}/purge/html-entry-points" \
    -H "Fastly-Key: ${FASTLY_API_TOKEN}" \
    -H "Accept: application/json"
}

for attempt in 1 2 3; do
  RESULT[cdn-a]=$(purge_cloudflare)
  RESULT[cdn-b]=$(purge_fastly)
  echo "attempt ${attempt}: cdn-a=${RESULT[cdn-a]} cdn-b=${RESULT[cdn-b]} deploy=${DEPLOY_ID}"
  if [[ ${RESULT[cdn-a]} == 2* && ${RESULT[cdn-b]} == 2* ]]; then
    echo "fan-out complete"
    exit 0
  fi
  sleep $((attempt * 2))
done

echo "fan-out incomplete after 3 attempts: cdn-a=${RESULT[cdn-a]} cdn-b=${RESULT[cdn-b]}" >&2
exit 1

The missing -e is the point of the script, and it deserves a comment in your repository so nobody “fixes” it. A fuller Node implementation with per-provider concurrency and backoff lives in the multi-CDN purge orchestration guide; this shell version is what fits in a deploy job that does not want a Node dependency.

The Overlap Window Both Builds Need

A split brain is survivable only because both builds’ assets remain fetchable. The instant you sweep the previous build’s hashed files out of the origin, every client still holding old HTML — whether from a lagging CDN, a browser cache, or a single-page-app session open since yesterday — starts failing to load chunks.

Asset overlap window A timeline in which the previous build's assets remain published from the deploy until well after the purge, overlapping with the new build's assets, so that clients holding either version of the HTML can resolve every referenced file. Both builds must resolve across the whole overlap window Previous build assets kept live New build assets published overlap window t0 deploy purge last stale HTML expires t0 + 7 d Sweeping old hashes at t0 breaks every client still holding old HTML
The previous build's files must outlive the slowest copy of the HTML that references them — including the copy stuck on a lagging provider.

Size the window from the slowest thing that can hold stale HTML, not the fastest. With two providers that is the larger of the two HTML edge TTLs, plus the DNS TTL if a client can be moved between providers mid-session, plus the browser TTL, plus a margin for sessions that never reload. A 7-day retention on the previous build’s asset prefix is cheap in object storage and eliminates the entire class of incident; the reasoning behind these numbers is developed in immutable Cache-Control and TTL tuning.

A useful corollary: never overwrite a hashed filename with different bytes. If a rollback republishes the old code under a new hash instead of restoring an old hash in place, both builds stay internally consistent no matter which provider a client is talking to.

The Reconciliation Check

A successful purge response is a claim about the future. Reconciliation is where you verify it. Run the check a few seconds after the fan-out, from outside the deploy’s own control flow, and let it re-run the fan-out when the providers disagree.

Reconciliation loop After the fan-out, a loop polls both edges and compares the deploy identifier in the HTML. Matching identifiers complete the release; a mismatch re-runs the idempotent fan-out, and three failed rounds escalate to on-call. Reconciliation loop after every fan-out Poll both edges every 5 s Same deploy id? compare HTML on both edges Consistent release complete Re-run fan-out idempotent retry Page on-call yes no after 3 rounds loop back and re-poll
The loop closes the gap between "the API said 200" and "both providers are actually serving the release".
// scripts/reconcile.mjs — assert both providers converge on one build id
// Usage: node scripts/reconcile.mjs 2f9c41a7
import { setTimeout as sleep } from 'node:timers/promises';

const deployId = process.argv[2];
const host = 'www.example.com';
const pops = { 'cdn-a': '203.0.113.10', 'cdn-b': '198.51.100.20' };
const idPattern = /data-deploy-id="([0-9a-f]+)"/;

async function servedBuild(ip) {
  // Node resolves the hostname normally; the Host header pins the request to one PoP.
  const response = await fetch(`https://${ip}/`, {
    headers: { Host: host, 'Cache-Control': 'no-cache' },
    signal: AbortSignal.timeout(6000)
  });
  const html = await response.text();
  return idPattern.exec(html)?.[1] ?? 'unknown';
}

let round = 0;
while (round < 3) {
  round += 1;
  const seen = {};
  for (const [name, ip] of Object.entries(pops)) {
    seen[name] = await servedBuild(ip);
  }
  const consistent = Object.values(seen).every((id) => id === deployId);
  console.log(`round ${round}: ${JSON.stringify(seen)} consistent=${consistent}`);
  if (consistent) {
    console.log('reconciled');
    process.exit(0);
  }
  await sleep(5000);
}

console.error(`providers still disagree after 3 rounds — expected ${deployId}`);
process.exit(1);

Two details make this check trustworthy. Cache-Control: no-cache on the request forces most edges to revalidate rather than answer from a local hop, so you are testing the CDN and not an intermediary. And the loop bounds itself: three rounds at five seconds is long enough for a Cloudflare purge to settle and short enough that a genuinely broken deploy is escalated within twenty seconds rather than being retried until the pipeline times out.

One more assertion is worth adding for a topology with a shield: compare what the shield serves against what each edge serves. If the shield has the new build and one edge does not, the edge purge failed. If the shield still has the old build, the fan-out ordering is wrong and the edges will re-pull the stale document as soon as they miss.

Catching It From Production Telemetry

The deploy-time check only sees the PoPs you pinned. A divergence that appears an hour later — because one provider’s shield expired an object and re-pulled a copy that was never purged — needs a signal from real traffic. The cheapest one is to report the build identifier the page was loaded with alongside the delivery provider, which every CDN exposes in a response header you can read from the document: Cloudflare sets cf-ray, Fastly sets x-served-by, CloudFront sets x-amz-cf-pop.

// src/report-build.js — one beacon per page load, sampled at 5%
if (Math.random() < 0.05) {
  const buildId = document.documentElement.dataset.deployId ?? 'unknown';
  navigator.sendBeacon('/telemetry/build', JSON.stringify({
    buildId,
    provider: window.__CDN_PROVIDER__ ?? 'unknown',
    at: Date.now()
  }));
}

Aggregate by buildId over a five-minute window. Under normal operation you see two identifiers during a rollout and one afterwards. A second identifier that persists past the HTML TTL, concentrated on one provider, is a split brain that your deploy-time check already passed — and it is the only way to notice one that started after the pipeline finished.

When to Reconsider

Blocking a release on a purge is not free, and there are situations where a softer policy is correct.

If your HTML edge TTL is already 30 seconds or less, a failed purge on one provider self-heals within a minute and the blast radius is a handful of users seeing a slightly older page. In that regime, failing the deploy — and leaving the pipeline half-finished, with new assets uploaded but the release marked failed — can be the more disruptive choice. Log loudly, alert, and let the TTL do the work.

If one provider is a genuine cold standby carrying no live traffic, its stale cache harms nobody until traffic actually moves. Treat a failed purge there as a pre-condition on the failover instead: no cutover until that provider’s cache has been verified fresh. The gating pattern is set out in DNS failover and orphaned fingerprinted assets.

And if one of your providers is CloudFront, “success” is a different shape: CreateInvalidation returns before the work is done, so the reconciliation loop rather than the API response is your only real signal. Budget minutes, not seconds, and read the CloudFront invalidation reference for the polling semantics before you set the loop’s bounds.