DNS Failover and Orphaned Fingerprinted Assets

A health check flaps in the middle of a deploy. DNS shifts traffic to the standby provider, that provider pulls from a standby origin that has not finished receiving the new build, and users get HTML referencing app.a1b2c3d4.js from an origin where that file does not exist yet. Every chunk request returns 404, the application never boots, and the CDN dashboards show a healthy green because the HTML was served perfectly.

This is the failure mode fingerprinting cannot protect you from on its own. Content hashing guarantees that a URL and its bytes never disagree; it guarantees nothing about whether the URL exists on the origin the request happened to reach.

The Failure, Step by Step

The evidence in a browser is a ChunkLoadError or a bare Failed to fetch dynamically imported module, with 404s on paths that are demonstrably correct. Confirm the diagnosis by asking each origin directly rather than through the CDNs, which will happily cache the 404:

HASH_PATH=/assets/app.a1b2c3d4.js

for origin in origin-a.example.net origin-b.example.net; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -I "https://${origin}${HASH_PATH}")
  printf '%-24s %s\n' "$origin" "$code"
done

# origin-a.example.net     200
# origin-b.example.net     404   <-- the standby has not received this object

A 200 from one origin and a 404 from the other, for a path the current HTML references, is the entire bug. Everything below is about closing the window in which that state is reachable by real traffic.

Replication Lag Is the Orphan Window

If both providers pull from one bucket, this failure cannot happen — which is the strongest argument for the single-asset-origin topology described in the multi-CDN purge orchestration guide. The moment you introduce a second origin for resilience, you also introduce a window in which the two disagree, and its width is your replication lag.

Three properties of the replication job decide how wide the band is. Direction — a job triggered by the deploy is bounded and predictable; continuous bucket replication is eventually consistent with no upper bound you can quote. Ordering — if HTML replicates before assets, the standby serves new HTML with missing chunks even without a failover, which is the same bug on a timer. Completeness — a replication job that skips objects it considers unchanged will skip nothing here, since every new hash is a new key, but it will happily leave the previous build behind if a lifecycle rule already removed it from the source.

The fix for ordering is the same as for any deploy: assets first, HTML last, everywhere. Replicate /assets/ to the standby and wait for that job to report complete before uploading HTML to either origin. The CI/CD asset pipeline guide sequences a single-origin deploy the same way; a second origin just adds one wait.

Measuring the Lag You Actually Have

Quote a real number rather than the replication service’s marketing figure. Write a uniquely named canary object to the primary at deploy time and poll the standby until it appears:

#!/usr/bin/env bash
# scripts/replication-lag.sh — how long until the standby has a new object
set -euo pipefail

KEY="assets/replication-canary-$(date +%s).txt"
printf 'canary\n' | aws s3 cp - "s3://assets-origin-a/${KEY}" >/dev/null
START=$(date +%s%N)

until curl -sf -o /dev/null -I "https://origin-b.example.net/${KEY}"; do
  sleep 0.5
done

printf 'replication lag: %d ms\n' $(( ($(date +%s%N) - START) / 1000000 ))
aws s3 rm "s3://assets-origin-a/${KEY}" >/dev/null

# replication lag: 47230 ms

Run it hourly and alert on the 99th percentile, not the mean. Replication that averages two seconds and occasionally takes four minutes gives you a four-minute orphan window, and the average tells you nothing about it. The measured p99 is also the correct value for the deploy job’s wait: block on replication for p99 plus a margin before uploading HTML, and the mid-deploy version of this incident stops being reachable.

Retention Windows That Must Span the Failover

The mirror-image failure is a standby that has only the new build. A client holding HTML from ten minutes ago fails over, requests the previous build’s chunks, and the standby returns 404 because its lifecycle rule swept them. Both origins therefore need a retention window that covers every source of stale HTML, and the window must be the longest of them, not the average.

Add the caching layers and you get minutes; add the session tail and you get days. A seven-day retention on hashed asset prefixes covers all of it at negligible storage cost, and it must be configured identically on both origins:

{
  "Rules": [
    {
      "ID": "retain-hashed-assets-7d",
      "Status": "Enabled",
      "Filter": { "Prefix": "assets/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 7 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

Note what this rule does not do: it never expires current objects. Hashed assets are only ever added, so “current” is the whole set and the rule only reaps versions superseded by an accidental overwrite. That is the safe default. A rule that expires objects by age instead — "Expiration": { "Days": 7 } — deletes chunks that are still referenced by any HTML older than a week and is the most common way teams manufacture this incident for themselves. The related reasoning about how long each caching layer can hold a reference is set out in immutable Cache-Control and TTL tuning.

Gate the Cutover on Asset Parity, Not Just Origin Health

Most failover configurations health-check a single path — /healthz or / — from a handful of probe locations. That check passes on an origin that is missing every chunk of the current build, because the HTML is present and the health path is not fingerprinted. A cutover gate needs three conditions, and the parity check is the one nobody has.

Point the health check at a fingerprinted path that changes every deploy, and the first gate starts doing real work. A route that returns the current build’s manifest — or simply a HEAD against the largest hashed chunk — turns a liveness probe into a readiness probe:

# Route 53 health check aimed at a build-specific path, not "/"
resource "aws_route53_health_check" "standby_assets" {
  fqdn              = "origin-b.example.net"
  port              = 443
  type              = "HTTPS_STR_MATCH"
  resource_path     = "/assets/manifest.json"
  search_string     = "2f9c41a7"
  failure_threshold = 2
  request_interval  = 10
}

The search_string is the deploy identifier. When the standby is behind, the string is absent, the check fails, and Route 53 stops steering traffic there — which is the correct behaviour, because a standby that cannot serve the current build is not a standby.

The Pre-Failover Parity Check

Run this before any deliberate cutover, and on a schedule between deploys. It reads the build manifest and asserts that every hashed file it names resolves on both origins.

// scripts/asset-parity.mjs — assert both origins can serve the current build
// Usage: node scripts/asset-parity.mjs dist/manifest.json
import { readFile } from 'node:fs/promises';

const origins = ['https://origin-a.example.net', 'https://origin-b.example.net'];
const manifest = JSON.parse(await readFile(process.argv[2], 'utf8'));

// A Vite/Webpack manifest maps source paths to emitted, hashed filenames.
const files = [...new Set(Object.values(manifest).map((entry) =>
  typeof entry === 'string' ? entry : entry.file
).filter(Boolean))];

async function head(origin, file) {
  try {
    const response = await fetch(`${origin}/${file}`, {
      method: 'HEAD',
      signal: AbortSignal.timeout(5000)
    });
    return response.status;
  } catch {
    return 0;
  }
}

async function pool(items, limit, worker) {
  const results = [];
  let index = 0;
  const runners = Array.from({ length: limit }, async () => {
    while (index < items.length) {
      const i = index;
      index += 1;
      results[i] = await worker(items[i]);
    }
  });
  await Promise.all(runners);
  return results;
}

let missing = 0;
for (const origin of origins) {
  const statuses = await pool(files, 16, (file) => head(origin, file));
  const absent = files.filter((_, i) => statuses[i] !== 200);
  console.log(`${origin}: ${files.length - absent.length}/${files.length} present`);
  for (const file of absent.slice(0, 10)) console.log(`  missing ${file}`);
  missing += absent.length;
}

if (missing > 0) {
  console.error(`parity check failed: ${missing} object(s) missing — do not cut over`);
  process.exit(1);
}
console.log('parity check passed: both origins can serve this build');

The concurrency pool matters at realistic manifest sizes: a build with 300 chunks across two origins is 600 HEAD requests, which takes about four seconds at 16 in flight and over a minute serialised. Keep it fast enough that nobody is tempted to skip it during an incident.

Extend the same check backwards to cover retention by running it against the previous build’s manifest, which you should be keeping as a deploy artefact:

# Current build must be present; previous build must still be present too.
node scripts/asset-parity.mjs dist/manifest.json
node scripts/asset-parity.mjs artifacts/manifest-previous.json

If the second command fails, a lifecycle rule is sweeping assets that live clients still reference — fix the retention policy before touching DNS.

Surviving the Window in the Client

Infrastructure gates shrink the window; they do not close it, because a client can be mid-navigation when DNS moves. Give the application one recovery path for a chunk that genuinely cannot be fetched. A dynamic import that fails is recoverable if you retry once and then reload against whatever HTML the new provider is serving:

// src/lazy.js — retry a failed chunk once, then reload to re-resolve the build
const RELOAD_FLAG = 'chunk-reload-attempted';

export async function lazy(loader) {
  try {
    return await loader();
  } catch (error) {
    // A second attempt handles a transient 404 during a replication gap.
    try {
      await new Promise((resolve) => setTimeout(resolve, 1200));
      return await loader();
    } catch {
      // Reload once — the fresh HTML names hashes the current origin has.
      if (!sessionStorage.getItem(RELOAD_FLAG)) {
        sessionStorage.setItem(RELOAD_FLAG, '1');
        location.reload();
        return new Promise(() => {}); // never resolves; the page is going away
      }
      throw error;
    }
  }
}

The sessionStorage flag is what stops a reload loop when the chunk is missing for a structural reason rather than a transient one — without it, an orphaned asset turns into an infinite refresh, which is a worse incident than the blank screen it was meant to fix. Clear the flag on a successful boot so the next genuine gap gets its own retry.

When to Reconsider

Prefer one asset origin. Almost every problem on this page is caused by the second origin, not solved by it. Object storage with cross-region replication and a global endpoint gives you regional redundancy without two divergent namespaces, and both CDNs pointing at one endpoint makes parity a non-question. Take two origins only when the requirement is surviving the loss of an entire storage provider.

Accept a slower failover in exchange for correctness. A cutover gated on parity is slower than one gated on a ping — usually by seconds, occasionally by the length of a replication job. If your availability target genuinely cannot absorb that, the answer is to shorten replication lag, not to remove the gate.

Do not fail over during a deploy at all. The simplest mitigation is a deploy lock that suppresses automatic cutovers for the duration of a release, with a manual override. It converts a rare, confusing failure into a rare, loud one — and the split-brain reconciliation check already gives you the signal to lift the lock as soon as both providers agree on the build.

Related

• Multi-CDN and origin shield purging — the parent guide, including why one asset origin is the default
• Split-brain caches across two providers — the same divergence expressed in the cache rather than the origin
• Shield hit ratio versus purge latency — the shield as a shared failure domain
• CI/CD asset pipeline integration — ordering asset sync ahead of HTML in the deploy job
• Immutable Cache-Control and TTL tuning — how long each layer can hold a reference to an old build