Nginx Open File Cache and Stale Fingerprints

A freshly deployed hashed bundle returns 404, or returns the previous release’s bytes, and every proxy cache purge you issue changes nothing — because the entry holding you back is a file descriptor, not a cached response.

This is one of the few ways a fingerprinted asset can go stale at all. The whole premise of content hashing is that a URL and its bytes are the same fact, so no cache can ever hold a wrong answer for a hashed path. open_file_cache breaks that premise, because it does not cache bytes keyed by URL — it caches filesystem lookups keyed by path string, and a path string can absolutely start resolving to something new while Nginx keeps believing the old result.

Symptoms That Point Here Rather Than at proxy_cache

Three signatures separate a descriptor problem from a response-cache problem:

  • The location block serving the file uses root or alias, not proxy_pass. proxy_cache is only consulted on the proxy path; a disk-served request never touches it, so purging it is a no-op by definition.
  • X-Cache-Status is absent from the response, or shows a value you hardcoded. The variable $upstream_cache_status is empty when there is no upstream.
  • The file demonstrably exists. ls -l on the origin host shows it, cat shows the right bytes, and Nginx still answers 404 or emits the old body.

The error log is the tiebreaker. A negative entry replayed from cache produces no new open() failure line at all — the log is silent, which is itself the clue. A genuine miss logs open() "/srv/assets/current/main-e5f67890.js" failed (2: No such file or directory) every time.

What Each Directive Actually Caches

open_file_cache is four cooperating directives, and each one caches a different thing with a different staleness window. Reading them as a single knob is what makes the failure mode confusing.

Directive What it caches Stale window Deploy risk
open_file_cache max=N inactive=T Open file descriptors, plus size, modification time, and inode number Entry survives T after its last use A held descriptor keeps pointing at a replaced inode
open_file_cache_valid T Nothing itself — sets how long before an entry is re-stat()ed Exactly T seconds Metadata (including size) is up to T seconds out of date
open_file_cache_min_uses N Threshold for admission into the cache n/a N=1 admits one-off probes and scanner requests
open_file_cache_errors on|off Negative results: ENOENT, EACCES, and directory-lookup failures Same as open_file_cache_valid 404s a hashed file that has since landed on disk
proxy_cache Full response bodies and upstream headers proxy_cache_valid Unrelated — never consulted for disk-served files
Two caches, two payloads open_file_cache stores open file descriptors, stat metadata such as size modification time and inode, and negative directory lookup results, all in memory. proxy_cache stores complete response bodies, upstream response headers, and a time to live on disk. Neither layer can invalidate the other. open_file_cache in-memory metadata file descriptor (open fd) size, mtime, inode from stat() directory lookup errors proxy_cache on-disk response store full response body bytes upstream response headers TTL from proxy_cache_valid separate layers: purging one never touches the other
A PURGE request reaches only the right-hand column; nothing in the left-hand column has a purge interface at all.

The asymmetry matters: proxy_cache has an eviction API, and open_file_cache has none. There is no directive, signal, or HTTP verb that drops a single path from it. Your only controls are the valid window, the inactive window, and a worker restart — which is what nginx -s reload effectively gives you, since the new worker processes start with empty caches.

The standard atomic deploy writes a timestamped release directory and repoints a current symlink at it with a rename. The rename is genuinely atomic at the filesystem level. The problem is that Nginx’s cache is keyed on the literal path string /srv/assets/current/main-a1b2c3d4.js, and that string does not change when the symlink flips. Nginx has no reason to suspect anything happened, so it keeps serving from the descriptor it opened against the previous release’s inode.

Unix keeps that inode alive as long as a descriptor references it, even after the directory entry is unlinked. So a deploy that prunes old releases immediately does not crash — it silently keeps serving the old bytes from an inode that no longer has a name. When the descriptor finally ages out and Nginx re-resolves the path, it gets the new release. Users see a version boundary that moves at open_file_cache_valid granularity, not at deploy granularity.

Symlink flip versus a cached descriptor A worker opens a file through the current symlink and caches the descriptor. Thirty seconds later the deploy writes a new release directory and renames the symlink onto it. One second after that the cached descriptor still resolves to the previous release, serving stale bytes or a 404. Only when the open_file_cache_valid window elapses does Nginx re-stat the path and pick up the new inode. T+0s Worker opens current/main-a1b2c3d4.js descriptor and stat() cached, resolving to release-a T+30s Deploy writes release-b and renames the symlink atomic at the filesystem level; the old inode survives T+31s Cached descriptor still points at release-a stale bytes, or a 404 once release-a is pruned T+90s open_file_cache_valid elapses, Nginx re-stats the new inode resolves and correct bytes are served
The rename is atomic; the descriptor cache is not. Between the flip and the next re-stat, the path and its contents disagree.

disable_symlinks controls whether Nginx will traverse a symbolic link at all. The default is off, which is what a release-symlink deploy needs. Setting disable_symlinks on makes Nginx refuse any path containing a link component — your current symlink stops working entirely and every asset returns 403. Setting disable_symlinks if_not_owner permits the link only when it is owned by the same user as the file it points to, which breaks the moment your deploy user and your web user differ.

The subtler interaction is cost. Any value other than off forces Nginx to lstat() every component of the path on every request, which is exactly the syscall traffic open_file_cache exists to remove. If you need the hardening, point root directly at the resolved release directory and template that path into the config at deploy time, rather than paying for link checks on every request.

sendfile and the Rewritten File

sendfile on tells the kernel to stream bytes straight from the page cache of an open descriptor to the socket, and Nginx sets Content-Length from the cached stat() result before the transfer begins. Overwrite a file in place while that is happening and two things go wrong at once. The response length was computed from the old size, so a shorter replacement yields a truncated body and a longer one yields a connection that stalls until it times out. And because the write is not atomic, a client can receive the first half of the old file followed by the second half of the new one — a syntactically broken bundle that no error handler catches.

This is the strongest practical argument for never mutating an asset path. Hashed filenames make in-place rewrites structurally impossible: new content means a new name, so the old descriptor stays valid for exactly the bytes it was opened against. Eight hex characters is the right default for the hash; 12 to 16 suits monorepos emitting thousands of chunks, where truncation collisions become worth worrying about.

A Configuration That Does Not Bite

The rules are: keep the valid window short enough to be shorter than any human’s patience, never cache negative results on hashed paths, and reload after a flip so no descriptor outlives its release.

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile      on;
    tcp_nopush    on;

    # Descriptor cache: aggressive on capacity, conservative on staleness.
    open_file_cache          max=20000 inactive=20s;
    open_file_cache_valid    10s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   off;

    # Required for a release-symlink layout; anything else costs an
    # lstat() per path component on every request.
    disable_symlinks off;

    server {
        listen 443 ssl http2;
        server_name assets.example.com;

        ssl_certificate     /etc/ssl/certs/assets.example.com.crt;
        ssl_certificate_key /etc/ssl/private/assets.example.com.key;
        ssl_protocols       TLSv1.2 TLSv1.3;

        # /srv/assets/current is a symlink to the active release directory.
        root /srv/assets/current;

        # Fingerprinted assets: 8+ hex characters before the extension.
        location ~* \.[0-9a-f]{8,}\.(js|css|woff2?|svg|png|jpg|webp|avif|ico)$ {
            # Never cache a 404 for a path whose file may be mid-upload.
            open_file_cache_errors off;

            add_header Cache-Control "public, max-age=31536000, immutable" always;
            add_header X-Served-From "disk" always;

            try_files $uri =404;
        }

        # HTML entry point: the one file that legitimately changes in place.
        location = /index.html {
            open_file_cache_valid  1s;
            open_file_cache_errors off;

            add_header Cache-Control "no-cache, must-revalidate" always;
            add_header X-Served-From "disk" always;
        }

        location / {
            try_files $uri $uri/ /index.html;
        }
    }

    server {
        listen 80;
        server_name assets.example.com;
        return 301 https://$host$request_uri;
    }
}

The matching deploy script writes a new release, flips the link, reloads, and only then prunes — keeping enough history that any descriptor still held by a draining worker resolves to a directory that exists:

#!/usr/bin/env bash
set -euo pipefail

RELEASE="/srv/assets/releases/$(date -u +%Y%m%dT%H%M%SZ)"
LINK="/srv/assets/current"

mkdir -p "$RELEASE"
rsync -a --delete dist/ "$RELEASE/"

# Atomic swap: build the new link beside the old one, then rename over it.
ln -sfn "$RELEASE" "${LINK}.staged"
mv -Tf "${LINK}.staged" "$LINK"

# New workers start with empty descriptor caches; old workers drain and exit.
nginx -s reload

# Keep three releases so draining workers never lose their inode.
ls -1dt /srv/assets/releases/* | tail -n +4 | xargs -r rm -rf

Verification: Deploy a New Hash and Curl It Immediately

The test that proves the configuration is safe is deliberately impatient — it writes a file with a hash nobody has ever requested and fetches it in the same second, with no sleep to hide a stale negative entry.

#!/usr/bin/env bash
set -euo pipefail

HOST="https://assets.example.com"
STAMP=$(openssl rand -hex 4)          # 8 hex characters, like a build hash
FILE="probe-${STAMP}.js"
RELEASE=$(readlink -f /srv/assets/current)

# 1. Poison the negative cache on purpose: ask before the file exists.
curl -s -o /dev/null -w 'pre-deploy GET  -> %{http_code}\n' "${HOST}/${FILE}"

# 2. Publish the file into the live release directory.
printf 'export const build = "%s";\n' "$STAMP" > "/tmp/${FILE}"
install -m 0644 "/tmp/${FILE}" "${RELEASE}/${FILE}"

# 3. Fetch immediately. No sleep — that is the entire point of the test.
CODE=$(curl -s -o /dev/null -w '%{http_code}' "${HOST}/${FILE}")
echo "post-deploy GET -> ${CODE}"

if [ "$CODE" != "200" ]; then
  echo "FAIL: open_file_cache is replaying a cached negative result"
  echo "      set open_file_cache_errors off for the hashed-asset location"
  rm -f "/tmp/${FILE}" "${RELEASE}/${FILE}"
  exit 1
fi

# 4. Confirm the bytes are the new ones, not a stale descriptor's.
if curl -s "${HOST}/${FILE}" | grep -q "$STAMP"; then
  echo "PASS: body matches the freshly written build marker"
else
  echo "FAIL: 200 returned but the body is stale"
  exit 1
fi

rm -f "/tmp/${FILE}" "${RELEASE}/${FILE}"

A 404 at step 3 with a 200 a few seconds later is the definitive fingerprint of open_file_cache_errors on. A 200 at step 3 whose body fails the grep means a descriptor is outliving its inode — check whether the deploy overwrote a path rather than creating a new one.

Which cache is holding the stale answer Starting from a fresh hash that returns 404 or stale bytes, the branch depends on whether the serving location proxies to an upstream or reads from disk. The proxy branch points at proxy_cache and its purge interface. The disk branch points at open_file_cache descriptors and stat results, which no purge can reach. Both branches converge on the same fix: publish files before anything references them, then reload. Fresh hash 404s or serves stale start here Does the location proxy_pass? or read via root / alias proxy disk proxy_pass path proxy_cache holds the body purge the key or cut the TTL open_file_cache is not used disk-served path open_file_cache holds the fd and the cached stat() result purging changes nothing here Fix: publish files first, then reload
The branch you are on decides whether purging is even a lever; both branches end at the same deploy ordering rule.

When to Reconsider

Turning open_file_cache off entirely is a legitimate choice and costs less than people assume. On local NVMe with a warm page cache, the open() and fstat() pair it eliminates is a few microseconds — irrelevant next to TLS negotiation. Disable it when your asset count is small, your traffic is modest, or your deploys are frequent enough that the staleness window is a recurring nuisance.

Keep it, and tune it carefully, in three situations. On network-attached storage — NFS, EFS, or a shared volume in a container platform — every metadata round trip is a network hop, and the descriptor cache can be the difference between one and ten milliseconds per request. At very high request rates against a large asset tree, the syscall savings compound. And on hosts with tight file-descriptor limits, max= gives you an explicit ceiling on how many stay open.

The opposite choice — open_file_cache_errors on — is only correct when you are being actively probed for paths that will never exist and the negative caching is genuinely protecting the filesystem. That is a security posture, not a performance one, and it belongs on a location that does not serve deployable assets. Never enable it on the hashed-asset block; the deploy risk outweighs the scanner savings every time.

If you move disk serving behind a proxy layer instead, the trade-offs change completely — see the immutable assets versus proxy cache purge comparison for how the same asset behaves when proxy_cache is in the path, and the Cache-Control immutable and TTL tuning guide for the header side of the same decision. The ordering guarantees the deploy script above relies on are worth wiring into the pipeline itself rather than a shell script on the host, which the CI/CD asset pipeline integration guide covers in detail.

Frequently Asked Questions

Can I flush open_file_cache without restarting Nginx?

Not selectively. There is no purge interface, no signal, and no directive that drops entries on demand. nginx -s reload is the closest thing: it starts fresh worker processes with empty caches while old workers finish their in-flight requests and exit. That is why the deploy script reloads immediately after the symlink flip rather than waiting for the valid window.

Is open_file_cache_valid the same as inactive?

No, and confusing them is common. valid is how long a cached lookup is trusted before Nginx re-stat()s the path — it governs correctness. inactive is how long an unused entry is kept before eviction — it governs memory. An entry can be re-validated many times over its life and still be evicted the moment it goes inactive seconds without a request.

Does open_file_cache apply to files served through proxy_pass?

No. It only affects files Nginx opens itself: root, alias, try_files, error_page targets, and the pre-compressed files that gzip_static looks for. A proxied request never opens a local file, so no entry is created. The corollary is that adding proxy_pass in front of a static tree makes this failure mode disappear and replaces it with the proxy-cache behaviour described in the Nginx cache purge configuration guide.

Why does the problem only show up for some users?

Each worker process keeps its own cache. With worker_processes auto on an eight-core host you have eight independent descriptor caches, populated at different moments and expiring at different moments. A request load-balanced onto worker three may get the new file while worker five is still replaying a stale entry — which produces the intermittent, unreproducible bug reports that make this failure so hard to pin down.