heredia.dev

Consistent hashing moves fewer keys

Tue, 28 Jul 2026 00:00:00 GMT

The obvious way to spread keys across N servers is hash(key) % N. It is one line, it is perfectly balanced, and it falls apart the moment N changes. Adding a fourth server to a pool of three does not move a quarter of the keys. It moves most of them, because the modulus changed for every key at once.

If those servers are a cache, that is a cold start across the whole fleet during the exact operation, a scale-up, that you were doing because you were already under load.

Consistent hashing fixes the blast radius rather than the balance. Hash the keys and the servers into the same space, arrange that space in a circle, and let each key belong to the first server it meets going clockwise. Adding a server only steals keys from its clockwise neighbour. Removing one only donates keys to its clockwise neighbour. Everything else stays put.

What to try

The counter under the ring reports how many of the 24 keys changed owner as a result of the last thing you did. That number is the whole argument.

With the three starting nodes, remove gamma. Five keys move, and they are exactly the five gamma was holding. Nothing else on the ring notices.
Switch to hash % N and remove gamma again. Seventeen of the 24 move, including keys on nodes that did not change at all.
Switch back to the ring and drag virtual nodes down to 1. The split becomes 5 / 0 / 19: beta lands in a short arc and ends up owning nothing.
Push virtual nodes up to 12. The split converges on 8 / 8 / 8, and removing gamma now moves exactly 8 keys, a clean third.

Step 3 is the honest part of the algorithm. Consistent hashing does not give you balance for free. Three points dropped at random on a circle produce very unequal arcs, and the keys follow the arcs.

Virtual nodes are the fix: hash each server under several distinct labels (gamma#0, gamma#1, and so on) so it occupies many small arcs instead of one large one. Averaging over more arcs narrows the spread, roughly with the square root of the number of points per server. Real implementations use 100 to 500 virtual nodes per server; 12 is only enough here because the picture has to stay readable.

The lookup

Sort the virtual nodes by position once, then every lookup is a binary search for the first position greater than or equal to the key’s, wrapping to index 0 when it runs off the end:

function successor(pos, ring) {
  let lo = 0;
  let hi = ring.length;
  while (lo < hi) {
    const mid = (lo + hi) >> 1;
    if (ring[mid].pos < pos) lo = mid + 1;
    else hi = mid;
  }
  return lo === ring.length ? 0 : lo; // [!code highlight]
}

The highlighted line is the wrap, and it is the only part of this that is easy to get wrong. To watch it happen: turn alpha and beta off, turn epsilon on, and trace user:1049. It sits at 0.907, past every virtual node on the ring, so the arc travels through 1.0 and around to epsilon#0 at 0.252.

Where the model stops

The ring answers one question, “who owns this key”, and it answers it without coordination: every client that knows the membership list computes the same answer. That is the property worth having.

It does not give you even load when keys are hot rather than merely numerous, it does not replicate anything (for that you take the next R distinct nodes clockwise), and it does not tell you when membership changed. That last one is somebody else’s job, usually a gossip protocol or a coordination service, and it is where the genuinely hard problems live.

This post contains an interactive diagram. Read it on heredia.dev.

What Cache-Control actually promises

Tue, 21 Jul 2026 00:00:00 GMT

Cache-Control looks like one setting with a lot of possible values. It is not. It is a list of independent directives, some of which are addressed to the browser sitting on someone’s laptop and some of which are addressed to a CDN edge node in another country. Reading it as a single knob is where most caching bugs start.

Two caches, not one

Every response passes through at least two kinds of cache, and they have different trust levels:

A private cache belongs to exactly one user. The browser’s HTTP cache is the obvious one.
A shared cache serves many users from the same stored response. CDNs, reverse proxies, and corporate egress proxies are all shared caches.

The directives split along that line:

Directive	Applies to	Meaning
`private`	shared caches	Do not store this at all
`public`	shared caches	Store this even if the request was authenticated
`max-age=N`	both	Fresh for N seconds
`s-maxage=N`	shared only	Overrides `max-age` for shared caches
`no-cache`	both	Store it, but revalidate before every reuse
`no-store`	both	Do not write it to disk or memory

The one people get wrong is no-cache. It does not mean “do not cache”. It means “cache this, but check with me before you serve it”. The directive that means what people think no-cache means is no-store.

HTTP/1.1 200 OK
Cache-Control: private, max-age=0, must-revalidate
ETag: "b1946ac9"
Content-Type: application/json

Freshness is a deadline, not a guarantee

max-age is measured from the moment the response was generated at the origin, not the moment your browser received it. A response that sat in a CDN for 50 seconds arrives with Age: 50, and a max-age=60 response is then fresh for 10 more seconds, not 60.

That is what the Age header is for, and it is why intermediate caches are required to add it. When you are debugging a cache, Age is the first thing to look at:

curl -sSI https://example.com/api/config \
  | grep -iE '^(age|cache-control|etag|vary):'

The lifecycle a cache runs for every request is short:

Find a stored response whose request key and Vary headers match.
Compute its current age and compare against max-age / s-maxage.
If fresh, serve it and stop. The origin never hears about the request.
If stale, revalidate with If-None-Match or If-Modified-Since.
On 304 Not Modified, refresh the stored response’s age and serve it.

Step 5 is the part worth internalising. A 304 is not a cache miss. It costs one round trip and zero bytes of body, and it resets the freshness clock.

Revalidation is cheap; recomputation is not

If you can produce an ETag without doing the expensive work, revalidation is nearly free. If computing the ETag requires rendering the whole response anyway, you have saved bandwidth and nothing else. That tradeoff decides whether validators are worth adding to a given endpoint.

const etag = `"${hashOf(record.updatedAt, record.version)}"`; // [!code highlight]

if (req.headers["if-none-match"] === etag) {
  res.writeHead(304, { ETag: etag, "Cache-Control": "max-age=0" });
  return res.end();
}

Serving stale on purpose

stale-while-revalidate decouples “how long this is fresh” from “how long I would rather serve something slightly old than make the user wait”. Inside the SWR window a cache serves the stale response immediately and refreshes it in the background.

location /api/config {
  proxy_pass http://upstream;
  # [!code --]
  add_header Cache-Control "public, max-age=60";
  # [!code ++]
  add_header Cache-Control "public, max-age=60, stale-while-revalidate=600";
}

For anything read-mostly whose staleness costs nothing, that second line is close to free latency. The failure mode is real but narrow: a user can see up to stale-while-revalidate seconds of old data after an update, so it does not belong on anything the user just wrote.

stale-if-error is the sibling directive, and it is the one that earns its keep during an incident:

// [!code focus:2]
const CACHE_HEADER =
  "public, max-age=60, stale-while-revalidate=600, stale-if-error=86400";

export function config(req, res) {
  res.setHeader("Cache-Control", CACHE_HEADER);
  res.setHeader("Vary", "Accept-Encoding");
  res.json(readConfig());
}

Vary is where it breaks

A shared cache keys on the request URL plus whatever Vary lists. Get that wrong and you either poison the cache or destroy your hit rate.

The “Vary” HTTP response header field describes what parts of the request message, aside from the method and target URI, might have influenced the origin server’s process for selecting the content of this response.

— RFC 9111, section 4.1

Two rules cover most of it. Vary: Accept-Encoding is almost always correct and almost always necessary. Vary: Cookie is almost always a mistake: cookies are high-cardinality, so every distinct cookie value gets its own cache entry and the hit rate collapses toward zero. If a response genuinely depends on the user, mark it private and let the browser cache it, instead of asking a shared cache to do something it cannot do safely.

A default worth starting from

Immutable, content-hashed assets: public, max-age=31536000, immutable.
HTML: no-cache plus a strong ETag, so a repeat navigation is a 304 rather than a full download.
Read-mostly JSON: public, max-age=60, stale-while-revalidate=600.
Anything user-specific: private, no-store, and stop thinking about it.

Pick from that list first, and reach for something more elaborate only when you can name the request that made it necessary.