Joseph Attia

Accept First, Write Later

Two identical servers under one identical traffic spike. The one that waits for its database saturates and starts failing requests; the one that answers first stays flat. waits for the write 1 core · 512 MB answers, then writes 1 core · 512 MB timeout timeout each mark is a request that never got an answer waiting response time response time
The two panels are the same machine running the same spike. Only one of them is still answering at the end.

Twenty thousand people want to sign the same birthday card in the same hour, and the box you can afford has one core and half a gig of memory.

The box is fine. The way you're asking it to work isn't.

A wish is not a transaction

Somebody signs the guestbook at a wedding. Nobody makes them stand at the table until the calligrapher has filed it.

Your endpoint, though, does exactly that. Parse the body, validate it, check out a database connection, run the insert, wait for the commit, then answer. The browser holds the line through all six.

Only one of those steps has to be durable before you reply. And it isn't the insert.

Because a birthday wish has three properties a payment doesn't:

Every one of them is permission to stop waiting.

Requests in flight equals arrival rate times service time. A bar showing that product against a fixed ceiling of sixteen workers. 20,000 wishes across 60 seconds, so 333 arrive each second 333 req/s × 45 ms = 15.0 in flight 16 workers 0102030 40506070 requests in flight the queue drains
Start at the default. You are one millisecond of database latency away from the dashed line.

The arithmetic that decides everything

The bottleneck is the doorway, not the room.

Requests in flight equals how fast they arrive multiplied by how long each one takes. Little's Law, and it holds whether you believe in it or not. Rearranged, your ceiling is the number of workers you have divided by the time each one is busy.

Service time is the only term you control. And it's dominated entirely by whatever you sit and wait on.

A database round trip and an in-memory append are the same three lines of code with an enormous gap in what they let you survive. Drag the slider above and watch the bar cross the line while the hardware never changes.

Which is the answer to the two panels in the first figure. Identical box, identical spike, identical arrival curve. The only difference is the service time term, and it's the only difference that was ever going to matter.

Notice what's absent from that formula. CPU. Memory. Cores.

You aren't out of compute. You're out of concurrency.

Arriving requests fill a bounded in-memory buffer which is flushed to the database in batches twice a second. Above eighty rows per second the buffer saturates and rejects. arrivals bounded buffer, 40 rows one round trip per batch database flush every 500 ms batches 0 · rows 0 · rejected 0
Count the arcs, then read the row tally. Those two numbers are supposed to disagree.

Accept, acknowledge, drain

The endpoint stops touching the database entirely. It appends to an array in memory and answers, and a timer on a completely separate schedule does the writing.

app.post('/wish', (req, res) => {
  buffer.push(clean(req.body))   // microseconds
  res.status(202).end()          // accepted, not created
})

setInterval(() => {
  const batch = buffer.splice(0, buffer.length)
  if (batch.length) db.insertMany(batch)   // one round trip
}, 500)

That 202 is doing real work. It means accepted, not created, and you haven't created anything yet, so it's the only honest thing to send.

Batching isn't a speed trick. Five hundred rows in one statement isn't five hundred times faster to write. It collapses five hundred round trips into one, and the round trips were the term strangling you.

The edge, and it's the whole cost of the pattern: the buffer is now the thing that fails. Bound it. An unbounded array under sustained overload is just a more expensive way to run out of memory.

Writes travel to the database while reads are served entirely from a static snapshot that is regenerated every five seconds. writes browserapp bufferdatabase regenerated every 5 s reads snapshot every reader gets a file, not a query snapshot age 0.0 s
Follow one read tick. It turns around before it ever gets near the box on the right.

The wall of wishes is a lie

A newspaper is printed once and read by everybody. Nobody's copy is live, and nobody minds.

So the read path never queries anything. A job renders the whole wall to a static file every few seconds, and every visitor gets that file from a cache at the edge, thousands of them, none touching your box.

Which matters more than the write path does. Twenty thousand people submitting once each will also refresh, scroll, and send the link to their family.

The cost is staleness, and it's a real cost. Somebody submits a wish, refreshes, and doesn't see it. That reads as broken.

The fix is free: echo their own wish back into the page from the client the moment you get the 202. They see theirs instantly. Everyone else sees it within the window.

A timeline of buffer flushes with a crash marker. The unflushed span before the crash is the data lost, and it grows with the flush interval. every tick is one flush to the database process dies lost at most 0.5 s of wishes at risk 120 round trips per minute
Drag until the hatched block looks like more than you can stomach. That number is your flush interval.

What you're actually gambling

Letters you've written but haven't posted yet.

Between one flush and the next, those wishes exist in exactly one place: memory, on one machine. If that process dies there, they were never anywhere else.

And the whole gamble is a single number. Shorter interval, smaller loss, more round trips. Longer interval, fewer round trips, bigger hole.

There's a cheap rung between those: append each wish to a local file before you acknowledge it. It survives a process crash, it doesn't survive losing the disk, and it costs one sequential write, which is nothing next to a network round trip.

Above that rung it stops being cheap. A managed queue survives more and costs money, a dependency, and the outage modes of somebody else's service.

Some engineers would call 202-and-buffer unacceptable for anything. For a payment, they're right. For a birthday note, the exposure is a few seconds and the alternative isn't a safer write. It's a stranger getting a connection error and never coming back.

A birthday is the one deadline that doesn't move.

The box was never too small.

It was too small for a server that insists on finishing before it answers.

← All posts