The Bug That Traveled at the Speed of Light

I didn't volunteer to work on this problem.

By the time it reached me, it had been open for weeks. Everyone knew about it. Everyone had looked at it. It was quietly burning time and credibility. This was at a shift-scheduling company, a while before I took over the attendance team.

A customer in Germany used the product for delivery drivers. Every Friday at exactly 5:00 PM they published open shifts, and the moment those shifts went live, hundreds, sometimes thousands, of drivers tried to claim them at the same time. And the system broke its own guarantees.

Five open shifts would turn into eight claimed. Or ten. Multiple drivers would successfully "take" the same shift. From the customer's perspective this was a disaster. From ours it was worse: the system believed it was behaving correctly.

What everyone had already tried

Mutexes were added. Guards were layered in. The code read like what you write when you think you're dealing with a race condition. The bug persisted. It only happened under extreme concurrency, only for this customer, and only when the traffic crossed the Atlantic.

The stack: a PHP app in Docker containers on Kubernetes in AWS us-east, nginx and php-fpm in every pod, autoscaled and load-balanced, in front of Aurora MySQL. Standard, and standard is the point. Nothing in it was misconfigured.

The mutexes couldn't work, and it's worth being precise about why. When Germany hit POST /shifts/take, the cluster did what it was built to do and fanned out: more pods, more php-fpm workers per pod. A PHP mutex lives in one process. It doesn't know about the worker next to it, let alone the pod next to that. So forty processes each held their own lock, each felt safe, and each hit the database.

The number that reframed it

I got a free hour, a month after I'd first proposed a fix, and read the raw logs. Millisecond timestamps, which pod handled which request, when state was read, when it was written.

The claims weren't spread across the shift's rows. They were hammering a single row, the shift's remaining-instances count, at two to four thousand requests per second. Requests were arriving a quarter of a millisecond apart.

And the write was the ORM's write. Load the shift, decrement in memory, save. On the wire that is UPDATE shifts SET instances = 4 WHERE id = ?. An absolute value. Two requests read 5, both compute 4, both write 4, and the second one silently erases the first. A lost update. MySQL did exactly what it was told, twice.

Aurora was fine. Every transaction was consistent against the data it could see. The bug lived above the database, in the gap between when a value was read and when it mattered.

Crossing each other over the Atlantic

That's how I pitched it, and it's still the best one-line version.

Every claim produced two things that had to travel:

• The response. Germany to us-east, read the count, decrement, write, and back to Germany with "success, four left." Round trip, roughly 100 ms. Frankfurt to northern Virginia is about 6,000 km; light in fiber does about 200,000 km/s; the physical floor for a round trip is around 60 ms, and real paths are longer.
• The truth. The committed row had to propagate to wherever the next request would read it from. That leaves the writer after the write, and it doesn't get there any faster than the response does.

So the acknowledgement and the updated state left us-east at about the same moment, going in different directions, and the acknowledgement won. The driver saw "four left." The next request, a quarter of a millisecond behind, read "five." So did the next several hundred. Each one was locally correct. Collectively they were wrong.

Where exactly the stale reads came from, I no longer remember. Aurora is one writer and a set of readers, and the readers lag the writer by tens of milliseconds on a good day and up to a second across regions. If the ORM read from a reader and wrote to the writer, every read was stale by that lag on top of everything else. That's the version where "crossing each other over the Atlantic" is literally true.

But I want to be honest about the title. The race exists without the ocean. Two requests in the same rack, two milliseconds apart, doing load-edit-save on the same row, lose updates. Distance didn't cause it. Distance and replica lag widened the window from a few milliseconds to a hundred-plus, and at 2,000 requests per second that's the difference between "six claimed, once in a while" and "ten claimed, every Friday at 5:00."

I walked someone else through it. Slowly. Step by step. They went quiet. Then said: "…oh."

The fix, and the fix I'd do today

The right fix is boring: make the decrement happen inside the database's own lock. UPDATE shifts SET instances = instances - 1 WHERE id = ? AND instances > 0, then check the affected-row count. The subtraction is evaluated against whatever the value is at that instant, under the row lock. Request two gets 3, not 4. Request six matches zero rows and gets a clean "sorry." No pod needs to know about any other pod, because the row lock is the serialization point the system was missing.

That needed a change to how the app talked to the database, and there was no appetite for it right then. The proposal I'd made a month earlier had sat. So I bent the knee and did the fix I could ship alone: I moved each shift's remaining count into a single global Redis instance and made the PHP code treat that number as the only source of truth. Read availability from Redis, DECR, attempt the MySQL writes, roll the counter back if anything fails, commit if everything succeeds. It worked for the same reason the SQL fix works: DECR runs on one single-threaded Redis process, so it's a serialization point. One request wins. The rest see the new value immediately and fail cleanly. MySQL still got updated, because why not, but nothing decided based on it anymore.

The mutexes were removed. PHP is one request per process, but php-fpm runs many processes per container across many containers, so a mutex in one process guards nothing that matters. They weren't just useless; they were misleading. They made the code look safe, which is the worst thing a lock can do.

We deployed. Friday came. 5:00 PM arrived. Nothing broke.

Where it is now

September 2026. I left that company in 2022 and I'd assume the Redis counter is still there, because fixes that work on Friday at 5:00 PM tend to be left alone. What I took with me is the question I now ask on any "impossible" race: not "where's the lock?" but "how far does the truth have to travel, and what beats it there?"