We Spent Days Debugging the Wrong System (and Accidentally Fixed Minnesota's Internet)

I wasn't on the initial page.

This was a few years into a frontend lead role at a company I'll leave unnamed. By the time I got pulled in, the incident had been running for a while and everyone thought they knew what it was. The frontend was broken. Pages weren't loading, some users saw blank screens, static assets failed intermittently. It looked like a bad UI deploy, so that's where everyone went.

Managers got pulled in. A backend Java dev tried to help. The CTO was in and out. Engineers would show up, poke around, disappear for a few hours, come back. This went on for a couple of days. Nothing changed.

Why it looked like a frontend bug

Our pages were a single-page app. The HTML was tiny; the first real request was the JS bundle from the CDN, and that bundle was what kicked off every other asset download. If the bundle didn't arrive, you got a blank screen. If the bundle arrived but a chunk behind it didn't, you got a half-rendered page and a console full of errors that pointed straight at our code.

So every symptom looked like ours. That's the trap: the delivery path was failing in a way that only ever surfaced as a frontend error.

The thing that didn't fit

First thing I did was skip the code and open the network tab. Some requests worked. Some didn't. Refresh and it might fix itself.

That didn't sit right. If a bundle is bad, it's bad for everyone. If a deploy is broken, it fails every time. This was flickering.

We still went through all the Cloudflare stuff. Cache rules, purges, page rules, edge config. Nothing wrong with any of it. And once nothing is wrong with any of it, you start telling yourself the bug is just subtle, and you keep looking in the same spot.

Grouping by the ID nobody reads

Cloudflare sticks a Ray ID on every response. I'd always ignored them. This time I started grouping the failures by it.

Different users, different times, same Ray path.

If the UI code was broken, the Ray IDs should have been all over the place. They weren't. Same failures, same route.

The accidental experiment

Here is the part I'd never have designed on purpose. We were a remote company, spread across several states, and the incident channel was a Zoom call. Somebody in one state couldn't load the site at all. Somebody two states over had it fine. A third person had it fail every third refresh.

That's a geographic experiment running for free. We asked more people around the company to load the site. Different states, different ISPs, different machines. Some couldn't reproduce it. Others hit it immediately.

Failures clustered by geography. Then by provider.

A lot of traffic in and out of the Twin Cities goes through a small number of interconnects and carrier hotels. ISPs and backbone providers peer there and hand traffic to CDNs like Cloudflare. When one transit provider in that path goes bad, packet loss, flaky peering, MTU problems, you don't get a clean outage. You get weird partial failures. Static assets fail, HTML loads, retry works, some users are fine, others are completely broken.

We traced the paths. There were a handful of transit legs between our users and Cloudflare, and every failure went through the same one. It belonged to a national carrier I'll call Hear-Me-Now, and not their wireless customers. Hear-Me-Now as a transit hop, one of the few between users and the CDN, intermittently dropping.

Going up the chain

Our CTO also ran a Twin Cities data-center business, so he knew how traffic actually moves through Minnesota and had relationships with the people running that infrastructure. We showed him the Ray IDs, the provider correlation, the clean separation between failing and non-failing paths. His other company was already looking at problems on a Hear-Me-Now transit leg. What we had matched.

They pulled Hear-Me-Now out of the routing mix.

The incident stopped. No deploy, no rollback, no config change. Site loaded.

The actual lesson

A carrier having a problem isn't the interesting part. That happens. The interesting part is how long we spent debugging a system that wasn't broken. Once "frontend bug" became the assumption, that's all anyone could see. The moment we thought about the delivery path instead, it was obvious.

What I do differently now, on any "it's intermittent" incident:

Ask "is it the same for everyone?" in the first hour, not on day three. If the answer is no, stop reading code.
Group failures by whatever request ID the edge gives you. Ray ID, request ID, trace ID. It costs nothing and it either confirms or kills the "our code" theory immediately.
Use the fact that your team is spread out. A remote company is a distributed probe network. Ask ten people in ten places to load the page before you ask one person to read the bundle.

I've told this story enough times that the details have worn smooth. I'm no longer certain whether we correlated on Ray IDs or on something the CDN logs exposed, and I couldn't tell you whether it was one leg in seven or one in eleven. What I'm certain of is the shape of it: two days inside the wrong system, twenty minutes once we stepped outside it.