The TLDR: On June 8, vast swathes of the internet suddenly became unavailable for nearly an hour. While the disruption was temporary, it revealed just how dependent we are on a handful of invisible companies for our online access. We explain what happened and why it is worrying.

Say hello to the CDN

The entire outage was the result of a software bug in a Content Delivery Network called Fastly. And here’s what a CDN does:

• When we browse a website, we assume that we are getting the information directly from there. This is almost entirely untrue. 
• In most cases, we are accessing a copy of that content from a server closer to our location—served to us by a Content Delivery Network. 
• Large websites hire a CDN to make and store copies of its content in locations around the world. These locations are called ‘points of presence’ (POPs).
• So when you click on a story on, say, BBC News, your device does not fetch its content all the way from the UK. Thanks to Fastly, it will bring a copy of that content from a POP closer to you.  
• The shorter transit period means that the content shows up a lot faster: “It basically enables really high performance for content, whether that’s streaming video or a site or all the little images that pop up when you go to an ecommerce site… Serving it really close to the user takes away a lot of the load time.”
• Fastly boasts that it has made loading pages on Buzzfeed 50% faster and allowed The New York Times to simultaneously handle 2 million readers on election night.
• CDNs like Fastly also act like traffic cops: “It is like orchestrating traffic flow on a massive road system… If some link on the internet fails or gets congested, CDN algorithms quickly find an alternate route to the destination.”

Here’s a visualisation of how Fastly’s service works:

Point to note: Fastly is a San Francisco-based company that has a $5 billion valuation. It operates POPs in “at least 58 cities around the world, including multiples in densely populated areas like Los Angeles, London, and Singapore. It lists their combined global capacity at a whopping 130 terabits per second.” And it services a number of very large companies including Amazon, Reddit, Spotify, eBay, Twitch, Pinterest—and of course media websites like New York Times, CNN, BBC News etc. 

The mighty software bug

In a blog post, Fastly’s senior VP of engineering explained what went wrong. The company issued an update on May 12—which accidentally introduced a bug in its software. But the bug remained in the system—kinda like a sleeper cell—until a single customer activated it on Monday by changing their configuration. This error was pushed out to multiple servers at the same time, creating a cascading effect. Soon, 85% of Fastly’s networks were returning 503 errors—where you click on a web page and it basically tells you that the server hosting the website can’t service your request for content.

Point to note: Fastly says it detected the problem within one minute, and restored service to 95% of the network in 49 minutes. Its servers were still a little slow for a couple more hours—as the POP servers had to first go back to the original websites to fetch copies of their content.  

A cautionary tale

Losing access to key parts of the internet for less than an hour doesn’t seem like a big deal. But it shows key vulnerabilities in our online access—which is now essential for all sorts of critical services. 

One: That a single company could cripple so many websites in one fell swoop is the consequence of a growing corporate consolidation. There are just a handful of companies that dominate the CDN market: Cloudflare, Akamai and Fastly:

“‘There’s a lot of concentration of usage within very few service providers,’ [Industry expert Angelique] Medina says. ‘Whenever any one of those three providers has an issue, typically it’s not something that lasts a very long time, but it has a major impact across the internet.’

 

That’s a big part, Medina says, of why these sorts of outages have been more frequent of late, and why they’ll only continue to get worse. Baseball needs a cutoff man; intersections need traffic cops. The fewer of those there are to rely on, the more connections get missed, and the bigger the crashes.”

Two: When one company has tens of thousands of servers, it becomes very challenging to anticipate and avoid errors if it introduces an update or change. And when something goes wrong within the system, the effect can be catastrophic:

“You end up with these cascading failures…They’re difficult to debug. They’re stressful and difficult to resolve. And they can be very difficult to detect early on when you’re thinking about making that change, because the systems are so complex and they involve so many moving parts.”

Three: While this outage was caused by a bug, in the future, it could easily be inspiration for a cyberattack, as one cybersecurity expert points out:

"CDNs are part of the internet's critical infrastructure and if threat actors hadn't already cottoned on to this as a direct attack vector to bring down the internet, they will now after monitoring [Tuesday's] misfortunate events.”

And therefore this:

"This in turn—raises major questions about the dangers of (power) consolidation in the cloud market and the unquestioned influence these often invisible actors have over access to information.”

The bottomline: Our internet is less a web than a stack of dominos that will topple in quick succession given the right nudge.

Reading list

Cnet offers the best overview, while Wired and The Conversation do a great job of explaining just how a CDN works. Vox looks at the consequences of a CDN industry dominated by a few players. Also check out: Fastly’s blog post on the outage.