Outage Chaos: Who Turned Off All The Computer?

by · Electronic Frontier Foundation

It's been a tough couple of weeks for anyone trying to use email, fly, spend money, use an Apple device*, or do signals intelligence. Technical service outages are nothing new, but why do so many big ones seem to be happening all at once? Are hackers, AI, and international conflicts to blame? And what can we do about it?

Since most service interruptions don't make the news, it's hard to get a baseline on how often we see them happen normally. But after a major incident like the Microsoft cloud outage on November 12, any smaller disruption is more likely to become a story as part of a pattern instead of an isolated incident. We would have heard plenty about the Air Traffic Control problems anyway, but the November 18 disruption to the PlayStation Network might have gotten less attention. There might appear to be a spike or an upward trend in outages, but we can't easily verify that, and the increased coverage will tend to make it seem more significant and interconnected than it necessarily is regardless.

One boring reason we might see an uptick in problems right now is the date on the calendar. With Thanksgiving travel, Black Friday, Cyber Monday, Giving Tuesday, and Surveillance Sunday (we assume) all coming up, organizations often schedule major system updates for the first three weeks of November. This helps them prepare for holiday traffic, and lets them shift the outage risk to the time before those dates when they may consider it far more important to maintain reliable service.

While none of these particular outages have yet been directly attributed to AI (or to malicious hackers), bank CEOs and the NSA briefed congress and reportedly attributed at least some of them to rushed attempts to patch vulnerabilities found by AI. That’s understandable: all software updates carry risk, and companies and governments are having to deploy more updates, and faster, than they’ve ever needed to before. Some of those updates are to important infrastructure, where an outage would be very disruptive, but the impact of leaving vulnerabilities unpatched would be even worse.

Vulnerabilities are a type of software bug that affects security. Once a vulnerability is found in a given piece of software and disclosed, typically the software maker will release a patch fixing that vulnerability. That’s why so many update notes say things like “performance improvements and security fixes” — the security fixes are patches to vulnerabilities.

Companies and governments that run infrastructure face a challenging tradeoff: deploy patches quickly or slowly and cautiously? Patching quickly is important to keep systems secure. But every software update runs a risk: any change to a deployed system can have unintended consequences that snowball into a disastrous outage.

This exacerbates a much deeper problem long discussed in the engineering culture of many, if not most organizations. Keeping networks, servers, and applications not just secure but resilient in the face of patches, reboots, and capacity changes requires an investment in staff, time, hardware, and processes for frequent maintenance, redundant systems, realistic staging environments, automated rollbacks, monitoring, and pre-deployment testing. These investments are rarely as attractive to decision-makers as a new product or feature, or at times, sadly, as layoffs.

This should come as no surprise when we think of our apps, web sites, airlines, and bank cards as supported by electronic infrastructure. As in the more visible physical world, maintenance can only be deferred for so long before the costs become immediate and dramatic. We accept the cost of an occasional short term bridge closure to reduce the risk of a bridge collapse, and highway departments schedule road work nearly every night of the year to repair and prevent the worst potholes.

When this kind of work is endlessly deferred in the world of software, we speak of technical debt. This can show up as routine steps that have to be done by hand instead of automatically on a schedule, a programmer's shortcut that works well in one specific situation but is more fragile in general, an operating system six months (or years) overdue for a reboot, or a piece of networking hardware no longer supported for patches by its manufacturer. On the other hand, when maintenance work is well-resourced and routine, it should be rare for even major security updates to cause significant issues.

What’s needed in the short term is a big investment in the people and training needed to roll out patches promptly and safely. What’s needed in the long term is a big investment in software security, eliminating entire classes of vulnerabilities, like memory safety, before they are ever written.

For organizations that have gotten used to patching slowly and cautiously, suddenly switching gears to patching quickly is a big challenge. And the flood of vulnerability disclosures enabled by AI means that everyone is being asked to switch gears like that at once.

Fortunately, it’s not a zero-sum game: patches can be rolled out quickly and cautiously, with investments in reducing technical debt and expanding maintenance budgets as described above. These measures are expensive and time-consuming, but necessary. For governments asking how to respond to AI-enabled cybersecurity threats, investing in the people and training necessary to stay on top of patches will pay big dividends in reliability and security.

For individuals, consider the trade-offs when relying on a single provider for many services you rely on. Consumer cloud and device offerings tie together many of the things we need to function on a daily basis like email, messaging, file storage, and even payments, but that convenience and simplicity comes with the risk of losing access to it all if that service goes offline (or suspends your account). Trying out interoperable alternatives is a great way to reduce your reliance on a single company.

Finally, for any critical data, learn the 3-2-1 rule for backups: Keep three different copies on two different types of storage with one copy in a different physical location. For many users, a laptop or other personal device, cloud storage, and a local backup disk exemplify this rule, as long as the laptop stores data on its own drives, not only in the cloud.

*Apple is an EFF donor. EFF donors do not influence our agenda or our positions.