Image Source: Unsplash

If you do business on Microsoft (MSFT) computers with CrowdStrike (CRWD) software, you had one hell of a day today. Millions of people and thousands of major businesses were impacted.

Major IT Outage Grounds Flights, Hits Banks and Businesses Worldwide
 

The Wall Street Journal reports Major IT Outage Grounds Flights, Hits Banks and Businesses Worldwide

A massive tech outage swept the globe, knocking out operations for banks, media companies and emergency services and forcing airlines to ground flights, exposing the fragility and interdependence of global digital technology.

A single update from cybersecurity-software company CrowdStrike (CRWD), a major provider of malware and virus protection to a large array of companies, caused outages for millions of users of Microsoft (MSFT) Windows devices worldwide. Major airlines including Delta Air Lines, United Airlines and American Airlines halted departures overnight.

The outage touched almost every industry. Multiple financial institutions, government entities and corporations reported tech issues. Some hospitals and school districts said computers were down. Still, financial markets were largely operating as normal, and many companies said systems were starting to be restored.

That one update from a single provider could plunge so many companies—from airlines’ check-in desks to consultants’ conference rooms—into a digital dark age serves as fresh warning of the world’s technological dependence. 

One of CrowdStrike’s main services is called Falcon, which monitors a company’s machines for hacking attempts, viruses and other threats. The Austin-based company has about 29,000 customers and went public in 2019.

CrowdStrike told customers in a status update seen by The Wall Street Journal that the problem was with a software change it had pushed via Falcon out to clients’ computers. The company said its engineers had undone the change but clients would need to use a workaround to download a fix to affected computers.

Some affected users may be back up and running soon, but for others it could take weeks depending on the system in use, said Simo Kohonen, founder of Finland-based network security company Defused. “The fix CrowdStrike has given is quite manual and may be difficult, in some cases, to deploy at large scale,” he said.

At Britain’s National Health Service, the outage disrupted the appointment and patient-record system at most doctor’s practices—leading to handwritten prescriptions. In India, at least some passengers received handwritten boarding passes.

In the U.S., many 911 and nonemergency call centers were disrupted. In New York City, some screens showing subway arrival times weren’t working.

Visa said it was aware of reports of people unable to make payments. Some JPMorgan staff had trouble logging on. Charles Schwab said certain online systems may be unavailable.

More than 27,000 flights around the world were delayed and 2,800 were canceled by midmorning Friday, according to flight-tracking website FlightAware.

CrowdStrike Statement

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We…

— George Kurtz (@George_Kurtz) July 19, 2024

Sorry

EXCLUSIVE: CrowdStrike founder and CEO @George_Kurtz speaks on TODAY about the major computer outages worldwide that started earlier today: “We’re deeply sorry for the impact that we’ve caused to customers, to travelers, to anyone affected by this.” pic.twitter.com/fWz6KhgrcZ

— TODAY (@TODAYshow) July 19, 2024

Sorry is what you say when you accidentally kick the cat. Sorry does not cut it. Expect lawsuits.

Blue Screen of Death – What to Do
 

As IT departments, Microsoft and CrowdStrike work to fix the outage, many people are still struggling to get their systems going and get rid of that “blue screen of death.”

Unfortunately, there may be no easy fix for most of us. It requires computer savviness or help from your IT department.

If you want to tackle this yourself, you can first try to reboot your computer. Microsoft says that sometimes helps but could require as many as 15 restarts before it takes.

If that doesn’t work, next check (or remember) if your hard drive has been encrypted with Microsoft’s BitLocker technology. If so, you’ll need to access your BitLocker recovery key to unlock it, says Grant Geyer, chief strategy officer of Claroty, a cyber defense company that recently raised $100 million in funding.

Once your hard drive is unencrypted, you can try to take your computer back to the version before the faulty update. Reboot your computer in safe mode and open up a command prompt in the search bar of your Windows screen. Type c:windowssystem32driverscrowdstrike and then type del C-00000291 to delete it. Hit Tab, then Enter and you should be back up and running, says Geyer.

If you’re not able to try the workaround yourself, it could take some time for your IT department to do it.

CrowdStrike Incompetence
 

I was in IT for over 20 years. We had a rollback plan for every update.

I was not in networking. I was in major bank applications on the technical end of things. My users were not end users but rather programmers who needed technical assistance to implement business needs.

Even a two-hour outage rollback was a disaster. We did changes on weekends, late at night, to minimize impact.

Answers Please
 

1. Are system changes now that complex that they can’t be rolled back for weeks?!
2. Did anyone test this?

The answer to number two is obvious. It’s either no, or the testers were grossly incompetent.

The answer to number 1: It’s idiotic to knowingly run software that cannot be rolled back for days, perhaps weeks if there are issues.

CrowdStrike offers protection against malware. Now we see that it needs to protect computers from its own updates.

More By This Author: