It started being noticed in Australia and was soon felt around the world—a software update from CrowdStrike that prevented Windows PCs from properly booting after restart. Users around the world saw the “blue screen of death.”
“There are two types of updates in the cybersecurity industry,” wrote ThreatLocker CEO Danny Jenkins in statement issued to press. “One is a program update which is normally pushed very, very slowly in batches. If there is a problem, it won’t affect every single customer at the same time.”
He continued, “There are also security and definition updates. Definition updates are pushed as fast as possible to everybody, normally in response to new threats. Everyday there is a new cybersecurity threat. As a response, cybersecurity companies have to create these definitions to allow the agents to know what a threat looks like. They push those updates as fast as possible so everyone gets them before the cyberattack happens. In this case, this was a definition update that was pushed to the CrowdStrike endpoints and shortly after that update was pushed, we saw nearly all of the computers running those updates taken offline.”
Why The Fix Will Take A While
Fortunately, today’s event was not a cyberattack, and it’s a relatively simple fix, says Robert Cioffi, CTO and co-founder of Progressive Computing. “But it requires some technical skill to do it. Even though technical people can walk the average user through the process, it’s time consuming because there’s no way to automate this because the problem and the fix happens before Windows loads.”
That’s why many techs will be working overtime. “How many technicians do you need to screw in this light bulb?” Cioffi jokes. “In this case, you probably need a million more than you have, right? Imagine if I was the only technician available at JFK International Airport and I had to visit all seven or eight of their terminals and visit the hundreds of gates and go to each computer at those gates to do this 5 minute. Do the math.”
CrowdStrike issued this statement on their home page: “CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This was not a cyberattack. The issue has been identified, isolated and a fix has been deployed. We are referring customers to the support portal for the latest updates and will continue to provide complete and continuous public updates on our blog.”
What does this tell us about critical infrastructure’s reliance on software updates? Huntress’ John Hammond, principal security researcher at Huntress, issued this statement: “This is a really interesting case, because ultimately, it’s the vendor (CrowdStrike) that had pushed the changes which broke things… not even the end-user organizations or businesses themselves. Critical infrastructure might have and EDR or XDR solution slapped onto it just to “check the box,” but the scenario where the provider accidentally breaks the infrastructure isn’t one you ever really think of. I think it is staggering to see just how widespread the issues were across different sectors and industries—it is quite a shock to see one of the most well-regarded vendors be the focal point here.”
Time For Some Talking Points On Preparedness
Henry Timm, CEO of Phantom Technology Solutions, suggests MSPs can use the incident as a talking point with customers. “Establish or re-evaluate your Incident Response plan. What would you do if your systems went down like this? Do you have a written plan?”
Cioffi agrees. “This is what I try to instigate with any business that I speak to, especially senior leaders or owners of businesses. … How do you operate your business when there’s no power or when there’s no Internet, or when telephony is down or Microsoft email is down or I can’t get to my OneDrive? Or, in this particular case, I can’t even turn on my computer. So these are the things that I think we ought to be thinking about and it helps prepare us for the bigger issues, whether they be a natural disaster or caused by malicious humans.”
For now, you may want to have pizza delivered to your team. And Timm suggests this: “Have empathy. This could happen to anyone. This is going to likely be a multi-week, manual restoration with many long hours.”
