From “Business Blunders,” a timely and entertaining newsletter by esteemed business writer, editor, and friend, Al Lewis. ~ Ilene
Canary in a CrowdStrike
Cybersecurity giant’s blunder is a warning for the whole world
Courtesy of Al Lewis, Business Blunders
“A full-blown rollout from a security vendor to every customer within minutes is very dangerous.” – Dave DeWalt, former CEO of cybersecurity company McAfee, in a recent Wall Street Journal article.
CrowdStrike has stumbled upon a obvious insight after its buggy software update shuttered governments, businesses, hospitals, airlines and consumers around the globe – leading to billions in damages.
The cybersecurity giant calls it “canary deployment.” This means gradually rolling out software updates in the future so that the company can check for looming catastrophes.
This way, not every Microsoft Windows user gets the dreaded “blue screen of death.” This way, only one airline shuts down instead of several of them. This way, only a few hospitals take a hit while CrowdStrike fixes its glitch.
Somehow, coming to this revelation is costing the company billions in market value (nearly $23 billion as of 7-26-24), as well as an unknown amount in future restitution and damage to its otherwise stellar reputation.
Not using canary deployment is like staging an outdoor rally for a presidential candidate – a risky proposition in itself – and not checking rooftops for snipers.
Thousands are paying the price. CrowdStrike should not have risked leaving all of its customers CrowdStuck.
Losses
CrowdStrike said 8.5 million computers running Microsoft Windows took the hit. CEO George Kurtz, has been summoned to testify before Congress to explain. Global financial losses from the debacle have been estimated as high as $15 billion, with much of it uninsured.
The following chart is from Parametrix Solutions, Inc.’s in-depth report on the financial losses caused by CrowdStrike’s costly mishap:
The hottest software stock of 2024?
CrowdStrike stock has plunged nearly 30% since its disaster unfolded last week. Millions of Americans who never heard of CrowdStrike are aware of it now, after being stranded at airports, locked out of their computer systems, and stymied in business.
Before all this happened, CrowdStrike was a Wall Street darling. For example, Downtown Josh Brown called it the “hottest software stock of 2024” on July 8, and gave Kurtz an absolutely cloying interview.
Brown is a money manager and frequent bloviator on CNBC. He’s smart, quick-witted and has a tight grip on the data, but like a lot of talking heads on CNBC, he talks his book. Following the crash, he wrote a piece on why he’s not selling his Crowdstrike shares.
For now, though, CrowdStrike has put the “down” in Downtown Josh Brown.
The lesson here is that you can lose a lot of money getting your stock tips on platforms where optimism bias abounds and nobody adequately explores the risk factors.
What could possibly go wrong? Everything, everywhere, all at once.
This will keep happening
This has happened before and it will happen again because global corporations are increasingly centralizing everything on computer systems.
A recent column from MarketWatch recalls a 2021 glitch from Amazon’s cloud services that crippled websites around the world:
“As tech companies get bigger, they create more ‘single points of failure’ that can lead to cascading catastrophes across our connected world.”
A case of the butterflies
We live in a complex web of cause and effect and the digital world puts us all in great peril. As big tech consolidates, it creates fewer options when global resilience requires more. The latest revolution in artificial intelligence will only accelerate this trend.
If you’d like to learn more about how the tiniest of actions can lead to enormous outcomes, I recommend Fluke: Chance, Chaos, and Why Everything We Do Matters by political scientist Brian Klaas.
Klaas recently wrote a Substack post observing that the CrowdStrike debacle is a warning of things to come.
He writes:
“There is often a trade-off between maximum optimization and resilience.
“Our endless worship at the Altar of Endless Optimization, in which we aim to squeeze every last drop of inefficiency out of our social systems, is making us more prone to disaster.
“Can we really trust our species to flawlessly govern unimaginably complex systems – systems we don’t always fully understand – that can be brought down by a single screw-up?”
Like the proverbial wings of butterflies spawning hurricanes, a software developer farts and the whole world gets gassed.
We’re going to need a lot more canaries.
*****
Al Lewis has written for The Wall Street Journal, Dow Jones, CNBC, Houston Chronicle, Denver Post, Rocky Mountain News, and until recently, The Messenger – one of the biggest blunders in digital media history — you can read My Latest Blunder (The Messenger) here in case you missed it.
Subscribe to Al’s Business Blunders Newsletter to keep up with the most spectacular business blunders as they, inevitably, arise. >
