A CrowdStrike executive told a US Congressional hearing on Tuesday that the company’s endpoint detection and response sensor has to continue accessing the Windows kernel, despite criticism by some cybersecurity experts that the kernel access contributed to the crash of millions of Windows computers around the world in July.
Access to the kernel by cybersecurity products helps protect operating systems from being tampered with, Adam Meyers, CrowdStrike’s senior vice-president of counter adversary operations, told the House of Representatives subcommittee on cybersecurity and infrastructure protection.
Members of the subcommittee are looking into the chaos around the world caused in July by a buggy CrowdStrike update configuration file. They started Tuesday’s hearing today with ominous words.
“The sheer scale of this error was alarming,” said Andrew Garbarino, chair of the subcommittee. The incident, which knocked 8.5 million Windows computers and servers offline, created an environment “ripe for exploitation by malicious cyber attackers through phishing and other efforts,” he said.
However, Meyers defended the company’s stand.
“Anti-tampering is very concerning, because when a threat actor gains access to a system, they would seek to disable security tools. And in order to identify that’s happening, kernel visibility is required. The kernel driver is a key component of every security product I can think of. Whether they would say they do most of their work in the kernel or not varies from vendor to vendor. But to trying to secure the operating system without kernel access would be very difficult.”
As CrowdStrike has stated earlier, the problem was in a configuration file update to the company’s Falcon sensor that is deployed in servers and PCs, and not a software code fix for Falcon. Software code fixes, he said, underwent a more rigorous pre-release testing process than the configuration files. The July problem was that one configuration file had a mistake in it that he likened to a command to move a chess piece to a square that didn’t exist. Falcon sensors looking for a line that didn’t exist reacted by crashing Windows
“This was a perfect storm of issues that resulted in the sensor failure,” Myers said. “We are deeply sorry and are determined to prevent it from happening again.”
It couldn’t happen again, he said, for two reasons:
- CrowdStrike is more rigorous in testing configuration updates, which are released about 10 times a day;
- Falcon administrators now have the option of installing configuration updates when they want to. That should eliminate the risk of all CrowdStrike customers’ Windows systems being knocked offline at the same time, as happened on July 19.
CrowdStrike customers include 538 Fortune 1,000 companies, 298 Fortune 500 firms, and 43 of 50 US states.
A congressman told the hearing that one insurance company estimates 25% of F500 firms around the world were affected, with firms suffering an estimated US$5.5 billion in losses.
Meyers avoided directly answering a question by Representative William Timmons about “making whole” victims such as travellers whose flights were cancelled. He responded that CrowdStrike worked with customers to get their IT systems up and running quickly. About 99% were restored by July 29.
Representative Eric Stalwell said he appreciated that CrowdStrike wants to protect customers against novel threats, but said “speed [of releasing updates] cannot come at the cost of operability.”
The chair of the Homeland Security Committee, Representative Mark Green, said, “a global IT outage that impacts every sector of the economy is a catastrophe you’d expect to see in a movie.
“To add insult to injury, the largest IT outage in history was due to a mistake” by a security vendor, he added. “It also appears that the update may not have been appropriately tested before being pushed out” to the Windows kernel of CrowdStrike customers.
“Mistakes happen,” Green said. “However we can’t allow a mistake of this magnitude to happen again.”
But when it came to questioning Meyers, the congressmen spared the whip.
After Meyers read an opening statement apologizing for the mess made by the July 18 update, which referred to a detailed technical report analysis, Green thanked him.
“There was a degree of humility that is impressive, and I appreciate the transparency that we have seen. I think some of the biggest lessons we learn come in times of adversity. You guys have shown the right attitude” in being open.