Update from CrowdStrike causes Windows systems to crash

A defective component in the recent CrowdStrike Falcon update is causing Windows systems to crash, disrupting organizations and services worldwide, including airports, TV stations, and hospitals.

The issue is affecting both Windows workstations and servers, leading to extensive outages that have incapacitated entire companies and networks comprising hundreds of thousands of computers.

Some reports indicate that emergency services in the U.S. and Canada have also been affected.

Workaround for CrowdStrike glitched update

Over the past few hours, users have reported that their Windows hosts are either stuck in a boot loop or displaying the Blue Screen of Death (BSOD) after installing the latest CrowdStrike Falcon Sensor update.

The security vendor acknowledged the issue, releasing a technical alert explaining that its engineers “identified a content deployment related to this issue and reverted those changes.”

“Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor,” CrowdStrike states in the alert.

The company identified the issue as a Channel File, which contains data for the sensor (e.g., instructions). As this file is just a component of the update, it can be addressed individually without removing the entire Falcon Sensor update.

For those already affected, CrowdStrike suggests the following workaround:

1. Boot Windows into Safe Mode or the Windows Recovery Environment.
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate and delete the file matching “C-00000291*.sys”.
4. Boot the host normally.

George Kurtz, President and CEO of CrowdStrike, announced that the company “is actively working with customers” and confirmed the problems are due to “a defect found in a single content update for Windows hosts”

“We recommend organizations communicate with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of our customers,” said George Kurtz.

CrowdStrike’s CEO also stated that a fix is available and advises customers to check the support portal for the latest updates.

Worldwide outage

By the time the correction was made, numerous large organizations across various sectors had already been affected.

Reports indicate that CrowdStrike’s update impacted 911 emergency service agencies in New York (EMS, police, fire department), Alaska, Arizona, and parts of Canada.

A 911 telecommunicator in Illinois mentioned that they were “working off of paper until things come back.”

In Catalonia, Spain, the health hotline was affected, prompting authorities to urge citizens to only call 061 in emergencies.

Dutch broadcaster NOS reported disruptions at Schiphol Airport, causing several KLM and Transavia flights to be grounded.

Melbourne Airport experienced “a global technology issue impacting check-in procedures for some airlines,” with Jetstar and Scoot passengers most affected.

Other airports in Berlin, Barcelona, Brisbane, Edinburgh, Amsterdam, and London also faced issues.

Zurich Airport recently announced that while inbound flights already in the air could land, no aircraft were currently taking off for Zurich, and there were no U.S. departures. Manual check-in procedures led to delays and cancellations.

In the U.S., the Federal Aviation Administration received requests to assist airlines (American Airlines, United, Delta) with ground stops until the IT issue was resolved.

Hospitals in the Netherlands, including Scheper in Emmen, Slingeland Hospital in Achterhoek, and emergency posts in Hoogeveen and Stadskanaal, were impacted.

In Barcelona, Terrassa University Hospital and the Catalan Oncology Institute faced issues earlier today but have started to resume normal operations.

On Friday morning, several television stations and news outlets, such as Sky News and ABC, experienced disruptions due to crashing computers.

On Reddit, many users expressed their frustration about tens of thousands of computers crashing after CrowdStrike’s update and the resulting impact on their companies:

• “Malaysia here, 70% of our laptops are down and stuck in boot. HQ from Japan ordered a company-wide shutdown.”
• “210K BSODS all at 10:57 PST… and it keeps going up… this is bad…”
• “Workstations and servers here in Aus… fleet of 50k+ – someone is going to have fun.”
• “Failing here in Australia too. Our entire company is offline.”
• “Same here in OZ. Entire company is down.”
• “Half the company down. Somehow it has hit our AWS servers also. Major service downtime for our customers.”
• “Entire org and trading entities down here. Half of IT are locked out.”
• “Seeing major issues here in NZ at the moment, company-wide outage impacting servers and workstations.”
• “Supporting Philippines and China Locations. All experiencing the same as well.”

Despite a fix being deployed and CrowdStrike providing a workaround for Windows hosts already crashing, companies will feel the effects from the issue for a while.

Admins are likely to face a long weekend, especially with computer fleets of tens or hundreds of thousands of computers, employees working remotely, off-premise data centers, or cloud environments where booting in safe mode is not an option.

Source: BleepingComputer, Ionut Ilascu