Summary
On Friday, July 19, 2024 at 04:09 UTC, as part of regular operations, CrowdStrike released a content configuration update for the Windows sensor to gather telemetry on possible novel threat techniques.
What caused the outage?
The patch contained a logic error which resulted into a Blue Screen of Death (BSOD) for Windows 11 versions 23H2, 22H2, and 21H2, and also Windows 10 versions 22H2 and 21H2. This BSOD caused a global outage for around 30K customers including healthcare industry, banks and airports.
Unfortunately due to the nature of the issue, it could not be resolved using another patch or automation and required manual system recovery using windows safe mode.
How to know if you are affected?
Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
- Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0409 UTC is the problematic version.
- Note: It is normal for multiple “C-00000291*.sys files to be present in the CrowdStrike directory – as long as one of the files in the folder has a timestamp of 05:27 UTC or later, that will be the active content.
Remediation
Shortly after the outage, CrowdStrike released a list of steps on how to remediate the BSOD.
- Reboot the host to give it an opportunity to download the reverted channel file. Its strongly recommend putting the host on a wired network (as opposed to WiFi) prior to rebooting as the host will acquire internet connectivity considerably faster via ethernet.
- If the host crashes again on reboot:
Updated 2024-07-22 1758 UTC - Option 1 – Build automated recovery ISOs with drivers
- Follow the instructions for Building CrowdStrike Bootable Recovery Images in this manual (PDF) or log in to view in the support portal. Updated 2024-07-26 2105 UTC
- Note: Bitlocker-encrypted hosts may require a recovery key.
- Review the following video on CrowdStrike Host Remediation with Bootable USB Drive.
- Follow the instructions for Building CrowdStrike Bootable Recovery Images in this manual (PDF) or log in to view in the support portal. Updated 2024-07-26 2105 UTC
- Option 2 – Manual process
- Hold the power button for 10 seconds to turn off your device and then press the power button again to turn on your device.
- On the Windows sign-in screen, press and hold the Shift key while you select Power > Restart.
- After your device restarts to the Choose an option screen, select Troubleshoot.
- On the Troubleshoot screen, select Advanced options > Startup Settings > Enable safe mode.
- Restart your device.
Note You may be asked to enter your BitLocker recovery key. When the device restarts, continue pressing F4 and then it will log you in to safe mode. Please note, for some devices, you need to press F11 to log in through safe mode. - Once in safe mode, right-click Start, click Run, type cmd in the Open box, and then click OK.
- If your system drive is different than C:\, type C: and then press Enter. This will switch you to the C:\ drive.
- Type the following command and then press Enter:
CD C:\Windows\System32\drivers\CrowdStrikeNote In this example, C is your system drive. This will change to the CrowdStrike directory.
- Once in the CrowdStrike directory, locate the file matching “C-00000291*.sys”. To do this, type the following command and then press Enter:
dir C-00000291*.sys- Permanently delete the file(s) found. To do this, type the following command and then press Enter.
del C-00000291*.sys- Manually search for any files that match “C-00000291*.sys” and delete them.
- Restart your device.
Recovery methods
If you receive the Windows Recovery screen, use one of the following methods to recover your device.
- Hold the power button for 10 seconds to turn off your device and thenpress the power button again to turn on your device.
- On the Windows sign-in screen, press and hold the Shift key while you select Power > Restart.
- After your device restarts to the Choose an option screen, select Troubleshoot > Advanced options > Startup Settings > Enable safe mode. Then, restart your device.
Note You might be asked to enter your BitLocker recovery key. When the device restarts, continue pressing F4 and then it will log you in to safe mode. Please note, for some devices, you need to press F11 to log in through safe mode. - If the screen asks for a BitLocker recovery key, use your phone and log on to https://aka.ms/aadrecoverykey. Log on with your Email ID and domain account password to find the BitLocker recovery key associated with your device.
To locate your BitLocker recovery key, click Manage Devices > View Bitlocker Keys > Show recovery key. - Select the name of the device where you see the BitLocker prompt. In the expanded window, select View BitLocker Keys. Go back to your device and input the BitLocker key that you see on your phone or secondary device.
- When the device restarts, continue pressing F4 and then it will log you in to safe mode.
- Once in safe mode, right-click Start, click Run, type cmd in the Open box, and then click OK.
- If your system drive is different than C:\, type C: and then press Enter. This will switch you to the C:\ drive.
- Type the following command and then press Enter:
CD C:\Windows\System32\drivers\CrowdStrikeNote In this example, C is your system drive. This will change to the CrowdStrike directory.
- Once in the CrowdStrike directory, locate the file matching “C-00000291*.sys”. To do this, type the following command and then press Enter:
dir C-00000291*.sys- Permanently delete the file(s) found. To do this, type the following command and then press Enter.
del C-00000291*.sys- Manually search for any files that match “C-00000291*.sys” and delete them.
- Restart your device.
Use System Restore
- Hold the power button for 10 seconds to turn off your device and then press the power button again to turn on your device.
- On the Windows sign-in screen, press and hold the Shift key while you select Power > Restart.
- After your device restarts to the Choose an option screen, select Troubleshoot > Advanced options > System Restore.
- If the screen asks for a BitLocker recovery key, use your phone and log on to https://aka.ms/aadrecoverykey. Login with your email id and domain account password to find the bit locker recovery key associated with your device.
To locate your BitLocker recovery key, click Manage Devices > View Bitlocker Keys > Show recovery key. - Select the name of the device where you see the BitLocker prompt. In the expanded window, select View BitLocker Keys. Go back to your device and input the BitLocker key that you see on your phone or secondary device.
- Click Next on System Restore.
- Select the Restore option in the list, click Next, and then click Finish.
- Click Yes to confirm the restore.
Note This will perform just the Windows system restore and personal data should not be impacted. This process might take up to 15 minutes to complete.