The recent CrowdStrike outage serves as a critical lesson for the
cybersecurity and IT community at large. INE Security, a global
provider of cybersecurity training and certification, is among the
industry leaders now examining how impacted businesses can come
together to ensure business continuity and security. CrowdStrike, a
leader in cloud-delivered endpoint and cloud-based workload
protection, faced a significant challenge when a problematic
Microsoft update led to widespread system disruptions for its
users. This incident underlines the complexities and potential
risks inherent in maintaining robust cybersecurity defenses and
provides a roadmap for how businesses manage outages going forward.
Overview of the Outage
The trouble began with a routine software update intended to
enhance security features. The update contained flawed code that
inadvertently caused Windows to experience a Blue Screen of Death
(BSOD) on any system that received the update. As systems went
down, companies found themselves unable to access vital data and
services, leading to operational disruptions and increased
vulnerability. CrowdStrike was able to immediately identify the
issue and worked quickly to deploy a fix in 79 minutes. Since then,
the company has worked directly with affected organizations to
implement fixes and mitigate the damage.
CrowdStrike executives acknowledge the immediate impact of the
outage was profound. Businesses of all sizes experienced downtime,
and the ripple effects on productivity and security were felt
widely. All said and done, approximately 8.5 million devices were
impacted, and Fortune 500 companies are estimated to have suffered
upwards of $5.4 billion in losses. The outage, while
well-intentioned and not malicious in origin, also opened the door
to opportunistic attacks. Cybercriminals swiftly exploited the
situation, deploying deceptive websites to distribute harmful
“updates” and causing even more chaos, severely disrupting the
supply chain and causing widespread concern and even panic in some
cases.
"In recent times, we've witnessed the profound impact of supply
chain vulnerabilities in cybersecurity incidents, reminding us of
the critical need for comprehensive cybersecurity training,” says
Brian Olliff, Cybersecurity Instructor at INE Security, a global
leader in cybersecurity training and certifications. Olliff points
to the SolarWinds incident as another stark and costly example of
seemingly benign software updates turning malicious and
compromising countless systems. “These incidents underscore a
crucial point: proper cybersecurity training can empower both
administrators and end users to detect and thwart such attacks
effectively. Cyber teams must be equipped with the knowledge to
navigate and secure against these intricate threats, ensuring
they're not just participants in cybersecurity, but proactive
defenders within their organizations."
The Importance of Swift Deployment and
Vigilance
Despite the fallout, the decision by companies to deploy the
update was fundamentally correct. In the fast-evolving landscape of
cyber threats, keeping software up-to-date is crucial. Updates
often include patches for vulnerabilities that, if left
unaddressed, could expose organizations to data breaches and
attacks. The issue in this instance was not the policy of regular
updates but rather the unforeseen errors within the update
itself.
Mitigating the Damage Through Manual Resets
When the systems went down, the immediate solution involved
manually rebooting Windows into Safe Mode, deleting a file, and
rebooting to restore functionality. One of the complications from
this process involved systems with BitLocker enabled, as fixing the
CrowdStrike issue triggered BitLocker's recovery mode, requiring
users to enter a lengthy recovery key. For many businesses, these
keys were stored on servers that were also affected by the outage,
making it difficult to access them and prolonging the recovery
process.
“This outage highlights the critical need for robust incident
response strategies,” says Brian McGahan, who is a CCIE in Security
and Director of Networking Content at INE Security. “Manual
resets are a stopgap measure, not a solution. They involve
significant labor and can lead to further issues if not performed
correctly. Organizations that had clearly defined procedures and
well-trained IT staff were able to minimize downtime more
effectively compared to those less prepared.”
The emphasis going forward, McGahan says, must be on both
preventing such outages and preparing to manage them efficiently
when they occur, ensuring strategies are in place for procedures to
follow for network and system outages.
Training as a Critical Defense
One of the key takeaways from the CrowdStrike outage is the
importance of proper training to respond to incidents like this.
Even with the best of intentions and high-level technical
expertise, incidents are going to happen. IT teams with
advanced training in network management and incident response were
better equipped to handle the challenges posed by the outage. They
could quickly diagnose the problem, understand the implications of
the flawed update, and implement manual resets and other corrective
measures effectively, ensuring business continuity in the face of
massive security challenges.
Key strategies for ensuring business continuity include:
- Rapid Assessment and Response: Immediate
identification of affected systems and deployment of manual resets
or patches to restore functionality.
- Cross-Departmental Communication: Keeping all
parts of the organization informed and coordinated in their
response efforts.
- Leveraging Redundancy: Using backup systems
and protocols that can operate when primary systems fail.
Training, however, should not be limited to technical staff
alone. All employees can benefit from understanding the basics of
cybersecurity. This knowledge can empower them to act swiftly and
appropriately in times of crises like this, reducing the potential
for human error, which often exacerbates the issue. Additionally,
it helps everyday users prevent cybersecurity attacks by improving
their overall awareness and response to potential threats.
Ongoing Monitoring and Security Post-Outage
The restoration of systems is just the beginning; ensuring they
remain secure against further issues is crucial. Teams must remain
vigilant, continuously monitoring systems to detect and address
vulnerabilities. Some effective measures include:
- Enhanced Surveillance: Implementing real-time
monitoring tools to detect unusual activity that could indicate a
breach or new vulnerabilities.
- Regular Audits: Conducting security audits to
ensure all systems are secure and any detected anomalies are
swiftly addressed.
- Update Rollout Oversight: Managing the
deployment of updates carefully, including staged rollouts and
thorough testing, to avoid incidents similar to the CrowdStrike
incident.
Long-term Implications and Lessons
The CrowdStrike outage is a potent reminder of the complexities
involved in cybersecurity management. It highlights the need
for:
- Robust Testing Protocols: Before deploying
updates, especially those that are widespread and critical, robust
testing must be a standard procedure to catch potentially
disruptive bugs.
- Incident Response Planning: A well-defined and
practiced incident response plan can drastically reduce the time
and impact of system outages.
- Continual Learning and Adaptation:
Cybersecurity is not static. Continuous learning, updating, and
testing are essential to adapt to new challenges.
- Comprehensive Training: Training should be
comprehensive and ongoing to prepare all levels of an organization
to respond effectively to cybersecurity incidents.
Conclusion
The CrowdStrike outage provides a valuable learning opportunity.
By understanding what went wrong and how different companies
responded, the cybersecurity community can improve its practices
and strategies. Most importantly, it underscores the critical
importance of balancing proactive security measures with rigorous
testing and training to safeguard against the unpredictable nature
of cyber threats.
About INE
Security:
INE Security is the premier provider of online networking and
cybersecurity training and certification. Harnessing the world’s
most powerful hands-on lab platform, cutting-edge technology,
global video distribution network, and world-class instructors, INE
Security is the top training choice for Fortune 500 companies
worldwide for cybersecurity training in business, and for IT
professionals looking to advance their careers. INE Security’s
suite of learning paths offers an incomparable depth of expertise
across cybersecurity and is committed to delivering advanced
technical training while also lowering the barriers worldwide for
those looking to enter and excel in an IT career.
Press Team
INE
917-715-0911
Press@ine.com