While technology outages are catastrophic for many businesses, they can still have silver linings. In the case of the recent CrowdStrike outage, the silver lining comes in the form of key learnings businesses can gather and implement to strengthen or reinforce their existing disaster recovery strategies. In some cases, the takeaway may even be the detriment of not having these strategies in place.
This article will touch on top lessons learned that may be helpful in assessing the state of your current disaster recovery strategy – and planning for the next unlikely incident.
The Need For a Sound Disaster Recovery Strategy
Disaster recovery strategies are comprehensive plans designed to ensure an organization's IT infrastructure and operations can quickly recover and continue functioning after a catastrophic event – like the recent CrowdStrike outage.
Businesses without such a plan in place likely experienced significant challenges and disruptions including extended downtime due to lack of procedures in place to quickly restore their systems and services, data loss without regular and reliable backups, and a resulting damage to customer trust and satisfaction.
By implementing a robust disaster recovery strategy, organizations can ensure they are well-prepared to handle disruptions, maintain operational resilience, and protect critical assets and data. The primary goal is to minimize downtime and data loss, ensuring business continuity and maintaining trust.
The Importance of Site Reliability Engineering
Site Reliability Engineering (SRE) integrates principles of software engineering and applies them to infrastructure and operations to improve observability. SRE facilitates a strategy to get ahead of disaster recovery. As your observability and automation increase, your reliability increases. Through comprehensive monitoring and tracking systems, insights into application performance and behavior are gathered then harnessed in an automated way to ensure quick recovery from what could otherwise be a disastrous situation. By building and maintaining reliable, scalable, and efficient systems to handle routine tasks, incident response, and system recoveries, human error is reduced and faster, more consistent resolutions are achievable.
Suppose you're administrating the quintessential perfect storm of an infrastructure environment. We're talking about a thousand, or thousands of AWS EC2 servers that are not capable of being part of a scaling group due to various constraints. In this hypothetical environment, you may choose to restore every EC2 from its snapshot. Determining the associated snapshot, creating a volume, replacing the attached volume on your EC2, and bringing back up the box would be quite the task.
So, how can a tremendous task like this be eased? Automation...
Any repetitive task should ultimately be addressed with automation, or tooling that's as close to automated as possible. In the given example, writing a CLI application that interacts with the AWS CLI would be a tremendous help. We're talking about reducing man hours to a fraction of the time.