As a follow up to the three disaster recovery (DR) best practices post, I want to provide some real life examples of what can happen when you don’t have a DR plan in place. Over my the course of my career in IT, I have witnessed each of the situations first-hand while working for a previous employer. In each of these situations, not having a Disaster Recovery plan severely impacted the business’s ability to operate, or caused it to close completely.
Situation 1 - Flat File Backups
A customer had been using a backup solution that only did flat file backups and assumed that it would be able to backup database files directly. The customer’s assumption was wrong that it would backup the database files because the files were exclusively locked by the database. When the customer had a catastrophic hardware failure and requested a full restore of their database there was nothing that could be restored from backup. The only other option that the customer had was a data recovery service. Unfortunately, nothing could be rescued from the drives and the bill for the data recovery service was several thousand dollars. The business had lost one of its most important assets, its database.
Situation 2 - Database Corruption Failure
A customer had a database corruption failure when the physical files on disk become unreadable by the database engine. This customer also assumed that the server was being backed up, but never took the time to test restores. When this catastrophic event happened, there was nothing to be restored. The customer spent about 500+ dollars on software that could read the raw database file format and would bulk export any readable data. The customer was only able to rescue about half of his data from the corrupted database.
Situation 3 - Natural Disaster
A customer was trying to move critical infrastructure during a hurricane in Florida. The customer was panicked because they were concerned that the local data center that was hosting their e-commerce site was going to be impacted by the storm. The customer was trying to execute a last minute migration in the middle of evacuating an impending storm. They had failed to plan ahead and execute before it was too late.
What can we learn from these three situations? Disasters can and will happen when we least expect it. Businesses should take some time to develop a disaster recovery plan and exercise it on at least a quarterly or annual basis to ensure everything is working properly. Waiting until the last minute to develop a DR strategy is bad for business and creates significant risk long term. Please reach out if we can help you with your DR plans!