You never know when disaster will strike. It could be malicious ransomware, a natural catastrophe, hardware failure, intentional sabotage or human error. No data center or IT infrastructure is immune to potential disaster. Revenue, customer satisfaction and employee operations are all dramatically affected. What’s even more alarming is that nearly half the companies that lose their data through disaster never re-open and 90 percent are out of business within two years.*
For the past 20 years, RPE has worked with retailers to develop and implement disaster recovery plans to improve speed of recovery should the unfortunate event of a disaster occur. A disaster recovery plan is a documented process and set of procedures to recover and protect a business’s IT infrastructure. It’s a comprehensive step-by-step process of actions to be taken before, during and after a disaster. See RPE back up and data recovery services.
RPE takes the disaster recovery planning process a step further and works with clients to execute the plan in a test environment. Many retailers fail to take the extra step of plan testing as it requires extensive preparation and coordination. But for those that do, the payoff is well worth the investment of time and resources. With any disaster recovery process, you can expect to have issues. An annual test allows the IT staff to identify stumbling blocks so in the event of an actual emergency, the retailer has already worked through possible worst-case scenarios.
There are many benefits to testing a disaster recovery plan annually:
- Provides a standard for testing the plan
- Addresses unexpected challenges
- Identifies opportunities for improved communication
- Tests the reliability of standby systems
- Minimizes decision-making during a disaster
- Improves elapsed time for restoration
Lessons Learned
This annual drill has already proven beneficial to one of RPE’s valued retail client. As a $500 million retailer with stores across the county and a large online shopping base, being able to flawlessly execute the disaster recovery plan is the difference between surviving or falling to the statistics.
RPE worked with the client to develop the initial disaster recovery plan and has tested the process in a live environment for the past two years. A successful disaster recovery plan requires the buy-in from all levels of the organization. Top management must support and understand the importance of testing the plan and allow for adequate time, resources and budget dedicated to the process.
There is not one plan that is right for every organization. Management must decide what systems are critical to be included in the plan, as well as how long each system can be down. Do you need the ability to recover in minutes or can you wait a few days? Remember, the faster the recovery, the more it will cost.
Helpful Tips
Based on hands-on experience, here are tips to improve the process as carrying out a plan in a live test scenario is often wrought with potential challenges and uncertainties.
Plan Ahead for the Test
One luxury that you will enjoy when you test the plan is that you’ll know when the plan is going to be tested. This allows for an additional level of planning that will not be available in an actual disaster event.
Execute the Plan With An Open Mind
Now it’s time to execute the actual plan. Within the document, it should clearly identify who is going to be involved (internal and external staff), what are their responsibilities and timelines, how can they be contacted, etc. Chances are that all involved will discover obstacles during the test. Don’t view these as failures. This is the whole reason to run the test.
Document and Resolve Deficiencies
It’s important to document EVERY bump in the road. You may or may not be able to resolve the issue on the fly, but you must eventually solve the problem and potentially update your plan document.
Communicate Well and Often
On the designated test date, an email notification should be sent to the team declaring the nature of the emergency such as “due to major interruptions in the power grid and internet facility in most of the state”. It should outline the next steps of activating the disaster recovery plan to the bring the system live. Most importantly, the communication must clearly state when the next update will be – typically every two to four hours. At the designated time, an update on the recovery process and key milestone should be issued. Remember that part of the plan is to assign who will be charged with distributing these communications.
Review Processes for Improvement
Shortly after completion of the test, it’s imperative for everyone that was involved in the process to meet and discuss what went right and, more importantly, what did not go as well as planned. This is the right time to make adjustments to improve for the future.
Remember that there’s no such thing as a failed test. Identifying and correcting deficiencies is the goal of any test. The real failure is not having and testing a disaster recovery plan. Most importantly, the payoff is big when a disaster recovery plan is developed and annually tested. Clients are able to get back up and running quickly as they have already worked through potential roadblocks.
If you do not have a well-tested disaster recovery plan, get started today. RPE can help you every step of the way from gaining buy-in from the organization to testing the plan.
*University of Texas Center for Research on Information Systems