A Disaster-Recovery Test Turns into a Disaster of Its Own

You may have seen this headline: "Lost data may have exposed 800,000 people in Calif." Shaya Tayefe Mohajer, a reporter working for the Associated Press, explains what happened:

“Four computer storage devices for the California Department of Child Support Services went missing somewhere between Boulder, Colo., and Sacramento earlier this month while they were in the possession of IBM and Iron Mountain, Inc… The cartridges had been sent to IBM's facility in Boulder as part of a disaster simulation, so the technology company could test whether it could run the state's child support system remotely.”


The identities of 800,000 people are now potentially at risk. The names, addresses, driver's license numbers, Social Security numbers, and other sensitive information: exposed.

On the eve of World Backup Day, this story really resonates with me. The agency *seemed* to have taken all the precautions. But it relied on third-party vendors for its backup and tape storage, which, in turn, outsourced the tape transport to yet another third party. Could the risk of exposure have been minimized? The article doesn’t mention if the data was encrypted. I hope it was. It’s all too common, though, for organizations to forgo encryption in favor of a faster, more reliable, and less complicated recoveries.

Of course, hindsight is 20/20; but we can learn from others’ mistakes. Adrian Moir has some great recommendations in his three-part blog series on "Considerations for Disaster Recovery" (Part 1; Part 2Part 3). I encourage you to read all three posts. One section in particular that I think is very relevant is his recommendation to leverage disk-based backup, replication and virtualization to create a “warm” DR site. This allows you to eliminate tape altogether (and its inherit risks).

Take, for instance, CMC Markets, a financial services organization that created a virtual DR strategy based on NetVault FastRecover, a continuous data protection and replication solution. This organization couldn’t afford any data loss or sustained downtime, so it installed a CDP server at each of its premises. The solution is now capturing byte-level changes on the protected systems and saving them to disk. Then those changes are replicated over the WAN to their virtual DR site in real time.

The organization was put through a real DR test when its Tokyo office suffered intermittent power outages as a result of the devastating earthquake and tsunami in 2011. Thankfully, because of the IT team's advance preparation, the organization was able to quickly restore its critical applications and associated data at virtual DR site. Seconds after the recovery was initiated, employees were able to access the lost data from the virtual standby servers. A perfect example of DR executed flawlessly.

To hear more about this story, be sure to check out John Maxwell’s session at SNW on Monday.

Session Title: Rapid Recovery: Maintaining Operations After the 2011 Japan Earthquake and Tsunami

Session Date: Monday, April 2, 2012

Session Time: 1:55 PM to 2:40 PM

Track: Business Continuity

Speaker: John Maxwell, Vice President, Product Management, Data Protection