Welcome to this 3-part data protection video/blog series by one of the hands-on experts in this field and guest blogger, Eugene Alfaro. This first one is based on a true story, we’re sad to say. Read about some harrowing highlights (or should we say lowlights?), as well as some of the key complexities and challenges surrounding what keeps you up at night. Also, learn more tips on how to get a full nights rest with tips from Holistic Data Protection: Establishing a Best Practice Framework.
Data Protection - Diary of a Data Loss
Thursday, Nov. 15 - 5pm
I love this new company! After just a few days here, I get to be a part of a community event to build homes for underprivileged families. Apparently, the company does this every year and encourages participation. Today, we're putting up the walls on a small house.
Friday, Nov. 16 - Noon
After we put up the walls on the last house, I could not stick around. My CIO at the new company called me back to the office to report that we had an Exchange outage. I'm rushing back to the office to handle it and save the day! I've been in this position before. It's probably just a service crash. Restart the Exchange Information Store service and we should be back up and running. We'd be heroes after just clicking the "restart" button.
Friday, Nov. 16 - 3pm
It's worse than I thought. The database is corrupted and won't start up. I'll just run ESEUTIL. It's worked wonders in the past. I just hope there is enough space available on the drive for the tool to run successfully. Can still be a hero if we get emails up by end of business day.
Friday, Nov. 16 - 3:15pm
Seriously!?! This server was just setup a few months ago and it's already over 90% full? Without any spare drives or expandable storage on the network, I have no choice but to run a restore of the backup from yesterday. REMEMBER: When planning storage utilization, always estimate high then raise it up even higher. Give a user 200GB and they'll ask for 500GB.
Friday, Nov. 16 - 4pm
Who decided that tapes from the most recent backup should be couriered to the Southern California office. It makes sense to ship tapes that are older than a week but not the most recent. We had to call the courier company to have the driver turn around and come back. REMEMBER: It's a good practice to send older media offsite but not the most recent. The most recent should be in a fire safe on the premises.
Friday, Nov. 16 - 5pm
We're loading the tapes and beginning to catalog them. I can't wait for the day when we get rid of tapes all together and just recover directly from a local drive or down the wire from an offsite facility. Tapes are so passé.
Friday, Nov. 16 - 6pm
Really? Really? The last Exchange admin enabled circular logging on the database which means that the transaction logs are purged. Hence, we'll only be able to recover up to the last full backup which was Thursday 10pm. We'll lose over a full work day of emails. Additionally, the way that the backup jobs are setup is so complicated, that' it will take me a few hours to decipher which is which. REMEMBER: Data protection is not just about making sure you have backups but also making sure that the systems you're backing up are setup correctly. Also, the setup needs to be simple because data recovery is a high-stress situation. Complexity will just create confusion making us prone to error.
Saturday, Nov. 17 - 2am
Waiting is horrible. I wonder if my new company has any documented RTO and RPO times yet? If so, I doubt they calculated the amount of time it takes to recover data from old tape media. REMEMBER: Always calculate data recovery time and assume a serial process i.e. after restoring this tape, we load the next, then the next, etc..
Saturday, Nov. 17 - 5am
The good news is that the database from Thursday's full backup successfully recovered. The bad news, any email transactions after that are lost. We have some tactics to help recover some of that but, REMEMBER: If you lose just one piece of data, data protection is considered a failure. Data protection is about feeling secure that we're truly protected and needs to be simple so that it can be executed successfully.
Saturday, Nov. 17 - 6am
I just got home and am going to bed now exhausted and worried about what will happen Monday morning. Luckily, I'm not the one that set this all up and I'm the "new" guy. That will give me an opportunity to recommend some significant changes. It will be hard to sleep this weekend expecting Monday Mania.
Learn more tips on how to get a full nights rest with our latest Best Practice White Paper: Holistic Data Protection: Establishing a Best Practice Framework.
About our guest blogger: Eugene Alfaro leads IT engineering for Cornerstone Technologies, an IT engineering services firm in San Jose, Calif. He has also architected, managed and operated corporate IT environments for multinational companies. He has been a speaker on topics such as virtualization, private/public cloud services, WAN optimization, enterprise storage and VoIP, among others.