Having worked in support for many many years, you find yourself asking the same question over and over again. Have you turned it on and off again – Sorry could resist.
The question you should ask is – Have you a latest backup of the system before we try these troubleshooting steps. You always get the same answer – Yes we backed up last night .This is when you become immune to actually thinking about the answer. It may have been backed up last night but has it been tested? – The age old saying of your backup is only as good as the last time you tested it has never been truer than the story of Toy Story 2.
Whenever we speak about backup its very hard to make it as exciting as say Cyber Security with James Bond stories of hacking and Government espionage , When I’m asked to explain why backup is important I always tell the story of Toy Story 2 and how it nearly became destined for the bin.
/bin/rm -r -f *
While working on the film and random command deleted the directory the film was stored on and the film directors could see all the animation being deleted in front t of their eyes.
The command that was executed was more than likely rm -r -f . The command generally means remove all files from said directory.
As panic set in , The IT team came to the rescue and advised – Don’t worry we have a backup of all the files and folders, Lets be heroes for the day and start the restore.
Have we tested the backup?
The restore actual went ahead ok so the IT team thought , But as the animators and film producers started work again some anomalies kept coming up in the animation , Woody’s hat would disappear , Random errors would crop up etc etc
The issue was found to be that the backup had only restored 90% of the files, An investigation showed that the tape being used to store that backups could only hold a certain amount of data , So the backups would be over written or not backed up at all.
Yes the backup had ran every night and yes they could restore some data , But the whole data set could not be restored due to the fact the backup was never tested.
The film was eventually saved by a remote worker that’s system was never on the Network, The moral of the story is not to have more remote workers – But to test your backups regularly
And that’s how we make backup exciting – If you don’t believe me ask Buzz
