Unbelievable! Two catastrophic failures with total potential data loss within the same company, three years apart. Each time, it was a miracle that the data was recovered. But man was it close.
First time it happened was on a Friday in March 2008. The company was running a ten-year-old NT 4.0 server. One of the old 16 bit SCSI hard drives failed and I was called in to install a new server and transfer data. I thought it was simple because it was the boot drive that failed, and rarely do you find important data on a boot drive. However, in this case, the software vendor insisted on placing their program and data on the boot drive and would,'t you know it, the drive would not spin (work). The boot drive and data was lost.
The administrative assistant was responsible for backups. She used flash drives and carried them home with her each evening. However, this evening, she was on a cruise in the Caribbean and could not be contacted. So here we are, data drive won’t spin and the backups are in the middle of the ocean. What to do:
I contacted every vendor I know to see if I could find a scsii drive like the one that failed. Could not find one. So I broke the news to my customer (they were devastated) and I began setting up the new server, minus critical data. The next day, Saturday, my wife and I went to a local flea market. Wouldn’t you know it, one of the flea market vendors that sold old computer parts, had the right scsii card and cables with three drives attached to it that matched my customer’s broken scsii drive. I bought them all for ten dollars. Next I plugged them into a workbench computer, and they all worked! (In fact, they still had company data on it from a local accounting firm, but that’s another story).
What I did was take one of the flea market drives apart, removed the platters, and replaced them with my customer’s drive platters. Drive platters degredate quickly when exposed to air, so if this was to work, I would have a limited time to read and backup the data. Talk about good fortunate, the drive spun, and I was able to get all the data off of the drive and onto the new server. Can you imagine the odds against that happening, being successful? I’m not that good, but sometimes it’s better to be lucky than good. There was jubilation and celebration in the office Monday morning when I installed the server with data intact!
We implanted a three-teir backup system. Data on the server mirrored data drives was backup to the boot drive. Data was backed up from the server to the Admin Workstation. Data was also placed on removable media that went home with the Admin each night. Sounds like an “air tight” system, aah.
Fast forward to November 2011. My customer dropped maintenance a year ago due to the economy. They also lost half of their employees, including the one that was responsible for the backup system on the new server. Since her computer was turned off, there were no internal backups, and, of course, and no external backups to removable media. In my absence, their crital data was moved from the mirrored data drives to the “boot drive” by the software vendor, thus negating the backup that occurred within the server. So here was are, three years later, a blown hard drive and no backup! Lightning struck twice!
So here I am, again, reporting to the customer the possibility of catastrophic failure. This isn't good for me, but I feel the pain and anxiety as much or perhaps more than the customer. Each time this happened, I sware it took ten years off my life from worry and stress. Fortunatly, this was a sada drive, and fairly new. I was able to find an exact match that day. But would swapping the platters work again? This is very risky. Well, it worked again, and the next day, their system was up and running with data intact. We reinstated the old backup system with new people, moved the critical data to the mirrored drives and made certain the software vendor had it in their record NOT TO MOVE IT AGAIN. Hopefully, we won’t be doing this again in three years.
Bottom line: make sure you have good backups every day. Make sure you are backing up the right data, and make sure you know how to restore you data if needed. Many companies that lose their critical data go out of business. Don’t let this happen to you!