Recovering accidentally deleted files

Let's have the good news first: if you happen to be using Windows 7 (and surely Win 8 when that arrives), file undelete is built in and it's easy to use - just follow the directions found at Microsoft's Recover lost or deleted files article. Easy enough, disaster averted, you won't get fired, nobody is going to be screaming at you and all is well.

If you are running Windows Vista, it's almost as easy. You need a tool called Shadow Explorer, but once you have that, recovering deleted files is not difficult. Hint: if you do have Vista, you might want to go get that tool now before you need it.

If you are using almost anything else (XP, Linux, Mac), you can almost certainly find a utility and instructions for getting back deleted files - a little Googling is all you need. One that I have had good luck with is Photorec, which is cross-platform, free, and includes source code for those who care about such things. Again, it's not a bad idea to have that installed and to know how to use it before you really need it.

That's good

So that's the good news. What's the bad news?

The bad news is that it's easy enough to screw this up. I'll talk about how shortly, but first let's talk about something that happened earlier this week.

A very depressed customer

A few days back I had a phone call from a long-time customer. I knew right away that he wasn't happy and his very first words told me why: "We've deleted our main database file", he told me.

My first thought was that this can't really be a problem, because they have backup. Not only do they have backup, but they have duplicated backup - they backup to tape AND they backup up over the Internet to a machine in a branch office. I had set up those backups myself, so I wasn't too worried. But then..

Failures

Apparently the tape backup had stopped working almost six months ago and somehow nobody noticed. Somebody should have noticed, because the software I set up sends notices of any problems by email and/or printouts. For whatever reason, these notices were missed.

It may be that knowing that two backup systems were in place made it easy to ignore the messages from the tape system. I don't know - I've had that happen with other customers at other times and that is one of the reasons I like to have multiple backups in place. People are just people and they can get so used to seeing notices that they just don't bother to read what they actually say. I've had customers get lazy about backups and do other really silly things. Stuff happens, and we had the backup to the branch office, right?

Application obsolescence

Well, unfortunately no. Two months ago, a certain application at that branch office reached the end of its useful life and was discarded in favor of something newer. The newer application ran on new hardware and the old hardware, no longer needed, was shut off.

Guess what else was running on that old piece of hardware? Yes, it was the software that received the Internet backup from the main office. So, as of two months ago, that backup stopped working.

Of course it would have generated error reports too, but if anyone did notice, they may not have looked much different from the other tape backup failures - if they were noticed at all.

Whatever happened, nobody knew that neither backup was working until the database file was deleted.

"Is there an undelete?"

That was my customer's next question. In fact, although his database runs on an SCO Unix operating system that dates back to the mid 1990's, it actually did have an "undelete" feature. Unfortunately, because it required a lot of extra disk space (it used versioning, much like Win 7 does today), it wasn't enabled by default, so almost nobody ever turned it on. So my answer was negative - no "undelete".

Data recovery

At this point, the only option is data recovery. He needs someone with knowledge of SCO Unix file systems to go in and try to dig out the deleted data. Such people are not impossible to find - again, a little Googling will turn them up.

You see, when a file is deleted, most operating systems don't really delete the data. Being somewhat lazy, most just return whatever space the file used to the "free list". That's why tools like that Photorec I mentioned above can successfully recover deleted files. They just have to understand a little bit about how the file system works and go grubbing about looking for the stuff you deleted. So he just needs a recovery company that knows a bit about SCO Unix and he can get his data back. Simple enough, except..

Don't touch anything

The number one rule of data recovery is that you stop using the system immediately.

The reason is that disk blocks put on that "free list" are up for grabs to be used to hold new data. It would be nice if the most recent blocks sent to the free list would be the last things used for new data, and indeed the operating system might work just that way, but obviously the less you do, the more chance that those blocks are still sitting there untouched.

If the system is tight on disk space (and many of these older systems are), of course there is even more danger of deleted data blocks being recycled for new needs, so the rule is "Stop". Shut it down, stop using it, ship it off to the folks who can dig out what you need.

Let's copy some stuff

Unfortunately, before he called me, he called the database people. They logged in, poked around, and made "safe" copies of some files that were not deleted.

Well, copying a file requires new storage space. While we hope that space did not come from any of the blocks that had comprised that big database file, we simply can't be sure of that - especially as he was already tight on space before this mess all started.

So, I don't know if he can get everything back. He might, he might not. If not, all he has is a two month old backup and a lot of work to do.

A lesson for you

I hope you learn two things from this. First, check your backups regularly. Look at reports, yes, but also do test restores - can you really get at the data that was supposedly backed up? The time to know that is long before you actually need anything from the backup.

Second, if you do find yourself in that stomach churning moment where something important has been deleted and there is no backup, take your hands off the keyboard and STOP.

This is not the time to be doing anything with that particular computer. Shut it down, turn it off and use some other computer to research your options. Don't install new software, don't go copying files, don't do anything until you have fully assessed the damage and your options for recovery.

This may be something you can fix yourself, but if it really is critical data, I'd recommend letting a professional take over immediately. Make sure that professional really does have experience with your specific hardware and software. In some cases, you may need two experts - for example, if data recovery brings back an incomplete file, my customer will need someone familiar with the database structure to fix that up as much as possible.

STOP!

But first - STOP. That's the most most important rule. Stop, research, think. Know what your worst case scenario is, know what the best case is. If there are other people in or outside of your company who might know something about this, talk to ALL of them before anybody jumps in and starts doing anything. Maybe somebody happened to make a copy of that file just yesterday for reasons you know nothing about. Maybe there actually is another backup you know nothing about. Maybe its easy to undelete the file, maybe there is a procedure to rebuild it from other data.

Stop, research, think. Don't rush into anything and don't let anyone else charge ahead until you are sure of what should be done. TALK to people first - that may save you from making things worse!

I hope you never face a situation like this. If you are not that lucky, I hope that you will remember what I said here.