Your Shameless Host
Hi! Welcome to the personal blog of Jason Stirk (Griffin) - a slightly unhinged web application developer living in Lismore, NSW (yes, that's in Australia).
I run a software consulting company called Aurora Software.
So this guy walks into a bar... Wait...
Anyhu, so I got a call yesterday evening - seems a company I did some small incidental support for (changing DNS server on their Smoothwall box a few months ago) had their mail server die over the weekend.
I used to work with the guy who set it up, so I wasn't all that worried about trying to recover it. From memory it was likely to be running RedHat 9 or something like that judging by the age, and whatever had happened (short of a head-crash) was bound to be recoverable in some shape or form.
On the phone a friend of the owners is explaining to me what he sees - blank GRUB screen. Supplying him with the commands to boot the kernel doesn't seem to be getting him anywhere. Bugger it - I'll be in there at some god-awful time tomorrow morning... 8:30...
This morning I get the real story. Seems they were having issues so this guy SSHed IN and RAN FSCK ON THE MOUNTED PARTITIONS.
Let's see what fsck might look like when run against a mounted partition:
# fsck /dev/hda1 fsck 1.34 (25-Jul-2003) e2fsck 1.34 (25-Jul-2003) /dev/hda1 is mounted. WARNING!!! Running e2fsck on a mounted filesystem may cause SEVERE filesystem damage. Do you really want to continue (y/n)?
Now that's a pretty big warning...
Regardless, the logs show that about 28 seconds after he initially shelled into the box (as root none-the-less) the kernel starts throwing up some major ext3 errors... Good... Gravy...
All of /boot disappears and ends up in lost+found. All of /var/spool/mail ends up in lost+found. 1/2 of /etc ends up in lost+found. <sigh>
Anyhu... So I am tasked with the job or recovering 1.8 gigs worth of mail stored in mbox files in the account homes (IMAP folders). Good work, considering all I thought to bring was a Fedora Core 1 CD set. (For the record, yes, Zanchey - I did think of FreeSBIE - but I was buggered if I was going to put my burner back in and knock out the ISO at 6:30 this morning!!!)
To cut a long story short, FC1 rescue mode doesn't include sshd, or SaMBa support, so I was left with nc to transfer all the 1.8gigs worth of email to the Win2K server (oh dear me... Not even going to start on this one...)
I must admit, I was pleasantly surprised to find a Windows build of nc which ended up making life a whole lot easier. The alternative was to mount the SMB shares on my laptop and use nc to copy the 1.8gig something like dead server -nc-> laptop -smb-> 2k server.
The migration appeared (at first) to be fairly simple... The IMAP server they were using allowed you to just drop mbox files into the account storage directories and it would read them. Of course, things are never that simple.
UNIX -> Windows always has one major hitch... Newlines... Yes, that's right folks... I then had to mangle all 1.8gb of mbox files to replace UNIX newlines with Windows newlines. Fun, fun fun. Thankfully, a quick combination of BASH scripting and PHP and we were on our way to success. Slowly, but on our way!
Moral of the story - always assume the worst, and take the most powerful tools you can find.