Primordial Ooze

If you recall, last week my site had a pretty major meltdown. It all started when I used the one-click install feature offered by my hosting company, dreamhost. For some reason, I had installed the wordpress blog into the root of my site, which is a mistake. To me, it’s better to install each web app in it’s own subdirectory and use .htaccess and mod_rewrite magic to make your url space the way you want it.

I wanted to “uninstall” the one-click install but it obviously worried me to click on the button to kick off the script. What if my whole site got deleted? You can probably guess where this is going.

I consulted the dreamhost gurus and someone responded, exactly like this:

Hello,

On Sat, 11 Feb 2006, you wrote:

> I'm not sure when I installed Wordpress, but I want 
> to remo ve the old site I added. However, I see 
> that the new "upgrade/remove" list shows the account 
> but lists my site's root (http://primordia.com) as the
> worpress install location.
>
> I want to remove this entry (and install a new Wordpress
> install in a subdirectory) but I am afraid to click 
> on the "remove" link since I don't want my whole site 
> blown away.
>
> URL / DB Software Last Updated Action
> http://www.primordia.com/
> oozewordpress
> wordpress 2004-12-08 18:48:02 [Upgrade to 2.0.1] [Remove]
>
> There is a copy/paste of what I see. I apologize for the bad
> formatting. Thanks.
>
> Nick

The Wordpress removal tool will only remove related 
files and databases. It will not touch any others you 
have added separate from Wordpress. You may continue without 
worry, then install to the new directory.

If there is anything more we can help you with, please 
let us know and we will be more than happy to accommodate.

Thanks,
Jeff T.

This was like over two months ago so removing the entry was on my to-do list. Last week, I finally started cleaning things up so I clicked on the link. A few minutes later, I got this e-mail:

From: DreamHost Installer Robot 
Subject: Success removing /home/ncodigno/primordia.com!

As you requested, we just removed:

/home/ncodigno/primordia.com

from your server.

If for some reason you need to re-install a one-click 
there, just return to our panel at:

https://panel.dreamhost.com/?tree=goodies.installer

Thanks!
The Happy DreamHost 1-Click Robot!

Sure enough, my entire site was gone. I wasn’t too worried and I took the news with surprising calm. After all, the dreamhost folks have restored individual files for me in the past with no problems. Plus, I perform my own backups every night. At any time, I thought, I could restore the site myself. I wanted to see how well dreamhost handled this situation. As I write this, it’s clear the experience could have been much better.

It was like 26 to 48 hours before they had restored the site. Even then, my /blog directory had the wrong files when I inspected the temp directory they created for my final review. /blog restored with the temp files I set up the morning after the site was down so they obviously pulled the wrong stuff from tape. Before that, an engineer had pulled some raw backup files and improperly unzippped things. He had responded in under 24 hours, but since he screwed up the restore, no one picked it up again for another 24 hours.

In a huff, I went and restored the two primary directories myself from my own backups. These are /blog and /wiki. My backups failed to contain my /cgi-bin directory, though, which is needed by the moinmoin wiki. I had to grab those files from the temp directory dreamhost provided for me so that was good.

In the end a few things will happen. I’ll e-mail this blog entry to their support team in hopes they can improve their restoration procedures. Second, I’ll make sure THEY perform risky procedures in the future. That’s what I pay them for, after all. Third, I’ve beefed up my own backup procedures so I can recover from errors like this myself. My faith in them is now somewhat diminished.

I still remain pretty loyal to these folks but that doesn’t mean I won’t send some criticism their way every now and then!

I’m back in business now and I’ve taken this opportunity to perform a massive clean and purge of some very old site files (from like 1999 and earlier), but leaving a few around.