Oh, that HURT! (Of Migrations, Sessions, and Downtime)

To my most beloved readers,

You may have noticed some strange behavior around slash7 this past 24 hours. First, the site was just gone. Then it required a username and password (and the notification listed my email address and AIM account for you to “get credentials”). I’ve never received so much reader mail in one day!

I sent a mass email to all of you, explaining that all was not lost. I had not forsaken you, I’d merely busted my server. I decided to upgrade Sunday night and debut my new (not-entirely-polished) site design as a Monday surprise. A little visual sherbet, if you will, for that most awful of days *.

But rake migrate caused the server to choke and die. (I had not realized Linux’s updatedb was running at the same time.) I was confused, but thankfully I had backed up the database first using mysqldump, because rake migrate really screwed stuff up. So I went and tried to restore the database, but that didn’t work right either. The server was under extremely high load… the signs were not well.

This was the main problem:

mysql> select count() from sessions;
+----------+
| count() |
+----------+
|  1649510 |
+----------+
1 row in set (0.00 sec)

I believe I am not alone in saying holy crap! that’s a lot of rows!. Yes, 1.65 million session records. Daaaayamn. I didn’t know this was going on; I naively assumed that if Rails handled sessions, that would include, you know, expiring them. But no.

If you’ve set your Rails app up to use the database to store sessions, you might want to check your sessions table, too, before trying to run your next rake migrate. Leeo has posted a good snippet for managing sessions to Rails Weenie. You can be sure I’m setting up a cron job tonight.

Of course, it wouldn’t be a real upgrade nightmare story if that were the end of it. No… migrations ate all the old comments beyond a few articles ago (again, I have them backed up, it’s just a matter of restoring them). And until I turned off page caching, nobody could post articles or comments. Feh!

Also, I had planned to spend the significant amount of time I’ve spent troubleshooting this whole situation doing up the rest of the sidebar, and the links for the file downloads and such. Please bear with. I will try to get to them tomorrow.

Major props to all who helped me in my hour(s) of need: foremost, the ever-helpful folks in Caboo.se — Zed Shaw, leeo, Rick Olson, courtenay, and all the others; and RobotDeathSquad from the #typo IRC channel on irc.freenode.net.

* I actually like my job.