Oh, that HURT! (Of Migrations, Sessions, and Downtime)
To my most beloved readers,
You may have noticed some strange behavior around slash7 this past 24 hours. First, the site was just gone. Then it required a username and password (and the notification listed my email address and AIM account for you to “get credentials”). I’ve never received so much reader mail in one day!
I sent a mass email to all of you, explaining that all was not lost. I had not forsaken you, I’d merely busted my server. I decided to upgrade Sunday night and debut my new (not-entirely-polished) site design as a Monday surprise. A little visual sherbet, if you will, for that most awful of days *.
But rake migrate caused the server to choke and die. (I had not realized Linux’s updatedb was running at the same time.) I was confused, but thankfully I had backed up the database first using mysqldump, because rake migrate really screwed stuff up. So I went and tried to restore the database, but that didn’t work right either. The server was under extremely high load… the signs were not well.
This was the main problem:
mysql> select count() from sessions; +----------+ | count() | +----------+ | 1649510 | +----------+ 1 row in set (0.00 sec)
I believe I am not alone in saying holy crap! that’s a lot of rows!. Yes, 1.65 million session records. Daaaayamn. I didn’t know this was going on; I naively assumed that if Rails handled sessions, that would include, you know, expiring them. But no.
If you’ve set your Rails app up to use the database to store sessions, you might want to check your sessions table, too, before trying to run your next rake migrate. Leeo has posted a good snippet for managing sessions to Rails Weenie. You can be sure I’m setting up a cron job tonight.
Of course, it wouldn’t be a real upgrade nightmare story if that were the end of it. No… migrations ate all the old comments beyond a few articles ago (again, I have them backed up, it’s just a matter of restoring them). And until I turned off page caching, nobody could post articles or comments. Feh!
Also, I had planned to spend the significant amount of time I’ve spent troubleshooting this whole situation doing up the rest of the sidebar, and the links for the file downloads and such. Please bear with. I will try to get to them tomorrow.
Major props to all who helped me in my hour(s) of need: foremost, the ever-helpful folks in Caboo.se — Zed Shaw, leeo, Rick Olson, courtenay, and all the others; and RobotDeathSquad from the #typo IRC channel on irc.freenode.net.
I like the new design! Spiffarific, you might say 😉
Well, I was one of the people trying to get to your site in the last 24 hours. It was worth the wait. Very nice!
I was trying to download your Typo themes. Is it just me or is Typo super-twitchy? Maybe it’s just Dreamhost?
In the constructive criticism area, I’d try to find a way to vertically align the left edge of your headers ("Oh, that hurt!") with the left edge of your top nav buttons ("blog"). I like the bold 7, but maybe shrink the "slash" down and tuck the ascender of the h inside the 7. I might bump the whole slash graphic one or two pixels to the left so the soft "s" curve visually appears to align with the harder "r" of "recent stuff". Now I’m sure they’re mathmatically aligned but to the naked eye it looks like the "slash" is a little to the right. Maybe try aligning on that sharp upward tip at the bottom of the s.
Or something.
Here’s a rake task you can use
<pre> desc "Clear old database-stored sessions" task :clear_db_sessions => :environment do CGI::Session::ActiveRecordStore::Session.destroy_all( ["updated_at < ?", 1.week.ago ] ) end
desc "Count database sessions" task :count_db_sessions => :environment do puts "Storing #{CGI::Session::ActiveRecordStore::Session.count} sessions" end </pre>
"I naively assumed that if Rails handled sessions, that would include, you know, expiring them. But no."
This is one of the things that annoys me about Rails–it’s inability to delete old session records (whether it’s in a DB or in millions of flat files in /tmp).
It seems to be a really stupid default to have sessions never get pruned without a cron job/rake task.
This is why you need two more types of people on opensource project core teams. Technical writers and anal as heck project managers.
While the developers are seduced by the cool factor, the end-users inevitably run up against the nitty gritty’s of the daily operations of a framework.
I have been a little surprised at this too. Its like building the fastest sleekest fighter jet with landing gear stuck in the open position.
I was very dissapointed of this!
The content of your show is great, I really enjoy it 🙂
i am not sure as to why 🙁
Well at last catched the problem!
I was very dissapointed of this 🙂
Well done, nice instructions…
Very nice write up.
Thanks for the write-up 🙂
The content of your show is great, I really enjoy it.
I had and the fix I found 🙂
The content of your show is great, I really enjoy it…
Thanks for the write-up.
I use Firefox in Ubuntu…
Well done, nice instructions!
Good idea…
Thats correct!
Well done, nice instructions 🙂
But I’m not sure why!
That’s awesome!
That is strange 🙁
I had and the fix I found!
Thanks. Updated appropriately…
That is strange!
Very nice write up!
Thanks for taking the time to do it 🙁
i am not sure as to why 🙁
There is some strange behaviour with this 🙁
That is strange 🙂
Well at last catched the problem 🙁
Good idea 🙂
The problem is my browser…
Thanks. Updated appropriately.
I thout to do it in my local version!
Very nice write up!
Well done, nice instructions 🙂
I think it would be usefull for other users also 🙁
I had and the fix I found.
Not really new.
Thats correct.
Gonna have to give it a try…
Thanks for the write-up.
That is strange 🙂
Thanks. Updated appropriately.
But I’m not sure why…