On Ma.gnolia, and data recovery
There’s a good podcast from Chris Messina and Larry Halff, about what really happened at Ma.gnolia. If you’re at all interested in what happened (i.e. how did they lose all their bookmark data), don’t hesitate to watch the video. I took some quick notes:
- half a terabyte database file got corrupted
- a mysql 5 database
- everything was running even though there was corruption, and eventually, the site went down
- backup system also failed, as it didn’t backup the data from mysql
- backup was just backing up corrupted data (file sync over a firewire network was the backup mechanism)
- a Rails application, he now recommends clouds over running your own infrastructure for startups
- a couple of xserves (for database, etc.) and four intel mac minis as front end web servers
- the site didn’t actually make any money
So I don’t know if Baron can rescue Ma.gnolia, per se, but I think the problem was largely:
Doing a file sync over the Firewire network, as the backup mechanism
You can’t safely backup MySQL that way. I don’t know what mechanism was used, but it sounds like rsync, and as much as I love rsync, I wouldn’t use it to backup a live running MySQL database that way.
With two servers, there should have been MySQL replication.
I’m curious if the data recovery Baron talks about is that of using the utility ddrescue? After all, ddrescue gets the raw data off the block device, without even trying to mount it. After that, you can attempt to recover the MySQL data off disk. In fact, I was surprised that the Ubuntu folk have a very nice Data Recovery page – no information about extracting MySQL databases, but its nothing a little hackery won’t get you.
I tried to ping Larry on Twitter, to ask what engine they were using… No response, per se. Good luck, and I hope the users get their data back, in time!
I was referring to some data recovery tools we’ve written specifically for MyISAM and InnoDB data. The InnoDB tools are on Google Code at http://code.google.com/p/innodb-tools/ but the MyISAM tools are still being developed.
Hi Baron!
That’s really cool stuff, I’ll definitely be playing with it. No reason for any hackery, when there’s already code on the Net :)
So rescue, then run innodb-tools. Very nifty.
Thanks!
The tools themselves can scan a raw device.
It would be nice to know whether this was InnoDB or MyISAM. InnoDB always checksums pages. MyISAM can but doesn’t by default. The checksum doesn’t prevent the corruption but may help you spot the problem sooner.
Hi Mark!
I have a feeling it was MyISAM. And will update all, if I get information from ma.gnolia.
-c