A Storage Engine for Amazon S3
A Storage Engine for Amazon S3, Mark Atwood
It looks mighty interesting, as transfers to Amazon S3 are free. I think it’ll work well in America and places where bandwidth rocks, but I don’t see this working too well in Australia. Oh how I wish the Internets will improve.
Mark has got all his stuff online at A MySQL Storage Engine for AWS S3. He was also kind enough to upload most of the notes, which made my reporting easier, and don’t forget to view the presentation.
Traditional storage engines use the local disk.
Networked engines: Federated, ODBC, HTTP, MemCacheD and S3 storage engine.
What is S3?
Contents can have 1 byte to 5GB. Amazon has several petabytes of storage for you to use :-)
Owning your own disks kind of suck. Pay for storage, even before you use it.
“An empty disk costs the same as a full one” – pay a lot of money to put disks in data centers. RAID isn’t “excellent”, then what about disaster recovery?
Can’t move existing database over, and the S3 storage engine isn’t ready for a full schema yet. There are hacks that allows this, but maybe it will be available next year.
Over a billion items in a bucket, and they all come back in good time.
A bucket is fully virtually hosted, you get a SQL CMS in the MySQL server. Save your EC2 work.
S3 is very distributed (geographically) and asynchronous. Writes are replicated, so your data may be reordered (and delayed). So there are no temporal guarantees.
Use the WHERE clause – otherwise it will do a full table scan, and you’ll be paying Amazon lots of money :-)
The talk ended with about 20 minutes to spare, and I do certainly hope he hacks on it more for the next year. He’s also soliciting feedback, so try it out if you can. And now, to run to the remainder of the talk on Highly Available MySQL Cluster on Amazon EC2! Two Amazon talks, with emerging technology goodness, at the same time? Pfft.
Technorati Tags: amazon, amazon s3, mysql, storage engine, networked engine, s3, mysqlconf, mysqlconf07
I’m glad you enjoyed the talk.
> transfers to Amazon S3 are free
Not quite: transfers between EC2 and S3 are free. Otherwise, you pay USD$0.20 per GB transferred in/out of S3.
or at least, according to http://www.amazon.com/gp/browse.html?node=16427261 and http://www.amazon.com/b/ref=sc_fe_l_2/102-3577618-7743337?ie=UTF8&node=201590011&no=3435361&me=A36L942TSJ2AJA
It’s an interesting idea but beyond the bandwidth issues some people are wary of having (more of) their data sitting in the US…
yup cos, you’re right. price dropping, but it still costs money to send data there – http://www.amazon.com/gp/browse.html?node=16427261#price