Memcache, keeping data in the handiest place: memory
While I ducked out of Giuseppe’s miniconf talk, on MySQL Proxy (a great session, might I add – it takes up 2 slots right up until lunch), I went over to the LinuxChix miniconf, to attend a talk about memcache, by Brenda Wallace. Brenda, works at Catalyst IT, in New Zealand – they use a lot of memcache, in the telco business.
Memcache: volatile cache for keeping data in. Its a daemon. The code, can connect to memcache, put values in, read values, delete values. An example of how to use memcache, is given in PHP5.
A killer feature, is the setting of expiry. You can tell it to cache for 30 seconds, and then forget about it, no worries there.
What do you store? Database, generated content (front page of a website, just like a blog even), web service lookups (useful in telco, or say, if you’re playing with the Flickr API), LDAP, things that are far away (from the other side of the pond, etc.).
Wikipedia made memcache famous, Twitter uses it a lot, and there are probably heaps more.
Many APIs, and there’s a postgres client too. There’s a memcache storage engine for MySQL as well.
Code should be written such that if its in memcache, use that, otherwise, get it from the database and put it in memcache.
Another nifty feature, is incrementing a value – increment functions $memc->inc(‘name’);. You can also read stats, to see a cache hit or miss.
Memcache doesn’t have locks. Memcache is not atomic. There are other libraries out there to do locking, there is a known Perl library for this.
What not to store? Remember, its completely volatile. Don’t store anything you’d be sad to lose, and make sure the real copy is safe elsewhere. There is no method to get list of keys in store. There is a 1MB limit per item, so if data is larger than that, you’re in trouble.
Where do you run it? Remember, it is memory hungry, but CPU lite. Running memcache on the webserver is the recommended method, so beware of the security.
There is no authentication. You just connect (no username, no password). So, when running on the web server, you probably want a firewall. In shared hosting, everyone on that host can read/write to your memcache instance.
No check for validity. No referential integrity, its not a database.
There is transparent failover! So if one fails, the client just automatically connects it to another.
Usage ideas? Communicate between layers (talk to a PHP app, from Java). Instead of squid, you can store stuff in memcache, if you want.
Some competing technologies: Tugela – same concept, but its saved to disk, so it will survive a reboot. This is the Wikipedia fork, of memcached. Couch DB is mention, but its not really a competitor, seeing that its a document database. Lucene is another competitor, but remember, its a fast indexer, and its non-volatile.
I haven’t looked into memcached much, but its quite clear, its a great technology to look at. Now the fact that you can use the MySQL storage engine, it might actually be really, interesting.
Technorati Tags: linux.conf.au, lca08, linuxchix, memcached
“No check for validity. No referential integrity, its not a database.”
So about as reliable as MySQL then.
Erm, MySQL supports referential integrity, for a while now Simon. And saying its not a database, or reliable, is rather silly, no? Seeing that its probably the most widely used open source database out there (you know, Google, Yahoo!, etc.). I suggest taking a look at a sensible, newer release, in the MySQL 5 range (and if you’re brave, the 5.1 RC)
Was just a gentle dig. Remember that until 2001 it didn’t support referential integrity or transactions and the developers claimed these things were bad ideas… Just doesn’t give me much faith.
As for “the most widely used…” — Windows is the most widely used desktop OS, and lookit that ;)
Haha, good point Simon. I think MySQL (the company) is still fixing mistakes of the past. We fixed a lot of complaints in 5.0, but said a bunch of silly things before. Give us another chance, it is after all 2008 :)
Memcache doesn’t have locks, though its increase/decrease methods are atomic! This way you can easily create locks in your own code.
To me thats a really important feature, thought its worth mentioning it.
[…] offer. I’m surprised the LinuxChix talks never made it online (there was a really interesting memcache talk […]
Just for clarity (since this has come up on the list), every memcached operation is atomic, but the cache is not transactional. The CAS operation exists in the text protocol for performing an atomic read/update operation, and is extended to every operation in the binary protocol.
Was just a gentle dig. Remember that until 2001 it didn't support referential integrity or transactions and the developers claimed these things were bad ideas… Just doesn't give me much faith.
As for “the most widely used…” — Windows is the most widely used desktop OS, and lookit that ;)