Archive for the ‘Databases’ Category

Trying to reliably make MyISAM crash; Maria is sturdy as

I’ve been very excited seeing that we announced the Maria Engine Preview. Giuseppe and I were trying to setup a demo, for Maria, at the lightning talks happening later today, towards the end of the MySQL MiniConf at linux.conf.au 2008. It involved MySQL with Maria, and an Asus Eee PC. For the demo, we wanted to show pulling the plug, which can be done either via a kill -9 `pgrep mysqld` or pulling out the battery of the Eee.

However, we failed to get MyISAM to reliably crash! Yes, imagine that, we actually want it to crash – pity that it might have only happened about 1/3rd of the time we tested it. The magic we were looking for:

check table t1;
+---------+-------+----------+---------------------------------------------------------+
| Table   | Op    | Msg_type | Msg_text                                                |
+---------+-------+----------+---------------------------------------------------------+
| test.t1 | check | warning  | 1 client is using or hasn't closed the table properly   |
| test.t1 | check | warning  | Size of datafile is: 23998464       Should be: 16000256 |
| test.t1 | check | error    | Recordlink that points outside datafile at 23000368     |
| test.t1 | check | error    | Corrupt                                                 |
+---------+-------+----------+---------------------------------------------------------+
4 rows in set (0.09 sec)

What we got instead was just a warning mentioning a client is using or hasn’t closed the table properly. Clearly, not so good for a demonstration.

When the magic of set storage_engine=maria; was run, and you crashed (either via pulling out the battery or doing a kill operation), Maria would survive the crash. At worst, you’ll see:

check table t1;
+---------+-------+----------+------------------------------------------------------------+
| Table   | Op    | Msg_type | Msg_text                                                   |
+---------+-------+----------+------------------------------------------------------------+
| test.t1 | check | warning  | Auto-increment value: 0 is smaller than max used value: 32 |
| test.t1 | check | status   | OK                                                         |
+---------+-------+----------+------------------------------------------------------------+
2 rows in set (0.20 sec)

So, Maria is clearly tough, as Giuseppe puts it. Give it a twirl (binaries, sources), and hop on over to the Maria Forums if you have questions. And if you’re checking out/building from source, you might find the Building MySQL from source guide handy

Alas, only a demo in a VM today . It seems more reliable inside a VM, not quite on the Eee. Besides, the poor SSD has been through quite a number of fsck’s now. BTW, one other thing to note: Maria log files seem to be pretty huge. Running it on a SSD that has about 1.5GB of free space (total, 4GB), is harsh.

For reference, the test (generally, wait for count to reach 128, switch terminals, and kill mysqld/pop battery off):

drop table if exists t1;

create table t1 (id int not null auto_increment primary key, b longblob) ;
select table_schema,table_name,engine
     from information_schema.tables
     where table_schema=schema() and table_name='t1';
insert into t1 values (1, repeat('a',1000000));
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;
insert into t1 select null,b from t1; select count(*) from t1;

Technorati Tags: , , , ,

Morning sessions at MySQL MiniConf

Upcoming MySQL Features – Stewart Smith

Stewart’s talk on Upcoming MySQL Features was sort of a roadmap of what one might expect to see in MySQL 5.1 and above – he touched on Falcon, online backup, batched key access, Maria, Proxy, Workbench, and some cluster changes. When he shares his slides, it might be great to link to Worklog items, and Forge pages about these new features and previews (because, believe me, the stuff thats coming in future, is clearly very exciting).

MySQL Indexing Methods – Jonathon Coombes

Sitting in Jonathon Coombes session on MySQL Indexing Methods now – he’s going through covering indexes, the B+-tree index, hash index, full text indexing.

Some select points, that aren’t in the slides (otherwise, the slides themselves are very verbose, and when they make it online, it will provide some great reading material):

  • InnoDB uses a B+-tree, and a secondary hash index.
  • MyISAM has R-Tree index support, so it can be used quite usefully for GIS applications. MyISAM is not the only engine that supports spatial indexing, but its the only one that uses R-Tree indexes.
  • T-Tree index is used in MySQL Cluster
  • Touched on the Lucene search engine, with at least about 8 hands going up, as to people using it in production. It does allow live indexing, does proximity searches
  • Sphinx, has a high indexing speed, and by default it sits outside the database, however you can set it to be a storage engine as well. Sphinx is distributed, so its similar to the way memcached is architectured. Distributed indexing. Full text fields.
  • Bitmap indexes, not available yet, but its something people are waiting for

Bit weird seeing Jonathon now showing off OpenQuery, as opposed to Cybersite!

MySQL Optimisation by Design – Arjen Lentz
Its a training course that Arjen is working on, a one day event. He has no slides, per se, so he’s going to use the whiteboard instead. So this is something you’ve got to hope to get a video of, at least.

If you use MySQL, quite often you stick a database abstraction layer in between. However, you never usually ever port to another database, and the abstraction layer always leaks.

Arjen now walks us thru engines available, by firing up the MySQL monitor, and going through the output of show storage engines;.

Arjen then goes through a demonstration, on how many rows per second you can INSERT. These are 60-byte rows, from data acquisition styled information (IP addresses from a router, web clicks, etc.). Using MyISAM. No indexes. Sustained inserts of about 150,000 rows per second (40 million rows, in about 5 minutes, till his disk got full), on his Mac OS X Leopard laptop (and he can’t max anything out). This demo, is of course, for Arjen’s new gizmo, that’s written using the MySQL C API, that currently is in development. The application makes use of a multi-row insert.

Using Blackhole, it holds at about 245,000 rows per second, for INSERTs. So, we’re seeing the slowdown, in really, disk I/O.

Codename: lossylogger.

Its called lossy, because its allowed to loose some data. If you’re tracking web clicks, does it really make a difference, if you’ve lost some clicks, in the grand scheme of things? Not really. It makes use of UDP, so you’re not opening connections either. So, occasionally, you get a buffer overflow, and occasionally you lose data, but its no big deal, in the grand scheme of things.

Most interestingly, at the tail-end of Arjen’s talk, I see the MySQL Australia account manager (sales), Ralph, walking in. Yes, we have sales in Australia/New Zealand now!

Technorati Tags: , , ,

MySQL Miniconf Starts!

Arjen and Stewart are on stage, and there’s an introduction session going on now. We’re now, introducing the ex-MySQLers (Arjen), and MySQLers (Stewart, Giuseppe, me). Trent has just walked in, so that makes all the MySQLers that are around at linux.conf.au.

Highlights of some of the attendees:

  • A user from LG, who has been using MySQL for about 3 years now
  • An technology manager in defense, interested in MySQL as an education exercise
  • A MySQL user for over 8 years
  • Systems administrator who’s been heavily using MySQL for 5 years, however with a total of about 8 years of use
  • Systems administrator at IBM, using MySQL for a long time
  • Travel startup in the Gold Coast, doing lots of MySQL, replication, proxy use
  • A software engineer at HP, in China, and they use MySQL for benchmarking on HP hardware
  • A Connector/J user
  • realestate.com.au, is a Perl+Apache+MySQL shop
  • A IBM guy who works in performance tuning
  • Canon Research Labs in Sydney, but now working in a new small-ish firm, using a lot of MySQL
  • MySQL for a VoIP company. They’ve tried mcluster, and it sucked ;) (and now, they want Cluster)
  • Bolt on’s, to old accounting packages. Using MySQL with Delphi; fiddling with Mondrian, Pentaho now and also looking at JBOSS
  • PHP developer using MySQL for CMS systems, using it for about 6+ years
  • m5 Networks, VoIP company, using MySQL, needs to scale, looking at Cluster
  • Lonely Planet, MySQL DBA, who needs more tips on scalability and high availability
  • Moodle host, also interested in high availability
  • Stronghold CMS, they’d like MySQL to support sequences on a transaction, and they don’t want it in a stored procedure. They work for the government, to some extent (afaik).
  • Using MySQL in education for research/study
  • Asterisk, and MySQL
  • Drupal, and optimising MySQL, so it can scale
  • vquence, who need to store over 100 million videos, and they’d like MySQL to scale for them

The trend is there are a lot of VoIP companies, and a lot of folk wanting high availability, and scaling to amazing lengths. Very interesting. OK, Stewart is going to tell us what’s new in MySQL now… He’s got a bottle of liquor available, in traditional Monty tradition :)

Technorati Tags: , , ,

CentOS, CentOSPlus, and MySQL versions shipping there

Peter posted that CentOS comes with a build of MySQL Enterprise. It should really be clarified that CentOS itself, comes with MySQL Community, as does Red Hat Enterprise Linux. On RHEL5/CentOS5, you’ll see:

mysql-5.0.22-2.2.el5_1.1
mysql-server-5.0.22-2.2.el5_1.1

The above are the default packages that CentOS provides. However, what Peter really is referring to is the CentOSPlus Repository, which by their own admittance is “not part of the upstream distribution and extend CentOS’s functionality at the expense of upstream compatibility. Enabling this repository makes CentOS different from upstream.”

The idea behind providing Enterprise builds, largely came from RHBZ#230412: No src.rpm available for mysql. Red Hat provides something called the Web Application Stacks product (RHWAS), in where they include MySQL Enterprise, amongst other software packages, and they charge for support (that differs from buying just a RHEL license). Max Spevack answers it pretty well, as this is the response he came to, after talking to folk at MySQL.

Its worth noting that CentOSPlus does not use Enterprise tarballs/SRPMS, but use sources from the BitKeeper tree. Its worth noting, that at the time of this writing, CentOSPlus does not include Enterprise RPMs for CentOS 5, just CentOS 4. Its also worth noting that patches are being applied, that are out of tree, to the RPMs. A snippet from the RPM changelog (rpm -q --changelog <packagename> if you must):

* Mon Dec 24 2007 Johnny Hughes <johnny@centos.org> 5.0.54-1.el4.centos
- upgraded to the 5.0.54 Enterprise BK tree. Removed patches that are
already incorporated into 5.0.54.


- added mysql-5.0.52-mysqldump-hang-33057.patch for mysql bug #33057


- added mysql-5.0.50-openssl-handshake-33050.patch fo mysql bug #33050


* Sun Dec 23 2007 Johnny Hughes <johnny@centos.org> 5.0.48-3.el4.centos
- modified the process of obtaining the Enterprise Sources to using bkf and
downloading directly from the MySQL Enterprise BitKeeper tree.

So, definitely interesting times. If you hit a bug, you might be wondering if its an upstream bug, or something that was provided by your packager. Of course, this is what is so great about the MySQL community – the ability to do just this! Currently, against 5.0.54, CentOSPlus is shipping 2 patches, with a few extra source packages that don’t come with regular tarballs. Will this number grow? Just remember, the “Enterprise” version you get in CentOSPlus is not “MySQL Enterprise”.

Technorati Tags: , , ,

At the ACM

Hmm. I’ve spent about 31 hours in flight and in transit, to get to Orlando, Florida. Good news is that I’ve arrived, all safe and dandy.

Singapore Airlines is now flying the A340-500 to Los Angeles or San Francisco, from Singapore. Its truly got to be the best plane for long haul flights. Notice that you get direct flights to America? No more transiting in Narita. I was given a seat in Executive Economy Class (I wonder why? Maybe its because of my collected miles/status, as it used to and still does happen on United, a Star Alliance partner). What’s cool there? Power. Yes, nice, in-flight power, suitable for devices that support 110V (read: all modern laptop PSUs).

Food was great (new menus), and I tried the much recommended Singapore Sling, and realised that it tastes pretty darn good. Highly recommended if you’re on Singapore Airlines anytime soon. Oh immigration – long queues at LAX as usual, however, they’re still taking 2 fingerprints, as opposed to 10. Odd?

United was acceptable as usual. No personal video screens, no power, but at least its those nicer/larger Economy Plus seats. I sincerely think that the best Star Alliance airline has got to be Singapore Air (trust me, Lufthansa doesn’t come close).

Village MySQL
Kaj, and the Village MySQL banner

Arrived in Orlando, the pick-up was ready, and check-in was quick. As I was about to head to my room, I bumped into Kaj, and decided that I’d join him for registration and breakfast. Got to meet quite a few new hires, though we mostly ended up talking to Marten and Monty.

Village MySQL
MySQL: The Future is Wide Open

After breakfast, I bumped into Giuseppe and Trudy. Was good to see Giuseppe and catch up (I’m sure we’ll be doing a lot more of that this month). At this point I was getting tired, and my legs were aching. Found my room (very nice, pictures on Flickr of course), tried to nap, failed, and visited the Mandara Spa. Had a quick massage, a good shower, and now I think I’m ready to rest. No, I think I’ll go get some sun in first. This jet lag thing-magic can be beaten!

Technorati Tags: ,

linux.conf.au nears, come to mysql mini-conf

While I haven’t spent much time going through the linux.conf.au 2008 schedules, it is now less than a few weeks away, and I’m stoked. If you’re there for the Mini-conf’s, I suggest freeing up your Tuesday, to spend it at MySQL.conf.au :) There are a bunch of interesting talks, and there will be a lot of MySQL’ers around. In fact, Giuseppe Maxia is flying in all the way from Sardinia, Italy, to give a talk. And I can assure you, he gives entertaining Proxy talks.

When I have more time to read through the schedules, I’ll work up a timetable, I’m sure. Time? There’ll be plenty, as I hop on a plane next Monday for a lengthy flight to Florida.

Technorati Tags: , , ,


i