Colin Charles Agenda

Over the weekend I read in the FT (paywall): Is Brussels safe? Ring a local resident to find out. I’m sure it will be fine, and you will want to be there for FOSDEM, happening 30-31 January 2016. 

There is the excellent one day track, that is the MySQL & Friends Devroom (site). Talks hail from Oracle, MariaDB Corporation, Percona and more. We don’t have a booth this year, but we do have amazingly good content on Saturday. I’m happy to have been part of the committee that chose the talks, but you know that this is a labour of love put on by Frédéric Descamps, Liz van Dijk, Dimitri Vanoverbeke, and Kenny Gryp. I’m sure the party will be awesome.

But that is not all! In the distributions devroom, you can see me give a talk at 11:00-11:20 titled Distributions from the view of a package. This is an important topic, because you start seeing MariaDB Server becoming the default in many distributions with the last holdout being Debian. But there is a lot of discussion, especially from the security standpoint there now, about MySQL overall. But that’s not the focus of my talk – I’m going to talk to you about how we, as upstream, have had to deal with distributions, changing requirements, etc. overall. I’ve done this since the MySQL days, so have quite a bit of experience dealing with it. 

If you are making software and want to be included and supported across all distributions, I highly recommend you coming to my talk. If you happen to decide to live in an ecosystem where there are forks, I also promise to make it useful for you.

And on Sunday, you will want to go visit the RocksDB Storage Engine for MySQL talk by none other than Yoshinori Matsunobu of Facebook. This will be at the main track and I highly recommend you visit it – I’m sure Sergei Petrunia will also make an appearance as he spends a lot of time on this too.

All in, I’m extremely excited to be at FOSDEM 2016. And you don’t need to ring a local resident to find out if its going to be safe/fun – come for the learning, stay for the beer ;-)

FOSDEM happens this year, February 1&2 2014. This year is a special year as it is the 10th anniversary: great content in the schedule, you must pre-register for dinner (lot’s of wonderful Belgian food & drink), and we have a shared booth in the expo hall.

Read more about the event with a wonderful post by Kenny. Again, much thanks to Liz, dim0, Kenny, Frederic for organising the venue, and the sponsors Oracle/SkySQL/Percona.

SkySQL is well represented: 

1. 13:35-14:05 – MariaDB CONNECT Storage engine – Serge Frezefond
2. 15:35-16:05 – SPIDER Storage Engine – Colin Charles, Stephane Varoqui
3. 18:05-18:35 – OSM data in MySQL – Hartmut Holzgraefe
4. 18:35-19:05 – MaxScale, the Pluggable Router – Massimiliano Pinto, Vilho Raatikka

See you all at the Friday beer event on 31.01.2013!

As Frederic posted, its time to submit talks for the MySQL & Friends Devroom at FOSDEM 2014. The next year, it will be on Saturday February 1 2014. I look forward to being in Brussels again, and I hope to see you there too.

Submit to the MySQL track here, and don’t forget to be there on Friday evening for the start of the beers. I’m told by Frederic & Kenny that we’re likely to have a much more interesting community dinner since things are getting larger year by year. See you at FOSDEM and remember, submit talks!

February is turning out to be a busy month. I’ll be giving a talk about MariaDB at the SkySQL Solutions Day in Hamburg, alongside Sergei Golubchik. This happens February 1 2013.

I’ll also be in Brussels for FOSDEM and presenting at the MySQL & Friends devroom. Here I will talk about MHA and how you can get automated MySQL failover. I’ll also be hanging out at the CentOS booth with Karanbir Singh giving away MariaDB stickers all weekend. If you’re at FOSDEM, don’t forget to sign up for the MySQL Community Dinner on Saturday night! Naturally, Friday’s beer activity is something I’ll be at too.

Busy long weekend ahead!

Frédéric Descamps of Percona.

Percona Toolkit is Maatkit & Aspersa combined. Opensource and the tools are very useful for a DBA.

You need Perl, DBI, DBD::mysql, Term::ReadKey. Most tools are written in Perl, and whatever is in Bash is being re-written in Perl. There is also a tarball or RPM or DEB packages.

Know your environment. The hardware & OS are crucial for you to know. How much memory/CPU do you use? Do you use swap? Is this a physical/virtual machine? Do you have free space? What kind of RAID controller? Volumes? Disk? What about the network interfaces? What IO schedulers are used? Which filesystem is the data stored on? To answer all that, just use pt-summary.

Know your MySQL environment. Version? Build? How many databases? Where is the data directory? What about replication? What are key InnoDB settings? Storage engine in use? Index type? Foreign keys? Full text indexes? To answer all this and more use pt-mysql-summary.

pt-slave-find shows you the topology and replication hierarchy of your MySQL replication instances. An inventory of replicas!

Where is my disk I/O going? Use pt-diskstats which is an improved iostat. There is pt-ioprofile but it can be dangerous in production.

Now its time to get more intimate with your database. Let’s try to find the answer to these questions: how are the indexes used? Are there duplicate keys? Which queries are eating most of the resources? You can use pt-duplicate-key-checker to check for duplicate/redundant indexes or foreign keys. pt-index-usage can tell you which indexes are unused. If you think you have bad SQL, check out pt-query-advisor.

You can use pt-query-digest to analyze the slow query log and show a profile of the workload. You mostly use this with slow query logs & tcpdump’s. Be careful when you have dropped packets — results may tend to be fake then!

After all this, its time to maintain your environment.

pt-deadlock-logger checks InnoDB status to log MySQL deadlock information. It needs to run continually to capture things.

pt-fk-error-logger extracts and logs MySQL foreign key errors.

pt-online-schema-change to alter tables. It makes a “shadow copy” and swaps them. Extremely useful for large, long-running ALTER. Facebook uses the same technique.

Validate your upgrades as upgrades are the leading cause of downtime. Are queries using different indexes? Is query execution plan different? New errors? See pt-upgrade for this. Best to run this on a third machine (i.e. the old machine and a new machine to see how it goes).

Verify replication integrity – pt-table-checksum. Perform an online replication consistency check or checksum MySQL tables efficiently on one or many servers. Use it routinely (mandatory for 95% of MySQL users). Put it in a weekly crontab. Repair differences with pt-table-sync.

Repair out-of-sync replicas – pt-table-sync

Measure delay acfurately – pt-heartbeat

Deliberately delay replication – pt-slave-delay

Watch & restart MySQL replication after errors – pt-slave-restart

When there are problems, get the symptoms when it hurts. Look at pt-stalk (wait for a condition to occur them begin collecting data – eg. everytime the threads go over 2,000 you have a problem, so it collects stuff – it calls pt-collect), pt-collect (collect information from a server for some period of time), and pt-sift.

pt-mext looks at many samples of MySQL SHOW GLOBAL STATUS side-by-side. Default STATUS shows counter since the MySQL instances started. It is very helpful to see a delta of recent activity.

The future: pt-query-digest will do query reviews; pt-stalk will do “magical fault detection algorithm”. Its all opensource and its all on Launchpad at lp:percona-toolkit.

Sergey Petrunia of the MariaDB project & Monty Program.

MySQL 5.5 GA at the end of 2010. MariaDB 5.3 RC towards the end of 2011 (beta in June 2011).

MySQL 5.5 is merged to Percona Server 5.5 which included semi-sync replication, slave fsync options, atuomatic relay log recovery, RBR slave type conversions (question if this is useful or not), individual log flushing (very useful, but not many using), replication heartbeat, SHOW RELAYLOG EVENTS. About 2/3rds of the audience use MySQL 5.5 in production, with only 2 people using semi-sync replication.

MariaDB 5.3 brings replication features brings group commit in the binary log, which is merged into Percona Server 5.5. Checksums for binlog events which is merged from MySQL 5.6. Sergey goes in-depth about the group commit for the binary log. To find out a little more about MariaDB replication changes, see Replication in the Knowledgebase.

There are several implementations of group commit. Facebook started it, followed by MariaDB & Oracle. Percona 5.5 is GA so the feature is there, its not in MySQL 5.6 (yet?), and MariaDB 5.3 is where its at. Seems like the MariaDB implementation is the best so far – refer to the Facebook benchmark performed by Mark Callaghan.

Annotated RBR poses a compatibility problem. MariaDB 5.3 has annotate_rows, while MySQL 5.6 has rows_query event. They are different events. So you cannot have a MariaDB 5.3 master and a MySQL 5.6 slave at this moment. So MySQL 5.6 will have a flag to mark “ignorable” binlog events which will be merged into MariaDB and this will make binary logs compatible again.

There is now also optimized RBR for tables with no primary key.

MySQL 5.6 also has crash-safe slave (replication information stored in tables). Crash-safe master (binary log recovery if the server starts & sees the binary log is corrupted). Parallel event execution is something that is new in MySQL 5.6 which is the most important feature for Sergey.

Pre-heating: There is mk-slave-prefetch (famous quote: “Please don’t use mk-slave-prefetch on #MySQL unless you are Facebook.”). There is replication booster by Yoshinori Matsunobu. There is a Python version of mk-slave-prefetch that Facebook uses.