MySQL optimisation Percona LeMug.fr

MySQL Optimization
(A couple of different ways.)

Morgan Tocker
morgan@percona.com

1
Tuesday, March 8, 2011

Show of Hands
★ InnoDB?
★ MyISAM?
★ XtraDB?
★ MySQL 4.x?
★ MySQL 5.0.x?
★ MySQL 5.1?
★ MySQL 5.5?
★ Something else?

2

About this presentation
★ Q: “Would you like to talk about MySQL optimization?”
★ A: Sure.

★ Optimization can be made at many levels.
★ I’m going to show 2-3 parts of the puzzle:
✦
Query Optimization.
✦
How InnoDB works.
✦
(Time Permitting) Our work at Percona on improving MySQL.

3

It all starts with EXPLAIN
★ Bookmark this manual page:
http://dev.mysql.com/doc/refman/5.1/en/using-
explain.html
★ It is the best source for anyone getting started.

4

Examples come from:
★ IMDB database loaded into
InnoDB tables (~5G).
★ Download it and import it for
yourself using imdbpy2sql.py:
http://imdbpy.sourceforge.net/

5

First Example

6

Find the Title Bambi 3.09s

ALL means
tablescan

Anticipated number In this case a sort is
of rows to be required because of the
examined order by

Additional filtering may
be possible before
passing to sort.

7

Aha! Now add an index:

8

We must revisit... 0.00s

Using = for comparison, but not
primary key lookup.

Identified title as a
candidate index,
chose to use it.

Size of the index
used (in bytes)

Anticipated number of
rows to be examined
dropped considerably.
9

Other ways of accessing 0.00s

At most one
Better type of ‘const’. We can
matching row.
stop when we find one row.

In InnoDB the primary key is
often much faster than all
other keys.

10

And.. 0.00s

Type is range. BETWEEN, IN()
and < > are also ranges.

Number of rows to be examined
has increased - we are not
specific enough.

Ignore the time with
EXPLAIN. Only look at
the time for a query.

11

Why’s that a range?
★ We're looking for titles between BambA and BambZ*
★ When we say index in MySQL, we mean trees.
✦
That is, B-Tree/B+Tree/T-Tree.
✦
Pretend they're all the same (for simpliﬁcation).
✦
There's no radically different indexing methods in MySQL
unless you play storage engine Bingo**.

* In reality the range is a little wider
12 ** The memory storage engine supports hash indexes

What’s that?

13

Could this be a range? 3.2s

14

No, we can’t traverse.

Do we head left or right here?

15

LIKE ‘Z%’ 0.05s

16

LIKE ‘T%’ 3.13s

17

LIKE ‘The %’ 3.07s

18

MySQL is (reasonably) smart.
★ It dynamically samples the data to choose which is the
better choice - or in some cases uses static statistics*.
★ This helps the optimizer choose:
✦
Which indexes will be useful.
✦
Which indexes should be avoided.
✦
Which is the better index when there is more than one.

19 * To refresh statistics run ANALYZE TABLE table_name;

Why avoid indexes?
★ B-Trees work like humans search a phone book;
✦
Use an index if you want just a few rows.
✦
Scan cover-to-cover if you want a large percentage.

20

Why avoid indexes (cont.)
★ We benchmarked this on a different schema:

Table scan has a relatively
fixed cost (red line).

The index has completely
different effectiveness
depending on how much it
can filter.

Hopefully MySQL switches
at the right point.

21

What you should take away:
★ Data is absolutely critical.
✦
Development environments should contain sample data
exported from production systems.
★ Input values are absolutely critical.
✦
Between two seemingly identical queries, execution plans
may be very different.

See also: http://www.mysqlperformanceblog.com
22 /2009/10/16/how-not-to-find-unused-indexes/

Improve this Query 3.41s

This number of rows is a
guess. It keeps changing
between examples.

23

We’re Spoiled for Choice.

24

Index on production_year 3.53s

25

Might work if... 0.92s

26

Index on title(50) 0.02s

27

Comparing the two:
★ mysql> EXPLAIN SELECT * from title WHERE title = 'Pilot' AND
production_year BETWEEN 2006 and 2009G

28

Composite Indexes.
★ What is better?
✦
INDEX py_t (production_year, title)
✦
INDEX t_py (title, production_year)

29

Index on py_t 0.02s

http://www.mysqlperformanceblog.com/2010/01/09/getting-around-
30 optimizer-limitations-with-an-in-list/

Index on t_py 0.00s

31

Recommendations
★ Index over multiple columns if it can improve ﬁltering. i.e.
✦
GOOD: Only some pilots made between 2006-2009.
✦
BAD: All pilots made between 2006-2009.

32

Recommendations (cont.)
★ Don't know what order to specify the columns?
✦
RULE: Think how to ﬁlter the fastest. Use that order left to
right.
✦
EXCEPTION: If there's a range (><, BETWEEN, %). Those
always go to the RIGHT.

http://www.mysqlperformanceblog.com/2010/01/09/getting-around-
33 optimizer-limitations-with-an-in-list/

How InnoDB Works


“Numbers everyone should know”

L1 cache reference 0.5 ns
Branch mispredict 5 ns
L2 cache reference 7 ns
Mutex lock/unlock 25 ns
Main memory reference 100 ns
Compress 1K bytes with Zippy 3,000 ns
Send 2K bytes over 1 Gbps network 20,000 ns
Read 1 MB sequentially from memory 250,000 ns
Round trip within same datacenter 500,000 ns
Disk seek 10,000,000 ns
Read 1 MB sequentially from disk 20,000,000 ns
Send packet CA->Netherlands->CA 150,000,000 ns

See: http://www.linux-mag.com/cache/7589/1.html and Google http://
35 www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

About Disks.
★ 10,000,000 ns = 10ms = 100 operations/second.
✦
This is about the average for a 7200RPM drive.
★ The actual time has dramatic variation.
✦
The variation is because disks are mechanical.
✦
We can much write faster sequentially than randomly.

36

[default] Everything is buffered!
★ When you write to a ﬁle, here’s what happens in the
Operating System:
Block 9, 10, 1, 4, 200, 5.

Block 1, 4, 5, 9, 10, 200

What happens to this
buffer if we loose power?

37

The OS provides a way!
★ $ man fsync
Hint: MyISAM just writes to the
OS buffer and has no durability.
Synopsis
#include <unistd.h>
int fsync(int fd);
int fdatasync(int fd);

Description
fsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache
pages for) the file referred to by the file descriptor fd to the disk device (or other
permanent storage device) where that file resides. The call blocks until the device
reports that the transfer has completed. It also flushes metadata information
associated with the file (see stat(2)).

38 http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/

Knowing this:
★ InnoDB wants to try and reduce random IO.
★ It can not (safely) rely on the operating system’s write
buffering and be ACID compliant.
✦
.. and InnoDB algorithms have to compensate.

39

Basic Operation (High Level)

Log Files

SELECT * FROM City
WHERE CountryCode=ʼAUSʼ

Tablespace
Buffer Pool

40

Basic Operation (cont.)

Log Files

UPDATE City SET
name = 'Morgansville'
WHERE name = 'Brisbane'
AND CountryCode='AUS'

Tablespace
Buffer Pool

41

Basic Operation (cont.)

01010

Log Files

UPDATE City SET

Tablespace
Buffer Pool

41

Why don’t we update?
★ This is an optimization!
✦
The log ﬁle IO is sequential and much cheaper than live
updates.
✦
The IO for the eventual updates to the tablespace can be
optimized as well.
★ Provided that you saved enough to recover, this
shouldn’t matter should it?

42

More on Logs...
★ Logs are only used during recovery.
✦
Not even read when we need to write down
dirty pages! Log Files
★ To ﬁgure out which pages need to be evicted we have two
lists - the ﬂush list and the LRU.
★ Log activities are all assigned a LSN (log sequence
number).

43

Log Files, Checkpoints, etc.
★ Most database systems work this way:
✦
In Oracle the transaction logs are called “Redo Logs”.
★ The background process of syncing dirty pages is
normally referred to as a “Checkpoint”.
★ InnoDB has fuzzy checkpointing.

44

Log Writing
★ You can change increase innodb_log_file_size.
This will allow InnoDB to “smooth out” background IO
for longer.

✦
Tip: Optionally you could change
innodb_log_files_in_group as well. Be aware that
your effective log ﬁle is innodb_log_file_size *
innodb_log_files_in_group

45

Log Writing (cont.)
★ You can also change
innodb_flush_log_at_trx_commit to 0 or 2 to
reduce the durability requirements of this write.
✦
Requires less ﬂushing - particularly helpful on systems
without writeback caches!
★ innodb_log_buffer_size may also help buffer
changes for longer before writing to the logs.
✦
Very workload dependent - tends to be more helpful for
writing big TEXT/BLOB changes.

46

In summary
★ On commit, the log has to be ﬂushed to guarantee
changes.
✦
Nothing else has to be done.
★ What visibility do we have into these operations?
★ How do we decide how much background work to do per
second?
★ What happens if we fall behind in background work?

47

Enhancements via Percona


Terminology

Oracle Product License Percona Equivalent License
Product
MySQL Server GPL Percona Server GPL

The InnoDB Storage GPL The XtraDB Storage GPL
Engine (Plugin edition) Engine
InnoDB Hot Backup Commercial XtraBackup GPL

(GPL = Completely free for you to use. Support not included.)

49

Performance Improvements
Transaction logs
Improved IO Path larger than 4G
+ adaptive checkpointing supported.
Improved Buffer Separate location
Pool Scalability of double write
buffer* Remove excess fcntl calls
Insert buffer controls
Improved Rollback
Separate purge Strip comments before Segment Scalability*
thread using query cache

Completely Data dictionary
disable the query memory Increased number of
cache. consumption undo slots*
controls
Per session configuration of
Faster page Support for different innodb_flush_log_at_trx_commit
checksums* page sizes*

50 * Changes on disk format (not backwards compatible)

Improved Buffer Pool Scalability
★ Additional patches to what is already available in the
InnoDB Plugin.
✦
Splits the buffer pool mutex into smaller mutexes:
• Flush list mutex
• LRU mutex
• Free mutex
• hash mutex

51

Data Dictionary control
★ Once an InnoDB table is opened it is never freed from
the in-memory data dictionary (which is unlimited in
size).
★ XtraDB introduces:
✦
--innodb_dict_size_limit - a conﬁguration item in
bytes.
✦
innodb_dict_tables - a status variable for number
entries in the cache.

52

Undo Slots
★ In the built-in InnoDB, the number of undo slots is
limited to 1024.
✦
This means the number of open transactions is limited to
1023. See: http://bugs.mysql.com/bug.php?id=26590
✦
Some statements require 2 undo slots.
★ In XtraDB, this is expanded to 4072 with --
innodb_extra_undoslots=1.

53 Warning: This is binary format incompatible!

Rollback Segments
★ In XtraDB, it’s also possible to have more than one
rollback segment.
✦
Each segment contains undo slots.
★ Conﬁguration is --innodb-extra-rsegments=N
★ This has the added effect of reducing mutex contention
on the rollback segment:
✦
“Mutex at 0×1b3e3e78 created ﬁle trx/trx0rseg.c line 167″

http://www.percona.com/docs/wiki/percona-
xtradb:patch:innodb_extra_rseg
http://www.mysqlperformanceblog.com/2009/10/14/tuning-
for-heavy-writing-workloads/


Fast Checksums
★ The InnoDB page checksum computation is slower than
it needs to be.
★ XtraDB has the option to use a faster checksum format.


Different Page Sizes
★ XtraDB now has support for different page sizes - 4K,
8K, 16K.


Separate Purge Thread
★ Cleans up a long history list length faster:

See: http://www.mysqlperformanceblog.com/2009/10/14/
57 tuning-for-heavy-writing-workloads/

Usability Enhancements
Show data
User / Index / Table statistics
dictionary
Show contents of
Log connection
the buffer pool
errors Disable automatic
Import / export of statistics
Save index statistics innodb_file_per_table regeneration
between restarts tables

Transactional
Import / export of Deadlock counter Replication
buffer pool
Advise in
contents Retain query processlist when
response time waiting on Query Improved slow
distribution. cache mutex. query log
Store buffer pool
in shared memory
segment Better handling of
Show Temporary Tables
corrupted tables

58

Show Buffer Pool Contents
mysql> SELECT d.*,round(100*cnt*16384/(data_length+index_length),2) fit FROM
(SELECT schema_name,table_name,count(*) cnt,sum(dirty),sum(hashed) FROM
INNODB_BUFFER_POOL_PAGES_INDEX GROUP BY schema_name,table_name ORDER BY cnt DESC
LIMIT 20) d JOIN TABLES ON (TABLES.table_schema=d.schema_name AND
TABLES.table_name=d.table_name);
+-------------+---------------------+---------+------------+-------------+--------+
| schema_name | table_name | cnt | sum(dirty) | sum(hashed) | fit |
+-------------+---------------------+---------+------------+-------------+--------+
| db | table1 | 1699133 | 13296 | 385841 | 87.49 |
| db | table2 | 1173272 | 17399 | 11099 | 98.42 |
| db | table3 | 916641 | 7849 | 15316 | 94.77 |
| db | table4 | 86999 | 1555 | 75554 | 87.42 |
| db | table5 | 32701 | 7997 | 30082 | 91.61 |
| db | table6 | 31990 | 4495 | 25681 | 102.97 |
| db | table7 | 1 | 0 | 0 | 100.00 |
+-------------+---------------------+---------+------------+-------------+--------+
7 rows in set (26.45 sec)

Source: http://www.mysqlperformanceblog.com/
2010/03/26/tables-fit-buffer-poo/
59 * Command may differ slightly between versions.

Save Buffer Pool Contents
★ Export the contents of the buffer pool to a ﬁle called
‘ib_lru_dump’ in the data directory:
✦
SELECT * FROM
information_schema.XTRADB_ADMIN_COMMAND /*!
XTRA_LRU_DUMP*/;
★ Restored ib_lru_dump:
✦
SELECT * FROM
information_schema.XTRADB_ADMIN_COMMAND /*!
XTRA_LRU_RESTORE*/;

Note: Not the actual contents - it takes 8 bytes to
60 remember the address of a 16K page.

Import/Export tables
★ Because --innodb-file-per-table still has
information (data dictionary, undo) in the global
tablespace you can’t just back it up by itself.
★ With a new setting, --innodb_expand_import=1,
this is no longer the case.

★ Tip: The import/export still has to be done with
XtraBackup. Documentation available here:
http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_expand_import

61

The Slow Query Log
$ tail /var/log/mysql.slow
..
# Time: 100924 13:58:47
MySQL Server
# User@Host: root[root] @ localhost []
# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608
SET timestamp=1285336727;
select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON
(cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;

$ mysql -e “SET GLOBAL log_slow_verbosity = ‘full’;”
$ tail /var/log/mysql.slow Percona Server
..
# Time: 100924 13:58:47
# User@Host: root[root] @ localhost []
# Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0
# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 Rows_affected: 0 Rows_read: 1
# Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: 25194923
# InnoDB_trx_id: 1403
# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes
# Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5
# InnoDB_IO_r_ops: 1064749 InnoDB_IO_r_bytes: 17444847616 InnoDB_IO_r_wait: 26.935662
# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
# InnoDB_pages_distinct: 65329
SET timestamp=1285336727;
select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON
(cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;

62

(Related) Xtrabackup Features
★ Report on fragmentation of indexes:
★ $ xtrabackup --stats --tables=art.link* --
datadir=/mnt/data/mysql/
...
table: art/link_out104, index: PRIMARY, space
id: 12, root page 3
leaf pages: recs=25958413, pages=497839,
data=7492026403 bytes, data/pages=91%
...

http://www.mysqlperformanceblog.com/2009/09/14/statistics-
of-innodb-tables-and-indexes-available-in-xtrabackup/
64

Basic Operation (again.)

Log Files

UPDATE City SET

Tablespace
Buffer Pool

65


01010

Log Files

UPDATE City SET

Tablespace
Buffer Pool

65

Set innodb_buffer_pool_size to
“50-80%” of memory.
01010

Log Files

UPDATE City SET

Tablespace
Buffer Pool

65

01010

Log Files

UPDATE City SET

Increase the size of the log
files (innodb_log_file_size
and
innodb_log_files_in_group) Tablespace
Buffer Pool

65

01010

Log Files Set
innodb_flush_log_at_trx_commit=2
if durability is not as important.
UPDATE City SET

and
Buffer Pool

65

01010

Log Files Set
UPDATE City SET
WHERE name = 'Brisbane' Typically use
innodb_flush_method=O_DIRECT
if using a Hardware RAID controller.
and
Buffer Pool

65

(Possibly) Move the log Set innodb_buffer_pool_size to
files to separate spindles “50-80%” of memory.
(sequential IO) 01010

Log Files Set
UPDATE City SET
and
Buffer Pool

65

(Possibly) Move the log Set innodb_buffer_pool_size to
files to separate spindles “50-80%” of memory.
(sequential IO) 01010

Log Files Set
If innodb_log_waits > 0 the innodb_flush_log_at_trx_commit=2
buffer being filled if durability is not as important.
(innodb_log_buffer_size)
UPDATE City SET
before'Morgansville' the log files
name =
writing to
may be too small.
and
Buffer Pool

65

The End.
★ Questions?

66

MySQL optimisation Percona LeMug.fr

More Related Content

Viewers also liked (12)

Similar to MySQL optimisation Percona LeMug.fr (20)

Recently uploaded (20)

MySQL optimisation Percona LeMug.fr