Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on ...

Using ZFSTM Snapshots With
Zmanda Recovery Manager for MySQLTM
on OpenSolarisTM 2008.11 OS
Caryl Takvorian
May 2009
Sun Microsystems, Inc.

This article was published by BigAdmin at:
http://www.sun.com/bigadmin/features/articles/zrm_mysql_zfs.jsp
To keep track of the latest content published by BigAdmin, subscribe
to the BigAdmin newsletter: http://www.sun.com/bigadmin/newsletter/.

Copyright © 2009 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.

U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard
license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms.
This distribution may include materials developed by third parties.

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a
registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. X/Open is a
registered trademark of X/Open Company, Ltd.

AMD, Opteron, the AMD logo, the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro
Devices.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation in the United States and other countries.

Sun, Sun Microsystems, the Sun logo, MySQL, OpenSolaris, Solaris, Sun BluePrints, Sun Fire, SunSolve, and ZFS are
trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries.

This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws in
other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether direct
or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities identified on U.S.
export exclusion lists, including, but not limited to, the denied persons and specially designated nationals lists is strictly
prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS
AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

2 Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on OpenSolaris 2008.11 OS

Table of Contents
Introduction................................................................................................................................................5
Architecture Overview...............................................................................................................................6
OS Configuration..................................................................................................................................6
The ZFS File System.............................................................................................................................6
Zpool Configuration.........................................................................................................................7
ZFS Configuration............................................................................................................................7
Configuration of Zmanda Recovery Manager 3.0 for MySQL.............................................................8
Installing Dependencies Using pkgadd...........................................................................................9
Installing Dependencies Using IPS..................................................................................................9
Installing ZRM...............................................................................................................................10
Types of Backup Available to ZRM...................................................................................................11
Raw-Copy Backup..........................................................................................................................11
Logical Backup...............................................................................................................................11
Raw-Quick (No-Copy) Backup .....................................................................................................11
Performance Testing Setup......................................................................................................................13
System Setup.......................................................................................................................................13
MySQL Software Release...................................................................................................................13
OpenSolaris Operating System Release..............................................................................................13
Zpool Configuration............................................................................................................................13
ZFS Configuration...............................................................................................................................14
ZRM Software Release.......................................................................................................................14
db_STRESS Load Generator...............................................................................................................14
1. Creating the test databases..........................................................................................................14
2. Creating test tables in pool2/mysql............................................................................................14
3. Generating the dataset.................................................................................................................15
4. Loading the dataset into the tables..............................................................................................15
5. Creating users.............................................................................................................................15
6. Creating a test scenario ..............................................................................................................15
Testing Methodology..........................................................................................................................15
Performance Test Results.........................................................................................................................17
1. MyISAM Results with Idle Database..............................................................................................17
2. InnoDB Results with Idle Database................................................................................................17
3. MyISAM Results with a 15 Percent Load on the Database ...........................................................19
4. InnoDB Results with a 15 Percent Load on the Database ..............................................................20
5. MyISAM Results with a 90 Percent Load on the Database ...........................................................21
6. InnoDB Results with a 90 Percent Load on the Database ..............................................................22
Product Information.................................................................................................................................23
OpenSolaris Operating System...........................................................................................................23
The ZFS File System...........................................................................................................................23
The Sun Fire X4500 Server.................................................................................................................23
Zmanda Recovery Manager 3.0 for MySQL.......................................................................................23
Summary..................................................................................................................................................24
About the Author ....................................................................................................................................24
For More Information..............................................................................................................................25
Licensing Information..............................................................................................................................27


Introduction
The objective of this report is to show how to deploy Zmanda Recovery Manager 3.0 for MySQL™ on
the OpenSolaris™ 2008.11 OS and how ZRM performs on a Sun Fire™ X4500 system using the ZFS™
file system. In particular this article demonstrates how ZRM takes advantage of ZFS snapshot
technology to offer a best-in-class open source data-protection solution for MySQL.
It is intended to be read in conjunction with the article Using Sun Fire X4540 Server With Zmanda
Recovery Manager 2.2 for MySQL Database by Thomas Hanvey (Sun Microsystems) and Dmitri
Joukovski and Ken Crandall (Zmanda), October 2008, which is available at
http://www.sun.com/bigadmin/features/articles/zrm_mysql_sfx4540.jsp.
All the steps described in this document were carried out on a single Sun Fire X4500 system which
acted as the ZRM backup server and the MySQL database server.
For more information on the OpenSolaris 2008.11 OS and Zmanda technology, see the Product
Information section at the end of this article. For further details on ZRM Enterprise, see:
http://zmanda.com/zrm-mysql-enterprise.html.
Zmanda Recovery Manager for MySQL is available for download from the Zmanda network:
http://network.zmanda.com/.


Architecture Overview

Figure 1: Architecture Overview

OS Configuration
For this guide, ZRM was deployed on a Sun Fire X4500 server running the OpenSolaris 2008.11
release.
For more information on the Sun Fire X4500 server, see the Product Information section at the end of
this article.
The system used during this test is a server with two dual-core AMDTM Opteron 285 processors running
at 2593 MHz with 16 Gbyte of RAM and 48 SATA drives of 512 Gbyte each, for a total of 24 Tbyte of
storage capacity.
The two boot drives are mirrored using ZFS volume management capabilities and the remaining 46
drives are available for backup data.

The ZFS File System
For more information on ZFS, see the Product Information section at the end of this article.


One of the most interesting features of ZFS when combined with backup technologies is its snapshot
capabilities. A snapshot is a read-only copy of a file system or volume which can be created very
quickly. Thanks to ZFS copy-on-write characteristics, snapshots are essentially a free operation, which
initially consume no additional space within the pool. However, as data within the active dataset
changes, the snapshot consumes space by continuing to reference the old data and so prevents the data
from being freed back to the pool.
ZFS snapshots include the following features:
● Persist across system reboots.
● Use no separate backing store. Snapshots consume disk space directly from the same storage
pool as the file system from which they were created.
● Recursive snapshots are created quickly as one atomic operation. The snapshots are created
together (all at once) or not created at all. The benefit of atomic snapshot operations is that the
snapshot data is taken at one consistent time, even across descendent file systems.

Zpool Configuration
For the purposes of this guide, the storage available on the Sun Fire X4500 server was split into two
pools:
● A backup pool made of 24 drives and two spares in a RAIDZ2 configuration. This provided 11
Tbyte of capacity for backups with a very high level of protection against drive failure and
performance sufficient to handle 300 to 350 Mbyte/sec of backup traffic over the network.
● A mysql pool made of 12 drives and two spares in a RAIDZ2 configuration was used to store
the MySQL databases.
First, we created the RAIDZ2 pool zmanda with six drives. Then, we added the other drives by groups
of six and verified at the end that we ended up with a pool with 10.9 Tbyte of physical space.
Here are zpool creation commands for pool zmanda:
# zpool create -f zmanda raidz2 c3t1d0 c4t1d0 c6t1d0 c7t1d0 c8t1d0 c9t1d0
# zpool add zmanda raidz2 c3t2d0 c4t2d0 c6t2d0 c7t2d0 c8t2d0 c9t2d0
# zpool add zmanda spare c8t0d0 c9t0d0

We used a similar command to create the pool pool2.
Here are the zpool creation commands for pool pool2:
# zpool create -f pool2 raidz2 c3t3d0 c4t3d0 c6t3d0 c7t3d0 c8t3d0 c9t3d0
# zpool add -f pool2 raidz2 c3t7d0 c4t7d0 c6t7d0 c7t7d0 c8t7d0 c9t7d0
# zpool add -f pool2 spare c8t4d0 c9t4d0

Finally we can check the ZFS pools just created:
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
pool2 5.44T 225K 5.44T 0% ONLINE -
rpool 464G 13.3G 451G 2% ONLINE -
zmanda 10.9T 252K 10.9T 0% ONLINE -


ZFS Configuration
Next, we need to create the ZFS file system which will be used by ZRM and MySQL. We will need
three different ZFS file systems:
● zmanda/zrm to hold the ZRM backups
● zmanda/logs to hold the MySQL binlogs
● pool2/mysql to hold the MySQL data files
So, first let's create the ZFS file system where ZRM will store the backup images, and mount it in the
default location expected by ZRM:

# zfs create -o mountpoint=/var/lib/mysql-zrm zmanda/zrm

Then, create the ZFS file system where the MySQL data files will be stored, and the third file system to
store the MySQL log files.
# zfs create zmanda/logs
# zfs create pool2/mysql

Finally we can list the file systems available to us:
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
pool2 1.35T 2.22T 1023G /pool2
pool2/mysql 356G 2.22T 356G /pool2/mysql
rpool 185G 272G 60K /rpool
rpool/ROOT 150G 272G 18K legacy
rpool/ROOT/opensolaris-1 150G 272G 121G legacy
rpool/dump 7.95G 272G 7.95G -
rpool/export 19.1G 272G 19K /export
rpool/export/home 19.1G 272G 18.9G /export/home
rpool/swap 8.16G 280G 216M -
zmanda 50.0G 7.08T 38.0K /zmanda
zmanda/logs 815K 7.08T 815K /zmanda/logs
zmanda/zrm 87.9K 7.08T 87.9K /var/lib/mysql-zrm

Configuration of Zmanda Recovery Manager 3.0 for MySQL
For more information on ZRM, see the Product Information section at the end of this article.
The Sun Fire X4500 server is configured as a ZRM server with Zmanda Management Console (ZMC).
To install ZRM and ZMC, follow these overall steps, which are explained in more detail in subsequent
sections.
1. From the Zmanda Network web site (which requires registration), download the following:
● ZRM installer binary for Solaris 10 systems with Intel® or AMD Opteron processors:
ZRM-enterprise-3.0-installer-intel.bin
● Prerequisite packages: solaris-10-zrm-dependency-intel.tar.gz
1. Extract the prerequisite packages, solaris-10-zrm-dependency-intel.tar.gz, and
install all the included packages.
2. Extract the binary package and run the Zmanda Rapid Installer to install ZRM.
3. After the installation is complete, the ZMC Welcome Page is displayed, and you can begin
configuring backup sets.


There are two ways of installing the required dependencies, using pkgadd or using Image Packaging
System (IPS).

Installing Dependencies Using pkgadd
The first option is to download the dependency package provided by Zmanda from the Zmanda
Network at http://network.zmanda.com. The required dependency packages are provided in a
compressed tar archive file, solaris-10-zrm-dependency-intel.tar.gz.
Extract all the packages:
# gtar zxvf solaris-10-zrm-dependency-intel.tar.gz

The dependency packages provided are as follows:
● SMCcoreu
● CSWgcc3corert
● CSWgcc3g++rt
● SMCgpgme
● SMCgrep
● SMClgcc346
● SMClgpger
● SMCliconv
● SMClintl
● SMCpcre
● SMCsudo
● SMCtar

These packages are also available from Sunfreeware at http://sunfreeware.org/ and from
Blastwave at http://blastwave.org.
Install each package using pkgadd:
# pkgadd -d <pkg_name>

Installing Dependencies Using IPS
The second option is to use the new OpenSolaris Image Packaging System. The most popular Solaris
packaging repositories such as http://blastwave.org and http://sunfreeware.org have already
converted the software they have on offer to the new IPS package format, and setting OpenSolaris to
use their repository is one single step.
Add the repository to the list of authorized package repositories:
# pkg set-authority -O http://blastwave.network.com:10000 blastwave

Let's make sure that everything is OK:
# pkg authority -H
AUTHORITY URL
blastwave http://blastwave.network.com:10000/
opensolaris.org (preferred) http://pkg.opensolaris.org:80/

Now we can start installing the CSW packages required by ZRM.


Note that the names of the Blastwave packages available for OpenSolaris IPS start with IPS instead of
the traditional CSW; for example, IPScommon instead of CSWcommon.

# pkg install IPScommon
DOWNLOAD PKGS FILES XFER (MB)
Completed 1/1 1/1 0.00/0.00
PHASE ACTIONS
Install Phase 146/146
PHASE ITEMS
Reading Existing Index 9/9
Indexing Packages 1/1

To show the ease of use of the OpenSolaris IPS packaging system, we'll install the complete list of
required packages in one single command:

# pkg install IPScurl IPSbzip2 IPSbdb4 IPScurlrt IPSexpat IPSfindutils IPSgfile IPSggettext
IPSgtar IPSiconv IPSlibidn IPSlibnet IPSoldaprt IPSossl IPSossldevel IPSosslrt IPSosslutils
IPSreadline IPSsasl IPSsunmath IPSzlib IPSglib2 IPSgcc3g++rt IPSmtx

Once finished, let's check that we have everything. (Once installed, the Blastwave packages use their
old CSW names.)

# pkginfo | grep CSW
application CSWbdb4 berkeleydb4 - Embedded database libraries and
utilities application CSWbzip2 bzip2 - a high-quality block-sorting file
compressor
system CSWcommon common - common files and dirs for CSW packages
application CSWcurl curl - Library and tool for common Internet
protocols
application CSWcurlrt curlrt - Library for common Internet protocols
Runtime
application CSWexpat expat - XML Parser Toolkit
system CSWfindutils findutils - A set of utilities for searching a file
system
application CSWgcc3corert gcc3corert - GNU C Compiler Run Time
application CSWgcc3g++rt gcc3g++rt - GNU C++ Compiler Run Time
system CSWgfile fileutils - GNU file utilities
system CSWggettext ggettext - GNU gettext
application CSWglib2 glib2 - the low-level core library for GTK+ and
GNOME
application CSWgtar gtar - GNU tape archiver
system CSWiconv libiconv - GNU iconv library
application CSWlibidn libidn - Implementation of the Stringprep, Punycode
and IDNA
system CSWlibnet libnet - the libnet packet construction library
application CSWmtx mtx - SCSI Media Changer and Backup Device Control
application CSWoldaprt openldap_rt - OpenLDAP runtime libraries (oldaprt)
application CSWossl openssl - Openssl meta package
application CSWossldevel openssl_devel - Openssl development support
application CSWosslrt openssl_rt - Openssl runtime libraries
application CSWosslutils openssl_utils - Openssl binaries and related tools
system CSWreadline readline - library to enable interactive line
editing
application CSWsasl sasl - Simple Authentication and Security Layer
system CSWsunmath libsunmath - Sun maths library
application CSWzlib zlib - Zlib Data Compression Library


Installing ZRM
Installing Zmanda Recovery Manager 3.0 for MySQL is a simple task of running the Zmanda Rapid
Installer which contains the ZRM Server, the ZRM Client, as well as the Zmanda Management Console
(ZMC) web user interface.
As we are connected to the Sun Fire X4500 through an SSH connection, we'll start the Rapid Installer
in text mode, as such:
# ./ZRM-enterprise-3.0-installer-intel.bin –-mode text

Once completed, the following software components are installed:
● The ZRM 3.0 for MySQL software
● Zmanda Management Console with MySQL, PHP, Apache, PHPmyadmin, Perl and various
Perl modules and their dependencies. These are installed under the /opt/zmanda/zrm and
/opt/zmanda/common directories.
To start or stop ZRM run:
# /etc/init.d/zmc_zrm_init start | stop | status

Types of Backup Available to ZRM
Depending on the datastore selected, ZRM offers a number of different types of backup.

Raw-Copy Backup
A raw-copy backup makes a copy of the binary disk image of databases stored on non-transactional
storage engines by using mysqlhotcopy. It works on live databases, but the whole database (or the
tables being backed up if only a few tables are selected) is locked for the duration of the backup.
Although raw backups can be restored more quickly than logical backups, they can only be restored to
the same version of MySQL server on the same platform architecture.
If any of the databases or tables are stored on a transactional storage engine (such as InnoDB), a logical
mysqldump backup is taken instead.

Logical Backup
Choosing this option forces a mysqldump backup that copies MySQL binary logs regardless of the
storage engine. MySQL binary logs track and save all database server transactions as a list of SQL
statements. To implement a logical backup strategy, binary logging must be enabled on the MySQL
server, and a path to the log files must be supplied (the default is /var/lib/mysql) in the Binary
Log Path field.
Logical backup method works with all MySQL storage engines except the MySQL cluster NDB
storage engine. Logical backups can also be restored to any platform architecture or database that
supports SQL.
Because logical backups require a read lock on the database(s) or tables being backed up, they can have
a greater impact on the applications using the MySQL database. Logical backups also result in
increased restore times, as restoring the data is accomplished by re-playing the transactions against the
target database instead of just copying files.


Raw-Quick (No-Copy) Backup
As its name suggests, this type of backup is the quickest on offer, but it is only available with raw
backups using snapshots (offered by ZFS and other filesystems). When the Quick (No-copy) snapshot
option is enabled, ZRM for MySQL uses the snapshot itself as the backup rather than transferring the
data into a standard backup archive on the ZRM server.
Quick snapshot backups are appropriate for large databases and for databases that have high transaction
rates. In addition to eliminating data transfer bottlenecks during backup, quick snapshot backups also
provide much faster database restoration than other backup methods.
However, because quick snapshot backups do not copy the data off of the MySQL server, they do not
protect data against server media failure. For this reason, quick snapshot backups can be converted at
any time into standard backups stored on the ZRM server by using the Convert Backup option on the
Reports menu tab.
Sun Microsystems' ZFS file system includes snapshot capability, which facilitates near-instantaneous
hot backups and rapid restores. ZRM makes use of this functionality by using an optional snapshot
plug-in that integrates ZRM for MySQL with ZFS. It creates temporary snapshots of the ZFS volumes
on which to perform a full backup. When snapshots are enabled, ZRM for MySQL can perform
backups with minimal impact on MySQL applications. Database writes will be blocked only during
snapshot creation, which typically takes less than a second regardless of database size.


Performance Testing Setup

System Setup
This section describes the system architecture and configuration implemented for the performance test.
The hardware used during this test was a Sun Fire X4500 server with two dual-core AMD Opteron 285
CPUs running at 2593 MHz with 16 Gbyte of RAM and 48 SATA drives of 512 Gbyte each.
The two boot drives are mirrored using ZFS volume management capabilities and the remaining 46
drives are available for backup data.
As a general rule, no specific OS tuning was done on the system, nor were the binaries of both MySQL
and ZRM optimized in any way. The objective of the tests was to deploy the solution out-of-the-box.
The only tuning performed was to limit the amount of memory available to the ZFS ARC cache to
4 Gbyte by adding the following line to /etc/system:
set zfs:zfs_arc_max = 4294967296

MySQL Software Release
The performance tests were carried out using MySQL 5.1 for Solaris 10 x64 (or x86_x64).
/opt/mysql/mysql/bin/mysql Ver 14.14 Distrib 5.1.30, for pc-solaris2.10 (x86_64) using
readline 5.1

OpenSolaris Operating System Release
OpenSolaris 2008.11 snv_101b_rc2 X86
Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 19 November 2008

Zpool Configuration
As seen previously, the storage available on the Sun Fire X4500 server was split into two pools:
● A backup pool made of 24 drives and two spares in a RAIDZ2 configuration. This provided
11 Tbyte of capacity for backups with a very high level of protection against drive failure and
performance sufficient to handle 300 Mbyte/sec to 350 Mbyte/sec of backup traffic over the
network.
● A mysql pool made of 12 drives and two spares in a RAIDZ2 configuration was used to store
the MySQL databases.


ZFS Configuration
Three ZFS file systems were created:
● zmanda/zrm to hold the ZRM backups
● zmanda/logs to hold the MySQL binlogs
● pool2/mysql to hold the MySQL data files

ZRM Software Release
The following software components were installed:
● The ZRM 3.0 for MySQL software
● Zmanda Management Console with MySQL, PHP, Apache, PHPmyadmin, Perl and various
Perl modules and their dependencies. These are installed under the /opt/zmanda/zrm and
/opt/zmanda/common directories.
ZRM was configured with:
● Level 0 backups
● No compression

db_STRESS Load Generator
db_STRESS is an open source tool (http://dimitrik.free.fr/db_STRESS.html) that can be used
to quickly generate large MySQL databases. Installing and configuring db_STRESS for our particular
needs involved the following steps.

1. Creating the test databases
- small_isam
- med_isam
- large_isam
- small_innodb
- med_innodb

2. Creating test tables in pool2/mysql
# /opt/mysql/mysql/bin/mysql -u root small_isam < BenchBASE_tab.sql
# /opt/mysql/mysql/bin/mysql -u root small_innodb < BenchBASE_tab.sql
# /opt/mysql/mysql/bin/mysql -u root large_isam < BenchBASE_tab.sql
# /opt/mysql/mysql/bin/mysql -u root med_innodb < BenchBASE_tab.sql
# /opt/mysql/mysql/bin/mysql -u root med_isam < BenchBASE_tab.sql


3. Generating the dataset
# /opt/db_STRESS-kit/bin/bench64 3821145 to create a dataset of roughly 10GB

4. Loading the dataset into the tables
# /opt/db_STRESS-kit/mysql/load_mysql.sh small_isam
# /opt/db_STRESS-kit/mysql/load_mysql.sh small_innodb
# /opt/db_STRESS-kit/mysql/load_mysql.sh med_isam
# /opt/db_STRESS-kit/mysql/load_mysql.sh med_innodb
# /opt/db_STRESS-kit/mysql/load_mysql.sh large_isam

5. Creating users
Creating a user called dim with password dimitri and giving this user access to all the databases:
mysql> GRANT ALL PRIVILEGES ON *.* TO 'dim'@'localhost' IDENTIFIED BY 'dimitri' with GRANT
OPTION;

6. Creating a test scenario
We created two different scenarios corresponding roughly to 15 percent CPU utilization of the system
by starting one WebX client and 90 percent CPU utilization by starting 30 WebX clients. (Note: WebX
is a software package bundled with the open source db_STRESS package.)
Scenario 1:
Starting one WebX client with the following command generated about 15 percent CPU utilization:

WebX x.db_STRESS -AntiDEAD 2000 -MAX 100000 -Base $dbname@localhost -User "dim" -Passwd
"dimitri" -ID 0 -Group Level20 -Timeout 1 -TotalTIME $duration -RW 0 -SHMKEY 555 -MaxID
2000

Scenario 2:
Starting 30 WebX clients, all with the following command generated about 90 percent CPU utilization:

WebX x.db_STRESS -AntiDEAD 2000 -MAX 100000 -Base $dbname@localhost -User "dim" -Passwd
"dimitri" -ID 0 -Group Level20 -Timeout 1 -TotalTIME $duration -RW 0 -SHMKEY 555 -MaxID
2000

Testing Methodology
Testing the performance of ZRM 3.0 for MySQL was done by conducting backups of several local
MySQL databases with tables of various sizes and storage engines, with the database either idle, 15
percent or 90 percent busy.
Each backup was performed as a Level 0 backup with no compression and three types of backups were
performed:
● Raw-quick backups using ZFS snapshots
● Raw-copy backups
● Logical backups


Each type of backup was conducted on the main two storage engines available to MySQL tables:
● MyISAM
● InnoDB
What was measured was the overall time taken by ZRM to complete the backup, as reported by ZRM
using the mysql-zrm-reporter command:

mysql-zrm-reporter --where backup-set=$dbname --output report.$dbname.$i --fields
backup-date,backup-set,backup-size,backup-time,read-locks-time,flush-logs-time,backup-
type,raw-databases,raw-databases-snapshot,logical-databases,backup-status

Each test was carried out three times in a row (except for the tests with the large database) and both the
slowest and fastest results were discarded.
The following actions were taken before each measured run:
● Delete all previous backups from /var/lib/mysql-zrm.
● Delete all previous ZFS snapshots created by ZRM.
● Stop all active MySQL clients (the db_STRESS clients).
● Stop the MySQL server.
● Unmount / mount the ZFS file systems, in order to remove any caching effect from ZFS from
consecutive runs.
● Start the MySQL server.
Note: Time did not allow us to conduct recovery tests.


Performance Test Results
A test result marked as" n/a "means that this particular test took too long to complete in a reasonable
time and was therefore not measured.

1. MyISAM Results with Idle Database
The first set of results were obtained by measuring the throughput while backing up idle databases
made of various MyISAM tables sizes.
Table 1: MyISAM Results with Idle Tables

Backup Type Table Engine Size (GB) Time (s) Throughput (MB/s)
raw-quick MyISAM 10.66 1 10,915.84
raw-copy MyISAM 10.66 68 160.53
logical MyISAM 11.44 575 20.37

raw-quick MyISAM 117.41 1 120,227.84
raw-copy MyISAM 117.41 759 158.40

raw-quick MyISAM 1065.84 1 1,091,420.16
raw-copy MyISAM 1065.84 6467 168.77
logical MyISAM 1065.84 n/a 0.00

The first observation that can be made here is that the raw-quick type of backup, which uses the ZFS
snapshot capabilities, can be not only very fast but independent of the size of the dataset.
The second observation is that a “logical” backup, also known as hot-backup, takes a lot longer to
complete than a “raw-copy” backup, but of course enables the backup to take place while the database
is still available to other users.

2. InnoDB Results with Idle Database
The second set of results were obtained by measuring the throughput while backing up idle databases
made of various InnoDB tables sizes.


Table 2: InnoDB Results with Idle Tables

raw-quick InnoDB 10.82 1 11,079.68
raw-copy InnoDB 10.82 91 121.75
logical InnoDB 8.08 411 20.13

raw-quick InnoDB 300.41 4 76,904.96
raw-copy InnoDB 300.41 2314 132.94
logical InnoDB 300.41 n/a n/a

Here as well, we can see that the “raw-quick” type of backup makes the most of the ZFS snapshot
capabilities.
We can also see that a “raw-copy” backup of an InnoDB table has a lower throughput compared to a
MyISAM table. This is because behind the scenes, ZRM uses mysqlhotcopy when dealing with
MyISAM tables and mysqldump for InnoDB tables. The “logical” backups however are consistent on
both types of storage engines.

Throughput with Idle Database
10,000,000

1,091,420
1,000,000

120,230
100,000 76,905
Throughput (MB/s)

small
10,920 11,080 medium
10,000
large

1,000

161 158 169
122 133
100

20 20 20

10
MyISAM quick MyISAM copy MyISAM logical InnoDB quick InnoDB copy InnoDB logical

Figure 2: Throughput Represented on a Logarithmic Scale: With Idle Database


3. MyISAM Results with a 15 Percent Load on the Database
The next set of results involved repeating the same measurement as previously, but this time while
db_STRESS was running MySQL queries on the databases generating a 15 percent load.

First we measured MyISAM results.

Table 3: MyISAM Results with a 15 Percent Load

raw-quick MyISAM 10.66 11 992.35


raw-quick MyISAM 1065.84 102 10,700.20
raw-copy MyISAM 1065.84 9622 113.43
logical MyISAM 1065.84 n/a n/a

“Raw-quick” backups are again much quicker than all other types of backup in this scenario, but
because we're backing up a live database under load, we can see the effect of the delay while we're
waiting to grab the lock on the tables before performing the snapshots.
The impact of the 15 percent load is quite noticeable on the backup throughput compared to the figures
for an idle database but it is still quite reasonable.


4. InnoDB Results with a 15 Percent Load on the Database
Then we move on to InnoDB results.
Table 4: InnoDB Results with a 15 Percent Load

raw-quick InnoDB 10.66 67 162.92

raw-copy InnoDB 300.41 2661 115.60

As mentioned above, “raw-quick” backups are again much quicker than all other types of backup in
this scenario, but because we're backing up a live database under load, we can see the effect of the
delay while we're waiting to grab the lock on the tables before performing the snapshots.
The impact of the 15 percent load is quite noticeable on the backup throughput compared to the figures
for an idle database but it is still quite reasonable.


Throughput with Database at 15% Load
100000

10700
10000 9048
Throughput (MB/s)

small
992
1000 medium
large
465
360

115 113 116
100 85
72

20 18
18 18

10
MyISAM quick MyISAM copy MyISAM logical InnoDB quick InnoDB copy InnoDB logical

Figure 3: Throughput Represented on a Logarithmic Scale: With Database at 15% Load

5. MyISAM Results with a 90 Percent Load on the Database
Finally, for the last set of results we used db_STRESS to heavily load the databases at 90 percent load
and repeated all the measurements.

First are the MyISAM results.

Table 5: MyISAM Results with a 90 Percent Load


raw-quick MyISAM 106.6 40 2,728.96



raw-quick MyISAM 1065.84 876 1,245.91
raw-copy MyISAM 1065.84 72202 15.12
logical MyISAM 1065.84 n/a n/a

Apart from the “raw-quick” backups, we can see here that the overall throughput is well below the
previous results, since the backup process is competing with the mysqld process for CPU time.
In almost all use-cases, it would be a much better strategy to off-line such a heavily loaded database
before taking a backup.

6. InnoDB Results with a 90 Percent Load on the Database
Then come the InnoDB results.
Table 6: InnoDB Results with a 90 Percent Load


raw-copy InnoDB 300.41 7722 39.84

The first raw-quick backup result on the 30.43 Gbyte dataset seemed oddly out of line compared to
other results collected.


Throughput with Database at 90% Load
10,000

2,729

1,246
1,000
531
Throughput (MB/s)

small
100 79 medium
53 large
40
35
23 26
15 15
10

3

1
MyISAM quick MyISAM copy MyISAM logical InnoDB quick InnoDB copy

Figure 4: Throughput Represented on a Logarithmic Scale: With Database at 90% Load


Summary
This guide is a quick introduction to configuring the Sun Fire X4500 server as a backup server for
Zmanda Recovery Manager 3.0. We have seen in this document how to deploy the Zmanda Recovery
Manager for MySQL on a system running OpenSolaris. We have successively learned how to:
● Create a ZFS storage pool using RAIDZ2
● Create a ZFS file system
● Install packages using the new OpenSolaris Image Packaging System
● Deploy ZRM in this environment
We have demonstrated the use of ZRM's ZFS snapshot integration in protecting two of the major
MySQL storage engines under varying workloads to help better understand the value and positioning of
this solution for Sun customers.
ZRM's usage of ZFS snapshot technology has shown its benefits to system and database administrators
as it can reduce the time needed to back up a database, independently of its size. Table 1 demonstrated
that the time needed to make a backup of a 1 TB MyISAM table could be reduced from 1h47mn to 1s.
In this test, we could take a backup of a 1 Tbyte database close to 100 percent utilization in just a few
minutes, or only a few seconds if the database were not heavily loaded. With ZRM's capability to
convert backups, nearline snapshots can be copied to standard backup stored on the ZRM backup
server without interfering with database utilization. (Please note: Results may vary.)

Product Information

OpenSolaris Operating System
OpenSolaris 2008.11 OS supports Dynamic Tracing (DTrace), ZFS, cryptographic infrastructures, IP
filter and User and Process Rights Management, and Predictive Self-Healing, which can further
enhance system reliability.

The ZFS File System
ZFS is a 128-bit general-purpose file system that spans the desktop to the data center, where space is
shared dynamically between multiple file systems from a single storage pool. Physical storage can be
added to or removed from storage pools dynamically, without interrupting services. This capability can
provide new levels of flexibility, availability, and performance.

The Sun Fire X4500 Server
The Sun Fire X4500 server has reached end-of-life and is no longer orderable. It is superceded by the
next-generation Sun Fire X4540 Server. The Sun Fire X4500 server can deliver the performance of a
four-way x64 server and provides 48 SATA drives with a total capacity of 48 Tbyte in 4U of rack
space. Its bigger brother, the Sun Fire X4540 server, is powered by two Quad-Core AMD Opteron™
CPUs operating at 2300 MHz and provides the same amount of storage capacity.


Zmanda Recovery Manager 3.0 for MySQL
Zmanda developed ZRM for MySQL with the explicit goal of providing simple, secure, and consistent
backups with minimal impact on applications and users. With ZRM for MySQL, DBAs can deploy
MySQL databases in mission-critical, high-transaction environments with confidence because they
know their data is protected by Zmanda. True to its open source mission, ZRM doesn't use any
proprietary data formats. Users can recover their MySQL data even without ZRM being installed.
ZRM eases day-to-day backup and recovery tasks with its web-based Zmanda Management Console
(ZMC). The management console integrates with the Zmanda Network to provide on-line
documentation and other support services for ZRM. Because it is open source and supports common
standards, as well as open data formats for backups, ZRM can free you from vendor lock-in.
Enabling advanced backup and recovery practices, such as raw, logical or hot-backups (Back up Live
MySQL databases without impacting applications and users), ZRM provides Continuous Data
Protection (CDP) for MySQL by combining file system-level snapshots along with data from MySQL
binary logs. File system snapshots help reduce database server downtime, especially when working
with large databases. ZRM uses a flexible framework of plug-ins for taking snapshots on a range of
operating platforms.
Zmanda offers three versions of ZRM for MySQL: the ZRM Enterprise edition, ZRM Cluster edition
and ZRM Community edition. The Enterprise edition includes a graphical installer, Zmanda
Management Console, command-line interface and plug-in framework.

About the Author
Caryl Takvorian is currently a Staff Engineer at Sun Microsystems. He joined Sun in 1998 in what is
now known as ISV Engineering where he assists ISVs to make the best use of Sun's technologies. His
technical skills are mainly in the Solaris OS and multithreaded applications and development tools.
Caryl also leads the Telecommunications segment for ISV Engineering.

For More Information
Zmanda Documentation
● Zmanda Recovery Manager for MySQL web site
http://www.zmanda.com/zrm-mysql-enterprise.html,
including ZRM datasheet (PDF): http://www.zmanda.com/zrm-datasheet.pdf
● Zmanda Recovery Manager for MySQL is available for download from the Zmanda network:
http://network.zmanda.com/
● Zmanda Network documentation for Zmanda Recovery Manager (requires registration):
http://network.zmanda.com/documentationEnterprise.php, includes:
● ZRM: Guide to MySQL Backup & Recovery and Backup and Recovery Benchmark for MySQL
with ZRM:
http://network.zmanda.com/zrm-whitepapers.php
● BigAdmin article: Sun Fire X4540 Server as Backup Server for Zmanda's Amanda Enterprise
2.6 Software: http://www.sun.com/bigadmin/features/articles/zmanda_sfx4540.jsp


● Backup and recovery tests for MyISAM and InnoDB engines with Zmanda Recovery
Manager for MySQL: http://solutions.mysql.com/solutions/white-
papers/partner_wp_zmanda_backup.php
● Sun Storage J4400 Array as Disk Storage for Zmanda Recovery Manager for MySQL
Database: http://www.sun.com/bigadmin/features/articles/zrm_mysql_j4400.jsp

Sun Fire Server Information
● Sun Fire servers: http://www.sun.com/servers/x64/x4540/
● Sun Fire X4500 server document collection:
http://docs.sun.com/app/docs/prod/sf.x4500#hic
● BigAdmin article: Using Sun Fire X4540 Server With Zmanda Recovery Manager 2.2 for
MySQL Database:
http://www.sun.com/bigadmin/features/articles/zrm_mysql_sfx4540.jsp

OpenSolaris Information
● OpenSolaris 2009.06 Image Packing System Guide (PDF):
http://opensolaris.org/os/project/pkg/files/ips.pdf
● OpenSolaris web site: http://opensolaris.org

ZFS Documentation
● Solaris ZFS Administration Guide (part of the Solaris 10 System Administration Collection):
http://docs.sun.com/app/docs/doc/819-5461
● Sun documents at http://docs.sun.com
● ZFS Best Practices Guide on the Solaris Internals web site:
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

MySQL Documentation
● MySQL site on sun.com: http://www.sun.com/software/products/mysql/index.jsp
● mysql.com web site: http://www.mysql.com/
● MySQL documentation site: http://dev.mysql.com/doc/
● MySQL training courses: http://www.sun.com/software/products/mysql/training.jsp
● MySQL download sites:
● Get MySQL: http://www.sun.com/software/products/mysql/getit.jsp
● MySQL Downloads: http://dev.mysql.com/downloads/


Discussions
● Zmanda forums: http://forums.zmanda.com/
● MySQL forums: http://forums.mysql.com/
● Sun Storage forums: http://forum.java.sun.com/category.jspa?categoryID=66
● Sun Hardware - Servers forums: http://forums.sun.com/forum.jspa?forumID=830

Other Sun Resources
● Sun training courses at http://www.sun.com/training/
● Sun BluePrints™ wiki (http://wikis.sun.com/display/BluePrints/Main, especially the
following:
● Storage BluePrints page:
http://wikis.sun.com/display/BluePrints/Storage+BluePrints
● Sun BluePrints article: Optimize MySQL Server on Sun x64 Servers and Storage:
http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x
64+Servers+and+Storage
● Sun Storage Administration wiki:
http://wikis.sun.com/display/StorageAdmin/Home
● BigAdmin Storage Tech Tips wiki:
http://wikis.sun.com/display/BigAdmin/Storage+Tech+Tips
● BigAdmin Storage Resource Collection (includes community content):
http://www.sun.com/bigadmin/collections/storage.html
● Sun Storage web site: http://www.sun.com/storagetek/index.jsp
● Storage Stop Blog: http://blogs.sun.com/storage

Support
● Sun resources:
● Register your Sun gear: https://inventory.sun.com/inventory/
● Services: http://www.sun.com/service/index.jsp
● SunSolveSM Online: http://sunsolve.sun.com
● Community system administration experts:
http://www.sun.com/bigadmin/content/communityexperts/

Events of interest to users of Sun products
● Sun and MySQL Around the World—Meetup Mashup:
http://www.sun.com/events/mysqltour/index.jsp
● Find a MySQL Meetup Group near you: http://mysql.meetup.com/


● Worldwide developer events and Sun Tech Days: http://developers.sun.com/events/
● Current events: http://www.sun.com/events/index.jsp

Licensing Information
Unless otherwise specified, the use of this software is authorized pursuant to the terms of the license
found at http://www.sun.com/bigadmin/common/berkeley_license.html.


Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on ...

More Related Content

What's hot (13)

Viewers also liked (8)

Similar to Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on ... (20)

More from webhostingguy (20)

Using ZFS Snapshots With Zmanda Recovery Manager for MySQL on ...