Installation of MySQL 5.1 Cluster Software on the Solaris 10 ...

Installation of MySQLTM 5.1
Cluster Software on the
SolarisTM 10 OS for x64
Platforms
Rainer Walter
Sun Microsystems
September 2008

Sun Microsystems, Inc.

Copyright © 2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights
reserved.

U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc.
standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license
terms. This distribution may include materials developed by third parties.

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX
is a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd.
X/Open is a registered trademark of X/Open Company, Ltd.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC
International, Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon
an architecture developed by Sun Microsystems, Inc.

AMD, Opteron, the AMD logo, the AMD Opteron logo are trademarks or registered trademarks of Advanced
Micro Devices.
Sun, Sun Microsystems, the Sun logo, MySQL, Solaris, Sun BluePrints, Sun Fire, and SunSolve are trademarks or
registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries.

This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import
laws in other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users,
whether direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to
entities identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially
designated nationals lists is strictly prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS,
REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE
EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Sun Microsystems, Inc.

Table of Contents
Summary...............................................................................................................................................................1

Introduction..........................................................................................................................................................2
The MySQL Software....................................................................................................................................2
MySQL Cluster Software...............................................................................................................................2
Node Types..................................................................................................................................................2
High Availability Concept..............................................................................................................................3
Partitioning.................................................................................................................................................3
Availability of the MySQL Cluster Software.....................................................................................................3

Scope and Limitations............................................................................................................................................4

Audience...............................................................................................................................................................4

Installation............................................................................................................................................................5
Prerequisites................................................................................................................................................5
Compilation of the Source............................................................................................................................5
Installation of the MySQL Server Software.....................................................................................................6

Configuration of a MySQL Cluster............................................................................................................................7
Overview.....................................................................................................................................................7
MySQL Cluster Configuration Files.................................................................................................................8
Starting and Stopping Components of a MySQL Cluster.................................................................................14
Cluster Messages and Error Log Files............................................................................................................16
User Accounts and File Ownership...............................................................................................................16
Storage Management in a MySQL Cluster.....................................................................................................16

Glossary...............................................................................................................................................................19

For More Information...........................................................................................................................................19

Acknowledgements..............................................................................................................................................21

Revision History...................................................................................................................................................21

1 Summary Sun Microsystems, Inc.

Summary
This document describes the installation of MySQLTM cluster software on three Sun FireTM X4100 M2 servers running
the SolarisTM 10 Operating System for x64 platforms. For two scenarios where data is loaded and manipulated, the
adjustment of the MySQL configuration necessary to execute these tasks is described. (Note: The installation was
performed with the Release Candidate Development Release of the MySQL 5.1 Community Edition.)

Introduction
The MySQL Software
MySQL software is the most popular relational database management system (RDBMS) software in the open source
space and part of the LAMP stack, which is a collection of software for dynamic web pages. Among the features that
differentiate MySQL software from other well-known RDBMS software are:
● Support for a broad variety of operating system platforms

● Choice of storage engines with different locking, indexing, and logging capabilities, enabling an optimal
tailoring of the database with regards to hardware resources and requirements of the application on top

MySQL Cluster Software
MySQL cluster software enables users to implement a MySQL database on commodity hardware using an approach for
fault tolerance, which ensures that data remains available even in the event of a failure of a database server or
storage component. The manner of clustering MySQL software described in this paper is accomplished on the
application level with no resources being shared among the cluster members at all, in contrast to cluster concepts
that operate on the operating system level, such as SunTM Cluster software, where the cluster members share one or
more storage resources. Implementing high availability for MySQL software using Sun Cluster software is described
elsewhere1.
While a detailed technical description of the design principles of MySQL cluster software can be found on the MySQL
web site2, an exhaustive discussion about the benefits and caveats of such a high availability solution is beyond the
scope of this paper. The following section provides a brief terminology of a MySQL cluster environment. It should be
noted that, while the availability and integrity of data in the MySQL database can be shielded against hardware
failures, existing client sessions, client connections, and active transactions are not protected by this MySQL
clustering concept and might suffer from such failures.

1 Sun Cluster Data Service for MySQL Guide for Solaris OS, August 2005, Part No: 819-1088-11 and
http://blogs.sun.com/luojiach/entry/installation_and_configuration_sun_cluster1
2 M. Ronström, L. Thalmann: MySQL Cluster Architecture Overview, a MySQL technical white paper, 2004;
New Features in MySQL Cluster 5.1, a MySQL technical white paper, August 2006.

Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms

2 Introduction Sun Microsystems, Inc.

Node Types
There are three different functions in a MySQL cluster, which are associated with respective daemons:
● Data (storage) node with daemon ndbd

● SQL node with daemon mysqld

● Management node with daemon ndb_mgmd
A server can host more than one function of a MySQL cluster.

High Availability Concept
High availability is achieved on the application level by providing redundant services and storage resources on
independent servers in a shared-nothing architecture. Network Database (NDB) is the storage engine required for this
architecture. The data portion of the database that one data (storage) node holds is called the cluster partition.
Cluster partitions are managed as replicas. A primary replica is the data portion, which is assigned to a data node in
the cluster; a backup replica is a copy of a partition that is assigned to another node.
Data manipulation statements (update , insert, delete ) and data definition statements (for example,
create table, create tablespace , and so on) are performed through a two-phase commit protocol,
ensuring that the database, being spread across multiple data nodes, remains consistent and that the content of
backup replicas is identical. The distribution of data in one replica across different storage nodes is accomplished by
horizontal partitioning.

Partitioning
Data in the database is evenly split into as many partitions as data nodes exist. Copies of partitions (replicas) can be
kept on data nodes other than the node that holds the original partition for redundancy. While the number of
partitions is always equal to the number of storage nodes, the number of replicas can be one or two, where a setup
with two replicas provides redundancy and is, therefore, recommended.
The number of replicas controls how many node groups are formed in a MySQL cluster. A node group is a subset of
data nodes, which holds replicas of the same partition; if all data nodes of a MySQL cluster that form one node group
are missing, one partition is not available and so the database cannot be brought online.

Availability of the MySQL Cluster Software
Until MySQL version 5.1 software, MySQL cluster software came in two flavors:
● As part of the "ordinary" MySQL software distribution3.

● As MySQL Cluster Carrier Grade Edition, a special version for network service providers and telecom
equipment manufacturers4, which offers some extra features not available in other distributions. The
versioning of this edition is different from the "ordinary" software distribution.

3 For MySQL software version 5.1: http://dev.mysql.com/downloads/mysql/5.1.html
4 More details on http://www.mysql.com/why-mysql/communications/
MySQL Cluster Carrier Grade Edition download: http://dev.mysql.com/downloads/cluster/index.html


3 Introduction Sun Microsystems, Inc.

The installation of the MySQL software was performed with the Release Candidate Development Release version 5.1.24
of the MySQL Community Edition software distribution. While cluster support is included in the package of MySQL
version 5.1.24 for the Solaris OS for SPARC® platforms, the official MySQL download site did not offer an analogous
package with cluster support for the Solaris OS for x64 platforms as of the writing this paper. Instead, the source of
MySQL version 5.1.24 was downloaded and was compiled prior to installation.
Planned future distributions of MySQL 5.1 binaries would no longer contain the MySQL cluster functionality; instead,
the official successor version of MySQL Cluster 5.0 software would be MySQL Cluster Carrier Grade Edition 6.2.

Scope and Limitations
This paper describes the installation of the MySQL software and the steps that are necessary to set up a shared-
nothing MySQL cluster in a minimal setup of three cluster nodes, which does not provide universal fault tolerance and
is not, therefore, recommended for production purposes.
The settings that need to be specified for loading and manipulating data in two different scenarios are discussed. An
investigation about performance optimization and about security aspects of a MySQL cluster environment is beyond
the scope of this paper.
The default paths to executables and configuration files might differ between MySQL installations performed from
binaries that are in Solaris package format and binaries built from source. For instance, the binaries of MySQL
daemons, such as ndbd , mysqld , ndb_mgmd , are located in the /bin folder of the MySQL installation directory
built from a Solaris package for SPARC platforms. However, installation on the Solaris OS for x64 platforms, which is
based on compiled binaries from source, keeps these daemon executables in the /libexec subfolder of the
installation directory, separate from other executables in /bin.

Audience
Readers of this paper should be familiar with concepts of relational databases and have basic knowledge of the SQL
programming language.


4 Installation Sun Microsystems, Inc.

Installation
The installation of MySQL 5.1.24 software that is described herein was performed on three Sun Fire X4100 M2 servers
with the following specifications:

● 2x dual-core AMD OpteronTM CPUs at 2.4 GHz

● 4 Gbytes of RAM

● Storage: Two 68-Gbyte internal hard disks

● Operating System: Solaris 10 5/08 OS for x64 platforms (s10x_u5wos09 X86)
The installation of the MySQL software from the source comprises three steps:
● Run the configure script in the MySQL source directory to produce makefile scripts.

● Use gmake to produce the appropriate binaries.

● Run gmake install to set up the directory structure and to copy binaries and configuration files.

Prerequisites
The MySQL source can be obtained in compressed tar format from the MySQL web site3.
For an installation of MySQL software from binaries that have been compiled from source, it is recommended that a
user account with the user name mysql exists and a group names mysql exists, which the mysql user is member
of. There is no need to set up a home directory for this user, nor to define a password.
For the compilation of the MySQL 5.1.24 source on the Solaris 10 OS for x64 platforms, the executables gcc (version
3.4.3) and gmake (version 3.80) in /usr/sfw/bin are used. Additionally, the ar executable in
/usr/ccs/bin is needed.
The PATH variable must, therefore, be adjusted:
PATH=$PATH:/usr/sfw/bin:/usr/ccs/bin

If 64-bit binaries are compiled, a symbolic link is needed in the directory /usr/sfw/lib/amd64 :
ln -s libgcc_s.so.1 libgcc_s.so



Compilation of the Source
The MySQL source can be used to produce both 32-bit and 64-bit binaries. The procedure for both differs in the
environment variables that are supplied when executing the configure script, while two options need to be
provided in both cases when calling the configure script:

● The option --prefix specifies the installation directory of the MySQL binaries, which has been set to
/opt/mysql . If omitted, the binaries are installed in /usr/local/mysql .
● The option --with-ndbcluster instructs the configuration process to include the binaries of the NDB
engine, which is needed for storage nodes and the management node (or nodes) of a MySQL cluster, and
which are not included by default.
Building a 32-bit version of MySQL software: After extracting the tar archive of the MySQL source, the script
configure is called with the following parameters from the directory of the extracted source files (mysql-
5.1.24-rc):
CC=gcc CXX=gcc ./configure --prefix=/opt/mysql --with-ndbcluster

Building a 64-bit version of MySQL software: After extracting the tar archive of the MySQL source, the script
configure is called with the following parameters from the directory of the extracted source files (mysql-
5.1.24-rc):
CC='gcc -m64' CXX='g++ -m64' LDFLAGS="-R/usr/lib/64 -R/lib/64 -R
/usr/sfw/lib/64 -L/usr/sfw/lib/64 -L/usr/lib/64 -L/lib/64 -lrt"
./configure --prefix=/opt/mysql --with-ndbcluster

The binaries are created by the following command, which is executed in the directory where the source files are
located:
gmake

Installation of the MySQL Server Software
The installation of the freshly compiled binaries is launched from the folder of the MySQL source files by issuing the
following command:
gmake install
After the installation of the MySQL software, a script must be run, which initializes the grant tables of the MySQL
server. This script is called mysql_install_db and it is located in the /bin folder of the MySQL installation
directory (/opt/mysql/bin , in this case).

This script creates a directory structure for two databases, mysql and test . Since these database directories must
be readable and writable by the MySQL server and, therefore, must be owned by the user mysql, the command
mysql_install_db , when executed in the context of the root user, should be run with the option --
user=mysql , which ensures that the necessary privileges and ownership flags are set.



By default, the database directories are set up in the MySQL installation directory under the subfolder data. If these
database files and directories need to located elsewhere, two alternatives can be followed:
● Run the mysql_install_db command with the option --datadir=<path for parent
directory of MySQL database files>
● Add an entry for datadir in the configuration file of the MySQL server, my.cnf , before running
mysql_install_db
The initial size of the folder containing the data files of the mysql database is 1.5 Mbytes. The size of the test
database is initially 1 Kbyte.

Configuration of a MySQL Cluster
Overview
A MySQL cluster environment comprises three different functions. Each function is characterized by specific sections in
the my.cnf file.

● A MySQL cluster management node provides the configuration data of the cluster to storage nodes. It stops
single storage nodes or shuts down the entire cluster. The MySQL cluster management node is based on the
ndb_mgmd process, which reads the cluster configuration from the config.ini file. Cluster
management is performed using the management console
<MySQL_Install_Directory>/bin/ndb_mgm .
● A MySQL storage (data) node provides the storage capacity for data in a MySQL cluster database. It uses the
NDB storage engine, which is based on the ndbd process. The ndbd process of a storage node obtains its
configuration upon startup from the management node.
● A MySQL SQL node provides access to the data in a database of a MySQL cluster. It is based on a running
mysqld process. This daemon accepts connections from client programs, such as MySQL monitor,
<MySQL_Installation_Directory>/bin/mysql , which provides an interface for executing
SQL code.
The environment used for experiments described here is depicted in Figure 1.


7 Configuration of a MySQL Cluster Sun Microsystems, Inc.

Figure 1. MySQL Cluster Topology

MySQL Cluster Configuration Files
MySQL Server Configuration File: my.cnf
The MySQL server configuration file must exist on each member of a MySQL cluster and defines the roles in the cluster
that each member has. Multiple my.cnf files can exist on one system, which are processed in a hierarchical
manner:
● A my.cnf file with global settings must be located in /etc or in /etc/mysql 5.

● Server-specific options can be supplied in a my.cnf that resides in the installation path of the MySQL
software or in its data directory.
● User-specific options are interpreted from a my.cnf file in the user's home directory.

In the setup described in this guide, one single my.cnf file in /etc/mysql was used.

my.cnf for the Management Node
The following entries were made in a my.cnf file that defines the management node of a MySQL cluster. While the
[mysqld] section exists in the template, the sections [ndb_mgm] and [ndb_mgmd] need to be added, which
both define the role as cluster management node of this server in the cluster.
[mysqld]

user = mysql
ndbcluster

5 MySQL 5.1 Reference Manual, revision: 10113, 2008-03-04. See also http://dev.mysql.com/doc/refman/5.1/en/ for
the most current, online version.



ndb-connectstring = 10.9.166.30
datadir = /export/mysql/MySQL_Data

[ndb_mgm]

connect-string = 10.9.166.30

[ndb_mgmd]

config-file = /etc/mysql/config.ini

The entries in [mysqld] are:

● user = mysql – This setting lets the mysqld daemon run in the context of the user mysql.
● ndbcluster – This server can act as an SQL node in a MySQL cluster. DDL statements can use the NDB
storage engine. Although the binaries of the installation are compiled with the ndbcluster option, the
NDB storage engine is not automatically available when the MySQL server is started. The NDB storage engine
is activated by this entry in the my.cnf , or by specifying the --ndbcluster argument in the start
command of the MySQL server. To verify that the NDB storage engine is present and usable, the command
show engines can be executed using MySQL monitor, as shown in Figure 2.

Figure 2. Output of a show engines Query

● ndb-connectstring – This SQL node tries to contact the management node running on a system with
the IP address specified. A connection to such a management node is established if (1) this SQL node is listed
and specified in the management server's config.ini or (2) the management server has "blank" entries
for unspecified SQL nodes. No user authentication is required.
● datadir – This is the path to the files and directories of a MySQL server that the script
mysql_install_db creates.
The entry connect-string in the section [ndb_mgm] specifies the IP address of the server hosting the NDB
management daemon ndb_mgmd that the cluster management console is supposed to talk to.

The section [ndb_mgmd] indicates that the host acts as cluster management node. The value for config-file
points to the location of the MySQL cluster configuration file config.ini .

my.cnf for the Storage Nodes
The section [mysql_cluster] signals that the host acts as storage node in the cluster. The
ndb-connectstring value points to the management node from which a cluster storage node shall obtain its
cluster-specific configuration.



[mysqld]

user = mysql
ndbcluster
datadir = /export/mysql/MySQL_Data

[mysql_cluster]


MySQL Cluster Configuration File: config.ini
The topology and the configuration of a MySQL cluster is defined and maintained in one single file on the
management node: config.ini . The location of this MySQL cluster configuration file is specified in the my.cnf
of the cluster management node.
The cluster storage nodes receive their individual configuration from their management node upon startup. Any
modification, therefore, requires a restart of the affected nodes. In special cases, for example, when settings of REDO
logs shall be changed, the affected data node must be restarted with the option --initial , which erases the
contents of the ndb_<NodeID>_fs folder, so that all data of databases using the NDB engine on this node are
lost unless an active replica node is available. Therefore, care should be taken when issuing the command:
<MySQL_Installation_Directory>/libexec/ndbd --initial

A template for a two-data-node cluster config.ini file (ndb-config-2-node.ini in
<MySQL_Install_Directory>/share/mysql ), which comes with the installation of the MySQL
software, was used to produce the operative version for the tests described herein. The config.ini used for the
test environment covers five aspects:
● General properties of the cluster

● Management node

● Storage nodes

● SQL nodes

● TCP/IP settings

General properties of the cluster are listed in the [ndbd default] section. This section specifies:

● How many replicas shall be produced and maintained: NoOfReplicas . Possible values are 1 (no
redundancy) and 2.
● How much physical memory shall be assigned to MySQL: DataMemory , IndexMemory . The values for
these parameters depend on the total amount of RAM available on the host.
● The location of cluster logs and other files of the cluster: DataDir . Note that my.cnf uses a parameter
with the same name for specifying the location for files and directories that a mysqld daemon relies on. In
the setup used for tests described in this paper, different locations were supplied for the DataDir of
mysqld (in my.cnf) and ndbd (in config.ini ).



Settings for storage nodes can be defined globally, that is, for all storage nodes, in the [ndbd default] section,
or individually for a single storage node in separate [ndbd] sections. For each storage node, an extra [ndbd]
section must exist, which at least contains a HostName entry.

A SQL node is specified with its host name in a separate [mysqld] section. If a [mysqld] section does not
contain a HostName entry, any MySQL server can connect to the cluster if its my.cnf file contains a ndb-
connectstring entry with the valid IP address of the cluster management node. It is recommended to define
one [mysqld] object in config.ini that does not have a specified host entry, which is helpful, for example,
for ndb_restore operations.

This is the initial version of config.ini :
[ndbd default]
NoOfReplicas= 2
MaxNoOfConcurrentOperations= 10000
DataMemory= 2048M
IndexMemory= 512M
TimeBetweenWatchDogCheck= 30000
DataDir= /export/mysql/mysql-cluster
MaxNoOfOrderedIndexes= 512

[ndb_mgmd default]
DataDir= /export/mysql/mysql-cluster

[ndb_mgmd]
Id=1
HostName= 10.9.166.30

[ndbd]
Id= 2
HostName= 10.9.166.28

[ndbd]
Id= 3
HostName= 10.9.166.29

[mysqld]
#Id= 4
HostName= 10.9.166.28

[mysqld]
#Id= 5
HostName= 10.9.166.29

[mysqld]
HostName= 10.9.166.30

[mysqld]
# One empty entry recommended

[tcp default]
PortNumber= 63132

Depending on the database load characteristics, default settings must be changed by modifying existing entries or
adding new entries in the appropriate sections to override default values. In the following, some workloads have been
executed on the MySQL cluster, and the adjustments needed in the config.ini file for a smooth operation are
described. All tests were run with tables using disk-based storage. No attempts were made to optimize performance
during these tests.



Scenario 1: Table With 3 Columns, Simple Records
The table used for the tests with insert, update, and index creation is described in Figure 3.

Figure 3. Schema of the Test Table

Test #1: Inserting 10 Million Records
An SQL procedure with a loop was used to create 10 million records.
Result: No modifications in config.ini are necessary.

Test #2: Update 10 Million Records
An SQL statement updates the Date field of all records.
This attempt produces errors of the following kind:
● Got temporary error 233 'Out of operation records in transaction
coordinator' from NDBCLUSTER
● Got temporary error 1217 'Out of operation records in local data
manager' from NDBCLUSTER
Result: Two parameters for the configuration of data nodes have default values that are not sufficient to run an
update statement for millions of records:
● MaxNoOfConcurrentOperations (Default: 32768). This is the number of records that is updated
simultaneously in a transaction.
● MaxNoOfLocalOperations (Default: 1.1 * MaxNoOfConcurrentOperations )
By default, MaxNoOfLocalOperations is a function of MaxNoOfConcurrentOperations , so only
the latter needs to be changed in config.ini .

When the value for MaxNoOfConcurrentOperations was increased to 10,000,000 (the number of records
that shall be updated in one transaction), the storage nodes could not be started due to a lack of memory (error 2327).
This situation might depend on the amount of RAM assigned to the MySQL software and the capacity in RAM allocated
by table data. The update statement must, therefore, have a where clause that the number of records to be updated
keeps within the limit of a tolerable value for MaxNoOfConcurrentOperations .
In the test environment, updating 500,000 rows in a single transaction was possible using the following setting:
● MaxNoOfConcurrentOperations = 1000000



Test #3: Inserting 2x 10 Million Records Into a Table From Two Client Connections in Parallel
Result: One parameter in section [ndbd default] needs to be increased to avoid error 1205 (Lock wait
timeout exceeded ):
● TransactionDeadlockDetectionTimeout (default: 1200 ms).
The value for TransactionDeadlockDetectionTimeout was increased to 10000, and the parallel
execution of the insert statement went smoothly.

Test #4: Inserting 3x 10 Million Records Into a Table From Three Client Connections in Parallel
Result: No changes need to be made to config.ini to have the three insert statements complete successfully.

Test #5: Table With 10 Million Rows: Creating an Index
An additional index was created on the Firstname field using an ALTER TABLE statement.

Result: No modification in config.ini is necessary to run the statement.

Scenario 2: Table With BLOB Field
The table used for tests with BLOB data in the MySQL software has the schema shown in Figure 4.

Figure 4. Schema of the Table With BLOB Field

Test #1: Insert 5000 Records With a Binary File
Five thousand individual binary files6 with file sizes ranging from 512 Kbytes to 3012 Kbytes were inserted using the
LOAD_FILE option in the INSERT INTO clause using a script with individual INSERT INTO statements.
Result: A data node was forced to shut down because the default value for SendBufferMemory was too low. The
largest file that could be inserted was 1624 Kbytes in size.

6 Binary files created with the command dd if=/dev/urandom of=<Path_of_File> bs=512 count=<N>



The following parameter was added in the [tcp default] section of the cluster configuration file:

● SendBufferMenory = 10M
Error 1220 was thrown while loading data into the table: Got temporary error 1220 'REDO log
files overloaded, consult online manual (increase FragmentLogFileSize)'
from NDBCLUSTER .
The following parameter was added to the [ndbd default] section of config.ini :

● FragmentLogFileSize (default: 16 Mbytes)
The value for FragmentLogFileSize was set to 64 Mbytes.

Error 1221 was thrown while loading data into the table: Temporary error: 1221 'REDO buffers
overloaded, consult online manual (increase RedoBuffer)' .
The following parameter was added to the [ndbd default] section of config.ini :

● RedoBuffer (default: 8 Mbytes)
The value for RedoBuffer was increased to 64 Mbytes.

Starting and Stopping Components of a MySQL Cluster
It is mandatory to start the cluster management daemon (ndb_mgmd ) first because the storage nodes get their
cluster configuration from the config.ini file on the management node. This daemon is launched by executing
the following command:
<MySQL_Installation_Directory>/libexec/ndb_mgmd

The management console is brought up with the following command:
<MySQL_Installation_Directory>/bin/ndb_mgm

The initial state of the cluster environment described herein can be displayed by the command show, as shown in
Figure 5.



Figure 5. MySQL Cluster Management Console

The cluster consists of two data nodes [ndbd(NDB)] and three SQL nodes [mysqld(API)] .
All data nodes and two SQL nodes are bound to specified IP addresses (that is, connections from other MySQL servers,
even when having suitable entries in their my.cnf , will be rejected).

One SQL node connection is defined in the config.ini file without an IP address, so that any host with a
mysqld process running and with an appropriate ndb-connectstring in its my.cnf can establish a
connection with the management daemon of this MySQL cluster and act as an SQL node.
When the cluster management daemon is running, the storage node daemon (ndbd) can be started. If the ndbd
process cannot contact the management node specified in the ndb-connectstring of the
[mysql-cluster] section of its my.cnf, an error message occurs: Unable to connect with
connect string: nodeid=0, <IP address>:1186. Retrying every 5 seconds .
The execution of <MySQL_Installation_Directory>/libexec/ndbd launches two processes called
ndbd: The first ndbd process (angel process) watches its child ndbd process and restarts it when it detects that the
child process is no longer active.
After starting the ndbd daemon on the storage nodes, the show command in the cluster management console
displays the status shown in Figure 6.



Figure 6. MySQL Cluster Running

The MySQL cluster management console is the preferred tool for stopping single data nodes or for shutting down the
whole cluster. A single data node can be stopped by issuing the command <NodeID> stop ; the whole cluster is
stopped with the command shutdown , as shown in Figure 7.

Figure 7. Shutting Down MySQL Cluster

The MySQL cluster can be run independently from any MySQL server process. Such a process is needed only to access
databases that use the NDB storage engine and retrieve or modify records and database objects.
It is recommended to start an SQL node using the following script:
<MySQL_Installation_Directory>/bin/mysqld_safe &

It is also recommended that the mysqld process runs in the context of a user other than root, for example, in the
context of the user mysql (see the section about file ownership on page 16). If the my.cnf file does not contain an
entry like user=mysql , the command shown above can be executed with the option --user=mysql .



Alternatively, the shell script mysql.server in
<MySQL_Installation_Directory>/share/mysql can be copied to /etc/init.d so that the
MySQL server process is automatically started when the server is restarted.
On the server running the Solaris 10 OS for x64 platforms, the following adjustments were made in order to make the
MySQL server start script executable:
● Replaced the first line by the expression #!/bin/bash

● Specified the MySQL installation directory in the variable basedir

● Specified the path to the database files created by mysql_install_db in the variable datadir

● Specified the path of the my.cnf file in the variable conf

The mysqld process on an SQL node can be stopped using the following command:
<MySQL_Installation_Directory>/bin/mysqladmin -u root shutdown

Since the storage node process requires that the cluster management node daemon is up and running, it does not
appear reasonable to provide an analogous start script for the ndbd daemon.

Cluster Messages and Error Log Files
The error log files of the MySQL cluster are located in the directory specified in the DataDir parameters for data
and management nodes of the config.ini file.

The management node maintains the ndb_1_cluster.log , while each data node has an
ndb_<NodeID>_error.log and trace files, if errors have occurred. The value for NodeID refers to the
numbering provided in the management console view (see Figure 6). The cluster log and the error logs are rolled over
when their size reaches a threshold value.

User Accounts and File Ownership
The context of the user, under which a mysqld or ndbd daemon is started, is critical.

SQL node: After the installation of the MySQL software, it is recommended to run the script mysql_install_db
as user mysql and not as user root . This user becomes the owner of the database directories and files, which are
created by the script. If the MySQL server is started with a different user ID, the MySQL server cannot access databases
and tables and produces errors upon startup.
Storage node: All content in the Datadir of the ndbd daemon are owned by the user that launched the ndbd
daemon. There is no recommendation regarding a preferred user for running ndbd as long as the same user is always
starting this daemon. In the tests with the MySQL cluster described herein, the ndbd process was launched as user
root.



Storage Management in a MySQL Cluster
Prior to MySQL software version 5.1, all database data using the NDB storage engine was kept in the physical memory
of the storage nodes assigned to the ndbd daemon. With version 5.1, it is possible to store table content in columns
that are not indexed in disk-based tablespaces, while the indexed columns of a table reside in memory. The keyword
STORAGE DISK in a CREATE TABLE statement in conjunction with the ENGINE=NDB clause indicates
that columns without index shall allocate pages on disk rather than in memory.
The amount of RAM that is at disposal for a MySQL cluster is specified by the parameters DataMemory and
IndexMemory in the [ndbd] section of the config.ini . The utilization of physical memory by data and
indexes can be determined by executing the command all dump 1000 on the MySQL cluster management
console7 (see Figure 8).

Figure 8. MySQL Cluster Management: All Dump 1000

The output of an all dump 1000 command is written to the MySQL cluster log, and shows the utilization of the
assigned memory for data and indexes as a percentage and as the number of pages, respectively, and might look like
the following:
2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Data usage is
42(27885 32K pages of total 65536)
2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Index usage is
47(30959 8K pages of total 65568)
2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 0 min: 0
max: 639 curr: 269
max: 0 curr: 259
max: 0 curr: 10
2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Data usage is
42(27883 32K pages of total 65536)
2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Index usage is
47(30958 8K pages of total 65568)
max: 639 curr: 269
max: 0 curr: 259
max: 0 curr: 10

Information about the utilization of disk space allocated in data files of tablespaces and undo files of log file groups
can be retrieved from the table INFORMATION_SCHEMA.Files (see Figure 9).

7 In planned future versions, the command ALL REPORT MEMORYUSAGE would do the same.



Figure 9. Data Files and Undo Files: Utilization of Disk-Based Storage

Tablespaces of the NDB engine do not have an autoextend feature. When creating a data file of a tablespace using the
CREATE TABLESPACE or the ALTER TABLESPACE statement, either a file name or the full path of
the data file must be supplied. If only a file name is supplied, the file is created in the default path
<DataDir>/ndb_<NodeID_fs> .
The same applies to undo files that are created for a log file group. If undo files shall be physically separated from data
files to minimize concurrent I/O during insert and update activities, the full path of the undo/data file must be
supplied in the CREATE LOGFILE GROUP and CREATE TABLESPACE statements, respectively.
In a MySQL 5.1 cluster database with tables using disk-based storage, three types of logging can occur:
1. REDO logging. The data nodes of a MySQL cluster use, in the default configuration, 16 sets of 4x16 Mbyte files for
circular logging, that is, old log entries can be overwritten if three local checkpoints have been completed. The default
size of 1 Gbyte in total for REDO logs can be adjusted by supplying custom [ndbd] parameters. With MySQL REDO
logs, there is no danger of a continuous growth of disk space consumption.
2. UNDO logging. If data nodes of a MySQL cluster use disk data objects, that is, log file groups and tablespaces in
conjunction with a STORAGE DISK clause, disk space is allocated in undo files. Because these undo files cannot be
automatically extended, there is no immediate risk of filling up the file system over time.
3. Binary logging. The setting for log-bin in the my.cnf file ensures that all statements that manipulate data
and database objects are logged. These log files are written to the Datadir target of the mysqld daemon on SQL
nodes of a MySQL cluster. This binary log is needed for recovery after restoring the database from a backup and for
replication. In contrast to the logging mechanisms mentioned before, if not maintained, these binary logs grow,
accumulate, and might consume a significant amount of disk space on SQL nodes.


19 Glossary Sun Microsystems, Inc.

Glossary
The following table contains common abbreviations and acronyms that are used in this paper.

Abbreviation Explanation
BLOB Binary Large Object
DDL Data Definition Language
LAMP Linux, Apache, MySQL, Perl|Python|PHP
NDB Network Database
RDBMS Relational Database Management System
SQL Structured Query Language

For More Information
Here are additional resources:
● Documentation:

● MySQL documentation site: http://dev.mysql.com/doc/

● MySQL Cluster 5.1 - New Features white paper:
http://www.mysql.com/why-mysql/white-papers/mysql_wp_cluster51.php

● MySQL Cluster Architecture Overview white paper:
http://www.mysql.com/why-mysql/white-papers/cluster-technical.php

● MySQL training courses: http://www.sun.com/software/products/mysql/training.jsp

● MySQL download sites:

● http://www.sun.com/software/products/mysql/getit.jsp
● http://dev.mysql.com/downloads/mysql/5.1.html

● MySQL forums: http://forums.mysql.com/

● Other web sites and resources:
● MySQL site on sun.com: http://www.sun.com/software/products/mysql/index.jsp
● mysql.com site: http://www.mysql.com/
● Features & Benefits - MySQL Cluster:
http://www.sun.com/software/products/mysql/cluster.jsp
● MySQL and MySQL Cluster FAQs: http://www.sun.com/software/products/mysql/faqs.jsp
● Sun BluePrintsTM article: Optimize MySQL Server on Sun x64 Servers and Storage
http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x64+Serv
ers+and+Storage


20 For More Information Sun Microsystems, Inc.

● Resources on BigAdmin:
● Setting Up MySQL Cluster Software Using Solaris Zones Partitioning Technology:
http://www.sun.com/bigadmin/features/articles/mysql_cluster_zones.jsp
● MySQL Resources for System Administrators:
http://www.sun.com/bigadmin/topics/mysql/
● Creating a Script for Scheduled MySQL Backups (community submission):
http://www.sun.com/bigadmin/content/submitted/sched_mysql_backup.jsp
● Database Resource Collection:
http://www.sun.com/bigadmin/collections/database.html
● Resources on Sun Developer Network (SDN):
● Using MySQL and Memcached on the GlassFish Application Server:
http://developers.sun.com/appserver/reference/techart/mysql_gf/

● Sun and MySQL: How It Stacks Up for Developers:
http://java.sun.com/developer/technicalArticles/mysql_acq/index.html

● Support:

● Register your Sun gear: https://inventory.sun.com/inventory/

● Services: http://www.sun.com/service/

● SunSolveSM Online: http://sunsolve.sun.com

● Events of interest to users of Sun products:
● Sun and MySQL Around the World--Meetup Mashup:
http://www.sun.com/events/mysqltour/index.jsp

● Find a MySQL Meetup Group near you: http://mysql.meetup.com/
● Worldwide developer events and Sun Tech Days: http://developers.sun.com/events/
● Current events: http://www.sun.com/events/index.jsp


21 Acknowledgements Sun Microsystems, Inc.

Acknowledgements
We want to thank the following colleagues for their kind help, assistance, and support during the work on this paper:
Ryan Arneson, Sun Microsystems in Broomfield, CO, for preparing the Sun Fire X4100 M2 servers, and for helping
immediately when a hardware piece caused trouble.
Matthias Schmidt, Sun Microsystems in Walldorf, for his help in preparing the shell scripts for data generation.
Jürgen Zimmermann, Sun Microsystems in Walldorf, for his support during the attempts to compile the MySQL source
code.
Ritu Kamboj, Sun Microsystems in Menlo Park, for helping with the parameters for the 64-bit compilation.
Hartmut Holzgraefe, Sun Microsystems, for reviewing the paper and for his valuable comments.

Revision History
Revision Notes Date
0.1 Draft for Revision 29.05.2008
1.0 First Version 16.06.2008

Licensing Information
Unless otherwise specified, the use of this software is authorized pursuant to the terms of the license found at
http://www.sun.com/bigadmin/common/berkeley_license.html.


Installation of MySQL 5.1 Cluster Software on the Solaris 10 ...

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to Installation of MySQL 5.1 Cluster Software on the Solaris 10 ... (20)

More from webhostingguy (20)

Installation of MySQL 5.1 Cluster Software on the Solaris 10 ...