My sql fabric webinar tw2

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
MySQL Fabric:
Easy Management of MySQL Servers
杜修文 Ivan.Tu@Oracle.Com
MySQL Principal Sales Consultant
Oracle LLC
July 17, 2014

2
Insert Picture Here
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |
An extensible and
easy-to-use framework
for managing a farm of
MySQL server supporting
high-availability and
sharding
MySQL Fabric

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |3
MySQL Fabric: Goals & Features
 Connector API Extensions
 Support Transactions
 Support full SQL
 Decision logic in connector
 Reducing network load
 Load Balancing
 Read-Write Split
 Distribute transactions
 Global Updates
 Global tables
 Schema updates
 Shard Multiple Tables
 Using same key
 Sharding Functions
 Range
 (Consistent) Hash
 Shard Operations
 Using built-in executor
 Shard move
 Shard split

Transaction Handling

 Routing Transactions
 Pre-declare properties of transactions
 Detecting transaction boundaries
 Push as much as possible to server
 Managing Session State
 Move session state between servers
− Easy to use
− Expensive and error prone
 Reset state after each transaction
− Transactions start with default session state
Transaction Handling

Routing Transactions
Shard #2
Shard #1
Shard #3
Switch
State
Store Executor
QUERY
KEY
KEY
QUERY
Contain decision logic
for routing queries
Contain information
about location of shards
SHARD#

Birds-eye View
High Availability Groups (Shards)
MySQL Fabric
Node
Application
XML-RPC
SQL
Connector
Connector
Connector
Operator

MySQL Fabric: Prerequisites
 MySQL Servers (version 5.6.10 or later)
 Server for meta-data backing store
 Servers being managed
 Python 2.6 or 2.7
 No support for 3.x yet
 MySQL Utilities 1.4.0
 Available at http://labs.mysql.com/

MySQL Fabric: Configuration
 Backing Store
 MySQL server
 Persistent storage for state
 Storage engine-agnostic
 Protocol
 Address where node will be
 Currently only XML-RPC
 Logging
 Chatty: INFO (default)
 Moderate: WARNING
 URL for rotating log
[storage]
address = localhost:3306
user = fabric
password =
database = fabric
connection_timeout = 6
[protocol.xmlrpc]
threads = 5
[logging]
level = INFO
url =
file:///var/log/fabric.log

10
Insert Picture Here
Connecting to a
MySQL Fabric Farm

Fabric-aware Connector API
 Load Balancing
 Read-Write Split
 Distribute transactions
 Fabric-aware Connectors
 Connector/J
 Connector/Python
 Connector/PHP
 Fabric-aware Frameworks
 Doctrine
 Hibernate
 Focus on Connector/Python

12
Insert Picture Here
Architecture for
Sharding

Benefits of Sharding
 Write scalability
 Can handle more writes
 Large data set
 Database too large
 Does not fit on single server
 Improved performance
 Smaller index size
 Smaller working set
 Improve performance
UID 10000-20000 UID 20001-40000

MySQL Fabric: Sharding Goals & Features
 Shard Multiple Tables
 Using same key
 Global Updates
 Global tables
 Schema updates
 Sharding Functions
 Range
 (Consistent) Hash
 Shard Operations
 Using built-in executor
 Shard move
 Shard split

Mapping the Sharding Key
 What is a sharding key?
 Single column
 Multi column
− Same table?
− Different tables?
 How is the key transformed?
 Hash
 Range
 User-defined
Compute
Shard#
KeyShard#
(X)
(X,Y,...)
RANGE
HASH
Something else

Sharded Tables
In desperate need
of sharding!
Foreign keys

Multi-table Query with Sharded Tables
SELECT first_name, last_name, salary
FROM salaries JOIN employees USING (emp_no)
WHERE emp_no = 21012
AND CURRENT_DATE BETWEEN from_date AND to_date;
 Referential Integrity Constraint
 Example query joining salaries and employees
 Same key, same shard: co-locate rows for same user
 JOIN normally based on equality
 Using non-equality defeats purpose of foreign key

Global Tables
Does not need
to be sharded

Multi-table Query with Global Tables
SELECT first_name, last_name, GROUP_CONCAT(dept_name)
FROM employees JOIN dept_emp USING (emp_no)
JOIN departments USING (dept_no)
WHERE emp_no = 21012 GROUP BY emp_no;
 JOIN with departments table
 Has no employee number, hence no sharding key
 Table need to be present on all shards
 How do we update global tables?

Sharding Architecture
Shards
MySQL Fabric Node
Application
Global
Group
Global Updates
Shard
Updates
Replication
Support global update
for off-line shards

Demo

Demo Server Side – Startup Instances
3306
3606
Fabric
Repository
3506
Fabric Node
32274
3308 3314 3408 3406

Demo Server Side – Configuration of Fabric
/etc/mysql/fabric.cfg
[DEFAULT]
prefix =
sysconfdir = /etc
logdir = /var/log
[storage]
user = fabric
password = welcome1
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1
[servers]
user = fabric
password = welcome1
[protocol.xmlrpc]
threads = 5
user = admin
password =welcome1
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =
[executor]
executors = 5
[logging]
level = INFO
url = file:///var/log/fabric.log

Startup Fabric
•Fabric help command
mysqlfabric help commands
•Instantiate Fabric
mysqlfabric manage setup
•Bring up Fabric
mysqlfabric manage start --deamonize

Demo – Configure by Fabric
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314 3408 3406

Fabric Group Operation
 Create 1 global group, 2 shard group
mysqlfabric group create gp-global
mysqlfabric group create gp1
mysqlfabric group create gp2
 Assign Servers to groups
mysqlfabric group add gp-global 127.0.0.1:3606
mysqlfabric group add gp1 127.0.0.1:3306
 Pick the master server by Fabric
mysqlfabric group promote gp1
mysqlfabric group promote gp2

Fabric Group Operations
• Let Fabric manage auto-failover for you
mysqlfabric group activate gp-global
mysqlfabric group activate gp1
mysqlfabric group activate gp2
• Define Range shard on to global group
mysqlfabric sharding create_definition RANGE gp-global
• Add shard table – employees to the shard created (shard map
id = 1) shard by id column
mysqlfabric sharding add_table 1 employees.employees id
• Assign rows with id value 1~999 to group gp1, 1000 above to
group gp2
mysqlfabric sharding add_shard 1 gp1/1,gp2/1000 --state=ENABLED

Demo – Databases is Ready
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406

Demo Application Side
– Need Connector/J 5.1.30 or later
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J

Demo Application Side
– Cache Fabric Configuration on creating DB Connection
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object

Java Application
- Global tables and DDLs are accessed by property fabricServerGroup
import com.mysql.fabric.jdbc.FabricMySQLConnection;
…
rawConnection = DriverManager.getConnection( baseUrl +
database + "?fabricServerGroup=gp-global“,user, password);
statement = rawConnection.createStatement();
…
statement.executeUpdate("create table employees.employees (emp_no int
not null,first_name varchar(50),last_name varchar(50), department_id
int,primary key (emp_no))");
…
statement.executeUpdate("insert into employees.departments values
(1,'Sales'),(2,'Financial')");

Demo Environment
– Dispatch Request Based On Cached Configuration with Connection
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object –
Shard Table
Write Sharded Table
Write Global Table

Demo Environment
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object –
Shard Table
Write Sharded Table
Write Global Table

Shared tables are accessed by property
fabricShardTable and fabericShardKey
rawConnection =
DriverManager.getConnection( "jdbc:mysql:fabric://127.0.
0.1:32274/mysql?fabricShardTable=employees.employees“, user,
password);
…
for (int i = 0; i < 4; ++i) {
connection.setShardKey(ids[i].toString());
ps.setInt(1, ids[i]);
ps.setString(2, firstNames[i]);
ps.setString(3, lastNames[i]);
ps.setString(4, departmentIds[i]);
ps.executeUpdate();
}

Demo Environment
– Read/Write Split within Shard
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object –
Shard Table
Read Only Operations
Read Global Table

Read Only Operations
FabricMySQLConnection newConn = (FabricMySQLConnection)
rawConnection;
newConn.setReadOnly(true);
ps = newConn.prepareStatement(
"select emp_no, first_name, last_name, d.name, i.variable_value
from employees.employees e, information_schema.global_variables i,
employees.departments d where e.department_id = d.department_id
and i.variable_name='port' and emp_no = ?");

Demo Environment – Failure Detected
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object
Write Global Table
Change Master to 3606
Change Master to 3308
X

Demo Environment – Failover within Shard
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308-M 3314-M 3408-M 3406
JavaConnector/J
Connection
Object
Write Shard Table
Write Global Table
X

40
Insert Picture Here
Closing Remarks

Thoughts for the Future
 Connector multi-cast
 Scatter-gather
 UNION of result sets
 More complex operations?
 Internal interfaces
 Improve extension support
 Improve procedures
support
 Command-line interface
 Improving usability
 Focus on ease-of-use
 More protocols
 MySQL-RPC Protocol?
 AMQP?
 More frameworks?
 More HA group types
 DRBD
 MySQL Cluster
 Fabric-unaware connectors?

My sql fabric webinar tw2

More Related Content

What's hot (20)

Similar to My sql fabric webinar tw2 (20)

More from Ivan Tu (13)

My sql fabric webinar tw2