SlideShare a Scribd company logo
Pedro Gomes
MySQL Senior Software Developer
Frédéric Descamps
MySQL Community Manager
EMEA & APAC
MySQL InnoDB Cluster
Advanced Configuration &
Operations
 
2 / 161
 
Safe Harbor
The following is intended to outline our general product direction. It is intended for information purpose only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any
features or functionality described for Oracle´s product may change and remains at the sole discretion of Oracle
Corporation.
Statement in this presentation relating to Oracle´s future plans, expectations, beliefs, intentions and ptospects
are "forward-looking statements" and are subject to material risks and uncertainties. A detailed discussion of
these factors and other risks that a ect our business is contained in Oracle´s Securities and Exchange
Commission (SEC) lings, including our most recent reports on Form 10-K and Form 10-Q under the heading
"Risk Factors". These lings are available on the SEC´s website or on Oracle´s website at
h p://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle
undertakes no duty to update any statement in light of new information or future events.
Copyright @ 2019 Oracle and/or its affiliates.
3 / 161
about us
Who are we ?
Copyright @ 2019 Oracle and/or its affiliates.
4 / 161
Pedro Gomes
@distributedpete
MySQL Replication Developer
MySQL Developer since 2012
Portuguese born and raised 🇵🇹
h ps://mysqlhighavailability.com
Copyright @ 2019 Oracle and/or its affiliates.
5 / 161
Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
living in Belgium 🇧🇪
h ps://lefred.be
Copyright @ 2019 Oracle and/or its affiliates.
6 / 161
MySQL InnoDB Cluster
New Member Provisioning & Recovery
Copyright @ 2019 Oracle and/or its affiliates.
7 / 161
Provisioning VS Recovery
Provisioning a server is the process of con guring and adding data to a server, which
can then potentially be used in a replication topology. Typically, this involves restoring a
backup or snapshot into a given instance.
The distributed recovery is the protocol that runs when a server is added to a cluster, so
that the new server nds a state donor, fetches missing state and synchronizes with the
rest of the cluster automatically.
As of MySQL 8.0.17, the distributed recovery protocol for Group Replication is able to
automatically orchestrate provisioning, in addition to using binary logs, in the process of
synchronizing the new instance with the rest of the group
Copyright @ 2019 Oracle and/or its affiliates.
8 / 161
How to Provision a New Member
Incremental Recovery
New node for a recent Cluster
From a Backup
Clone (NEW 8.0.17)
Copyright @ 2019 Oracle and/or its affiliates.
9 / 161
Incremental Recovery
Based on the good old Asynchronous replication
Fetches only the missing data based on the member GTIDs info
Also provides the member with group transactional information
Copyright @ 2019 Oracle and/or its affiliates.
10 / 161
Based on the brand new Clone plugin
Fetches a full snapshot of the donor data
Replaces the server data and requires its
restart
Still requires incremental recovery for
group transactional information
 
Clone
Copyright @ 2019 Oracle and/or its affiliates.
11 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
12 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
13 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
14 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
15 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
16 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
hello, I would
like to join !
Copyright @ 2019 Oracle and/or its affiliates.
17 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
I have no
data yet !
Copyright @ 2019 Oracle and/or its affiliates.
18 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
19 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Option 1: Incremental
donor joiner
binlog
Copyright @ 2019 Oracle and/or its affiliates.
20 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Option 1: Incremental
donor joiner
binlog
Copyright @ 2019 Oracle and/or its affiliates.
21 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Option 1: Incremental
donor joiner
binlog
Copyright @ 2019 Oracle and/or its affiliates.
22 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Option 1: Incremental
binlog
Copyright @ 2019 Oracle and/or its affiliates.
23 / 161
MySQL InnoDB Cluster Provisioning
Copyright @ 2019 Oracle and/or its affiliates.
24 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Option 2: Clone
binlog
donor joiner
clone
Copyright @ 2019 Oracle and/or its affiliates.
25 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
Option 2: Clone
binlog
donor joiner
clone
Copyright @ 2019 Oracle and/or its affiliates.
26 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
hello, I would
like to join !
binlog
Copyright @ 2019 Oracle and/or its affiliates.
27 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
hello, I would
like to join !
binlog
And I have
no data yet !
Copyright @ 2019 Oracle and/or its affiliates.
28 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
We are not
able to provide
you incremental data !
Use CLONE !
binlog
Copyright @ 2019 Oracle and/or its affiliates.
29 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog
donor joiner
clone
binlog
Copyright @ 2019 Oracle and/or its affiliates.
30 / 161
MySQL InnoDB Cluster Provisioning
binlog binlog binlog
Copyright @ 2019 Oracle and/or its affiliates.
31 / 161
MySQL InnoDB Cluster Provisioning
Copyright @ 2019 Oracle and/or its affiliates.
32 / 161
binlog binlog binlog
restored
backup
MySQL InnoDB Cluster Provisioning
What kind of provisioning will happen ?
Copyright @ 2019 Oracle and/or its affiliates.
33 / 161
binlog binlog binlog
restored
backup  
Incremental Recovery !
MySQL InnoDB Cluster Provisioning
What kind of provisioning will happen ?
Copyright @ 2019 Oracle and/or its affiliates.
34 / 161
MySQL InnoDB Cluster Provisioning
Copyright @ 2019 Oracle and/or its affiliates.
35 / 161
MySQL InnoDB Cluster Provisioning
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to force the use of Clone when adding an instance even if
Incremental Recovery is possible.
36 / 161
MySQL InnoDB Cluster Provisioning
Just specify it in the MySQL Shell:
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to force the use of Clone when adding an instance even if
Incremental Recovery is possible.
37 / 161
Recovery Process
The recovery process happens when a previous member joins back a Cluster/Group.
By default and if possible (based on GTID set), the Incremental Recovery method is used.
However, sometimes applying a large amount of binary log events might be slower than
performing a Clone. You have the possibility to con gure the treshold:
group_replication_clone_threshold (default is 9223372036854775807 !):
Copyright @ 2019 Oracle and/or its affiliates.
38 / 161
Recovery Process (2)
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to force the use of Clone when rejoining an instance even if
Incremental Recovery is enough to recover.
39 / 161
Recovery Process (2)
It's possible ! You need to connect to the instance you want to rejoin and set
group_replication_clone_threshold to 1.
As Clone will be called and the change is not persisted, no need to set it back to its original
value. Never persist it to 1 or to a really low value on a live Group, never!
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to force the use of Clone when rejoining an instance even if
Incremental Recovery is enough to recover.
40 / 161
Recovery Process (3)
Copyright @ 2019 Oracle and/or its affiliates.
41 / 161
Force a Donor during Clone
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to add a node (mysql3) but it should only clone from server
mysql2
42 / 161
Force a Donor during Clone
This is again possible, you need to connect on the other members (in this case mysql1 and
unload the clone plugin).
Please note, that to be able to unload the plugin you need to be connected using the
classic protocol (3306).
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to add a node (mysql3) but it should only clone from server
mysql2
43 / 161
We connect using the classic protocol and
we uninstall the plugin:
Force a Donor during Clone (2)
Copyright @ 2019 Oracle and/or its affiliates.
44 / 161
We connect using the classic protocol and
we uninstall the plugin:
Then we add the instance forcing Clone:
Force a Donor during Clone (2)
Copyright @ 2019 Oracle and/or its affiliates.
45 / 161
Force a Donor during Clone (3)
Copyright @ 2019 Oracle and/or its affiliates.
46 / 161
Force a Donor during Clone (4)
We can also verify this on the new joiner Performance_Schema.clone_status table:
Copyright @ 2019 Oracle and/or its affiliates.
47 / 161
Force a Donor during Clone (5)
Don't forget to install the Clone plugin again:
Copyright @ 2019 Oracle and/or its affiliates.
48 / 161
Force a Donor during Clone (5)
Don't forget to install the Clone plugin again:
Copyright @ 2019 Oracle and/or its affiliates.
49 / 161
MySQL InnoDB Cluster
Monitoring
Copyright @ 2019 Oracle and/or its affiliates.
50 / 161
Monitoring Provisioning & Recovery process
Currently, when the CLONE is triggered within MySQL Shell, you can follow its progress:
Copyright @ 2019 Oracle and/or its affiliates.
51 / 161
Monitoring Provisioning & Recovery process
And we already saw that the status and the progress is also available in performance_schema
tables on the joiner:
Copyright @ 2019 Oracle and/or its affiliates.
52 / 161
Monitoring Provisioning & Recovery process
Copyright @ 2019 Oracle and/or its affiliates.
53 / 161
Incremental Recovery process
It's also possible to evaluate the amount of transactions a member needs to process before
being in sync with the Group when joining. You can nd a User-De ned Report to see that
amount :
More Info: h ps://lefred.be/content/mysql-innodb-cluster-recovery-process-monitoring-
with-the-mysql-shell-reporting-framework/
Copyright @ 2019 Oracle and/or its affiliates.
54 / 161
By default the status()
method disables the
extended mode.
It's possible to have much
more information enabling
it.
MySQL InnoDB Cluster - status()
Copyright @ 2019 Oracle and/or its affiliates.
55 / 161
MySQL InnoDB Cluster - status({extended})
The extended option allows 4 values (or boolean):
0: disables the command verbosity (default)
1: includes information about the Group Protocol Version, Group name, cluster member
UUIDs, cluster member roles and states as reported by Group Replication and the list of
fenced system variables
2: includes information about transactions processed by connection and applier
3: includes more detailed stats about the replication machinery of each cluster member
Copyright @ 2019 Oracle and/or its affiliates.
56 / 161
MySQL InnoDB Cluster - status({extended:1})
Copyright @ 2019 Oracle and/or its affiliates.
57 / 161
MySQL InnoDB Cluster - status({extended:2})
Copyright @ 2019 Oracle and/or its affiliates.
58 / 161
MySQL InnoDB Cluster - status({extended:3})
Copyright @ 2019 Oracle and/or its affiliates.
59 / 161
Monitoring MySQL InnoDB Cluster -
Performance_Schema
Other than the status() in the MySQL Shell, it's possible to get information from
Performance_Schema tables:
Copyright @ 2019 Oracle and/or its affiliates.
60 / 161
Monitoring MySQL InnoDB Cluster -
Performance_Schema (2)
Copyright @ 2019 Oracle and/or its affiliates.
61 / 161
Monitoring MySQL InnoDB Cluster -
Performance_Schema (3)
Copyright @ 2019 Oracle and/or its affiliates.
62 / 161
Monitoring MySQL InnoDB Cluster -
Performance_Schema (4)
Copyright @ 2019 Oracle and/or its affiliates.
63 / 161
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I would like to know which of my Secondary-Master is slower ?
64 / 161
Performance_Schema provides to us all what we need to discover which Secondary-Slave
takes more time to "fully replicate" events.
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I would like to know which of my Secondary-Master is slower ?
65 / 161
Let's run this bash code on all our Secondary-Masters (mysql1 and mysql2):
for i in $(seq 1 10)
do
if [[ $i -eq 1 ]]; then echo "repl delay transport time time to RL apply time"
mysql -BN -e "
SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
'rep delay (sec)',
LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
'transport time',
LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP 'time RL',
LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP 'apply time'
FROM performance_schema.replication_applier_status_by_worker t1
JOIN performance_schema.replication_connection_status t2
ON t2.channel_name=t1.channel_name
WHERE t1.channel_name='group_replication_applier'";
sleep 2
done
Copyright @ 2019 Oracle and/or its affiliates.
66 / 161
Let's run this bash code on all our Secondary-Masters (mysql1 and mysql2):
for i in $(seq 1 10)
do
if [[ $i -eq 1 ]]; then echo "repl delay transport time time to RL apply time"
mysql -BN -e "
SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
'rep delay (sec)',
LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
'transport time',
LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP 'time RL',
LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP 'apply time'
FROM performance_schema.replication_applier_status_by_worker t1
JOIN performance_schema.replication_connection_status t2
ON t2.channel_name=t1.channel_name
WHERE t1.channel_name='group_replication_applier'";
sleep 2
done
This script will query some performance_schema tables 10 times every 2 seconds (we write
on our cluster every 2 secs), and all members have their time in sync: ntpd.
Copyright @ 2019 Oracle and/or its affiliates.
67 / 161
Monitoring MySQL InnoDB Cluster -
Performance_Schema (5)
 
Copyright @ 2019 Oracle and/or its affiliates.
68 / 161
Monitoring MySQL InnoDB Cluster -
Performance_Schema (5)
 
Copyright @ 2019 Oracle and/or its affiliates.
69 / 161
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I would like to estimate how many transactions per second could my
cluster process with the current configuration.
70 / 161
To perform this estimation, we will block all writes to a member of our cluster. As a result,
the apply queue will grow on that server. When the queue is large enough, we will enable
writes again on the node and see how many transactions and how much time the node
needed to process everything and be back in sync with the cluster.
Currently, the application is performing 100 tx/sec.
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I would like to estimate how many transactions per second could my
cluster process with the current configuration.
71 / 161
On the node that we will lock, let's run this small bash command:
while true
do
mysql -BNe "select now(), transactions_behind, count_transactions_remote_applied
from sys.gr_member_routing_candidate_status, performance_schema.replication_group_member_stats
where member_id=@@server_uuid;"
sleep 1
done
Copyright @ 2019 Oracle and/or its affiliates.
72 / 161
On the node that we will lock, let's run this small bash command:
while true
do
mysql -BNe "select now(), transactions_behind, count_transactions_remote_applied
from sys.gr_member_routing_candidate_status, performance_schema.replication_group_member_stats
where member_id=@@server_uuid;"
sleep 1
done
Copyright @ 2019 Oracle and/or its affiliates.
73 / 161
Copyright @ 2019 Oracle and/or its affiliates.
74 / 161
Copyright @ 2019 Oracle and/or its affiliates.
75 / 161
So we can estimate that our MySQL InnoDB Cluster is able to process approximately 350
tx/sec.
So what will happen if we increase our workload to 360 tx/sec ?
Copyright @ 2019 Oracle and/or its affiliates.
76 / 161
 
Copyright @ 2019 Oracle and/or its affiliates.
77 / 161
The cluster will start
slowing down and this
might have a major impact
depending of the
consistency level.
 
Copyright @ 2019 Oracle and/or its affiliates.
78 / 161
More Info: h ps://lefred.be/content/using-the-new-mysql-
shell-reporting-framework-to-monitor-innodb-cluster/
Monitoring MySQL InnoDB Cluster - MySQL
Shell UDR
It's also possible to constantly monitor the status of your MySQL InnoDB Cluster using
User-De ned Reports in MySQL Shell:
Copyright @ 2019 Oracle and/or its affiliates.
79 / 161
[http_server]
port=8080
[rest_api]
[rest_router]
require_realm=somerealm
[rest_routing]
require_realm=somerealm
[rest_metadata_cache]
require_realm=somerealm
[http_auth_realm:somerealm]
backend=somebackend
method=basic
name=Some Realm
[http_auth_backend:somebackend]
backend= le
lename=/etc/mysqlrouter/mysqlrouter.pwd
Monitoring MySQL Router
Since MySQL 8.0.17 it's possible to query the MySQL Router using its REST API !
You need to enable it in the Router's con g le:
Copyright @ 2019 Oracle and/or its affiliates.
80 / 161
Monitoring MySQL Router (2)
You can then query the REST API using curl:
$ curl -s -u fred:fred 
http://192.168.91.2:8080/api/20190715/routes/myCluster_default_rw/destinations
{"items":[{"address":"mysql2","port":3306}]}
Or use/create a plugin for the MySQL Shell.
More Info:
h ps://lefred.be/content/mysqlrouter-8-0-17-and-the-rest-api/
h ps://lefred.be/content/mysql-router-8-0-17s-rest-api-mysql-shell-extensions/
Copyright @ 2019 Oracle and/or its affiliates.
81 / 161
 
Monitoring MySQL Router (3)
Copyright @ 2019 Oracle and/or its affiliates.
82 / 161
Monitoring MySQL Router (4)
Copyright @ 2019 Oracle and/or its affiliates.
83 / 161
MySQL InnoDB Cluster
Network Monitoring
Copyright @ 2019 Oracle and/or its affiliates.
84 / 161
 
All good ;)
Network monitoring - the basics
Copyright @ 2019 Oracle and/or its affiliates.
85 / 161
 
When members crash or are expelled
Network monitoring - the basics
Copyright @ 2019 Oracle and/or its affiliates.
86 / 161
 
When a majority of the
members becomes
unreachable
Network monitoring - the basics
Copyright @ 2019 Oracle and/or its affiliates.
87 / 161
Network monitoring - Adapt to faulty networks
To avoid frequent member expels due to faulty networks make use of:
group_replication_member_expel_timeout
Copyright @ 2019 Oracle and/or its affiliates.
88 / 161
Network monitoring - Deal with failure scenarios
Con gure your rejoin a empts for each member (this method persists the change)
Copyright @ 2019 Oracle and/or its affiliates.
89 / 161
Network monitoring - Deal with failure scenarios
Con gure your rejoin a empts for each member (this method persists the change)
If you prefer to not persist it:
Copyright @ 2019 Oracle and/or its affiliates.
90 / 161
Network monitoring - Deal with failure scenarios
Con gure your timeout during loss of quorum
Copyright @ 2019 Oracle and/or its affiliates.
91 / 161
Network monitoring - Deal with failure scenarios
Be prepared: whom shall be the primary when failure strikes
Copyright @ 2019 Oracle and/or its affiliates.
92 / 161
Network monitoring - Adapt to higher latencies
Copyright @ 2019 Oracle and/or its affiliates.
93 / 161
Network monitoring - Adapt to higher latencies
group replication write concurrency
Copyright @ 2019 Oracle and/or its affiliates.
94 / 161
Network monitoring - Adapt to higher latencies
group replication write concurrency
Copyright @ 2019 Oracle and/or its affiliates.
95 / 161
Network monitoring - Adapt to higher latencies
group replication write concurrency
Copyright @ 2019 Oracle and/or its affiliates.
96 / 161
Network monitoring - Adapt to higher latencies
Copyright @ 2019 Oracle and/or its affiliates.
97 / 161
Network monitoring - Adapt to higher latencies
Copyright @ 2019 Oracle and/or its affiliates.
98 / 161
Network monitoring - Adapt to higher latencies
Copyright @ 2019 Oracle and/or its affiliates.
99 / 161
Network monitoring - Adapt to higher latencies
Copyright @ 2019 Oracle and/or its affiliates.
100 / 161
Con gure your write
concurrency in the group
Network monitoring - Adapt to higher latencies
 
Copyright @ 2019 Oracle and/or its affiliates.
101 / 161
MySQL InnoDB Cluster
Operations
Copyright @ 2019 Oracle and/or its affiliates.
102 / 161
Upgrading to a newer version
Before upgrading, don't forget to use util.upgradeChecker() in MySQL Shell. This
operation must be performed on the Primary Master:
Copyright @ 2019 Oracle and/or its affiliates.
103 / 161
Upgrading Process
PM SM SM
8.0.16 8.0.16 8.0.16
Copyright @ 2019 Oracle and/or its affiliates.
104 / 161
Upgrading Process
PM SM SM
8.0.16 8.0.16 8.0.16
util.checkForServerUpgrade( )PM
Copyright @ 2019 Oracle and/or its affiliates.
105 / 161
Upgrading Process
PM SM
8.0.16 8.0.16 8.0.16
stop mysqld on a SM
Copyright @ 2019 Oracle and/or its affiliates.
106 / 161
Upgrading Process
PM SM
8.0.16 8.0.16 8.0.17
upgrade MySQL binaries
Copyright @ 2019 Oracle and/or its affiliates.
107 / 161
Upgrading Process
PM SM
8.0.16 8.0.16 8.0.17
start mysqld
Copyright @ 2019 Oracle and/or its affiliates.
108 / 161
Upgrading Process
PM SM
8.0.16 8.0.16 8.0.17
join
the group
Copyright @ 2019 Oracle and/or its affiliates.
109 / 161
Upgrading Process
PM SM SM
8.0.16 8.0.16 8.0.17
Copyright @ 2019 Oracle and/or its affiliates.
110 / 161
Upgrading Process
PM SM
8.0.16 8.0.16 8.0.17
stop mysqld on the other SM
Copyright @ 2019 Oracle and/or its affiliates.
111 / 161
Upgrading Process
PM SM
8.0.16 8.0.17 8.0.17
upgrade the MySQL binaries
Copyright @ 2019 Oracle and/or its affiliates.
112 / 161
Upgrading Process
PM SM
8.0.16 8.0.17 8.0.17
start mysqld
Copyright @ 2019 Oracle and/or its affiliates.
113 / 161
Upgrading Process
PM SM
8.0.16 8.0.17 8.0.17
join
the group
join
the group
Copyright @ 2019 Oracle and/or its affiliates.
114 / 161
Upgrading Process
PM SM SM
8.0.16 8.0.17 8.0.17
Copyright @ 2019 Oracle and/or its affiliates.
115 / 161
Upgrading Process
PM SM
8.0.16 8.0.17 8.0.17
stop mysqld on the PM
Copyright @ 2019 Oracle and/or its affiliates.
116 / 161
Upgrading Process
PM SM
8.0.17 8.0.17 8.0.17
upgrade MySQL binaries
Copyright @ 2019 Oracle and/or its affiliates.
117 / 161
Upgrading Process
PM SM
8.0.17 8.0.17 8.0.17
start myslqd
Copyright @ 2019 Oracle and/or its affiliates.
118 / 161
Upgrading Process
PM SM
8.0.17 8.0.17 8.0.17
join
the group
Copyright @ 2019 Oracle and/or its affiliates.
119 / 161
Upgrading Process
PMSM SM
8.0.17 8.0.17 8.0.17
Copyright @ 2019 Oracle and/or its affiliates.
120 / 161
Upgrading Process Info
The lowest version of a Group must always be the Primary Master
If ejected from the Group, the lowest version instance won't be able to rejoin the Group
again
Copyright @ 2019 Oracle and/or its affiliates.
121 / 161
Downgrading
Downgrading an existing instance won't work as downgrading MySQL Server is not
supported:
[ERROR] [MY-013171] [InnoDB] Cannot boot server version 80016 on data directory built by version 80017.
Downgrade is not supported.
However it's possible to downgrade your MySQL InnoDB Cluster without downtime, by
using a previous backup.
Before 8.0.17, any 8.0.x could join a MySQL InnoDB Cluster 8.0. Since 8.0.17, the minor
version is also checked.
Copyright @ 2019 Oracle and/or its affiliates.
122 / 161
Downgrading (2)
So if you want to have a lower version to join a cluster, you need to set
group_replication_allow_local_lower_version_join to ON on the joiner.
But don't forget that this instance running with a lower version should be promoted to be
the Primary-Master as soon as possible to avoid any possible issues related to protocol and
variable di erences.
Copyright @ 2019 Oracle and/or its affiliates.
123 / 161
Backup
Taking a backup of your MySQL InnoDB Cluster is trivial. You can use exactly the same
method as you are using for a standalone MySQL Server:
Physical (MySQL Enterperise Backup (MEB), Xtrabackup)
Logical (mysqldump, mysqlpump, mydumper)
Snapshot (lvm, MySQL Clone)
Copyright @ 2019 Oracle and/or its affiliates.
124 / 161
Backup (2)
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA using MySQL InnoDB Cluster retains data consistency, I only want to
backup one member of the Group and preferably a Secondary Master, no need
to waste disk space with multiple backups of the same data.
125 / 161
Backup (2)
This is possible ! Let's de ne the best backup solution.
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA using MySQL InnoDB Cluster retains data consistency, I only want to
backup one member of the Group and preferably a Secondary Master, no need
to waste disk space with multiple backups of the same data.
126 / 161
Backup (3)
The perfect backup solution for you MySQL InnoDB Cluster, should then perform the
following steps:
check if the node where the script run is part of the cluster
check if the node is indeed a secondary master
eventually check if the node is lagging behind (large apply queue)
ensure that the backup is not running on another member
Copyright @ 2019 Oracle and/or its affiliates.
127 / 161
Backup (3)
The perfect backup solution for you MySQL InnoDB Cluster, should then perform the
following steps:
check if the node where the script run is part of the cluster
check if the node is indeed a secondary master
eventually check if the node is lagging behind (large apply queue)
ensure that the backup is not running on another member
Here, you can nd a working script calling MEB to use as a cron job :
h ps://lefred.be/content/how-to-backup-your-innodb-cluster/
Copyright @ 2019 Oracle and/or its affiliates.
128 / 161
DDLs
There is nothing special about DDLs. When DDLs are performed, the cluster is not blocked
completely.
Of course, concurent DDLs and DMLs are not a good idea when using a cluster in Multi-
Primary mode.
If you use a Multi-Primary Cluster, you must swicth to Single-Primary mode the time of the
DDL.
Don't forget that MySQL 8.0 supports also Instant DDLs like adding a column.
Copyright @ 2019 Oracle and/or its affiliates.
129 / 161
DDLs on Multi-Primary Clusters
Copyright @ 2019 Oracle and/or its affiliates.
130 / 161
DDLs on Multi-Primary Clusters
Copyright @ 2019 Oracle and/or its affiliates.
131 / 161
Configurable Consistency Guarantees
Consistency Levels
Copyright @ 2019 Oracle and/or its affiliates.
132 / 161
mysql> show variables like
'group_replication_consistency';
+-------------------------------+----------+
| Variable_name | Value |
+-------------------------------+----------+
| group_replication_consistency | EVENTUAL |
+-------------------------------+----------+
 
Consistency: EVENTUAL (default)
By default, there is no synchronization point for the transactions, when you perform a write
on a node, if you immediately read the same data on another node, it is eventually there.
Copyright @ 2019 Oracle and/or its affiliates.
133 / 161
mysql> show variables like
'group_replication_consistency';
+-------------------------------+----------+
| Variable_name | Value |
+-------------------------------+----------+
| group_replication_consistency | EVENTUAL |
+-------------------------------+----------+
 
Consistency: EVENTUAL (default)
By default, there is no synchronization point for the transactions, when you perform a write
on a node, if you immediately read the same data on another node, it is eventually there.
Since MySQL 8.0.16, we have the possibility to
set the synchronization point at read or at write or both (globally or for a session).
Copyright @ 2019 Oracle and/or its affiliates.
134 / 161
Consistency: BEFORE (READ)
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to load balance my reads without deploying additional
restrictions on which server I read from to avoid reading stale data, my group
writes are much more than my group reads.
135 / 161
Consistency: BEFORE (READ)
Copyright @ 2019 Oracle and/or its affiliates.
As a DBA, I want to load balance my reads without deploying additional
restrictions on which server I read from to avoid reading stale data, my group
writes are much more than my group reads.
As a developer, I want specific transactions in my workload to always read up-
to-date data from the group, so that whenever that sensitive data is updated, I
will enforce that reads shall read the most up to date value.
136 / 161
Consistency: BEFORE (READ)
Copyright @ 2019 Oracle and/or its affiliates.
137 / 161
Consistency: AFTER (WRITE)
Copyright @ 2019 Oracle and/or its affiliates.
I want to load balance my reads without deploying additional restrictions on
which server I read from to avoid reading stale data, my group writes are
much less than my group reads.
138 / 161
Consistency: AFTER (WRITE)
Copyright @ 2019 Oracle and/or its affiliates.
I want to load balance my reads without deploying additional restrictions on
which server I read from to avoid reading stale data, my group writes are
much less than my group reads.
I have a group that mostly does reads-only, I want my read-write transactions
to be applied everywhere once they commit, so that subsequent reads are done on
up-to-date data that includes my latest write. Without paying at reads.
139 / 161
Consistency: AFTER (WRITE)
Copyright @ 2019 Oracle and/or its affiliates.
140 / 161
Consistency: BEFORE_AND_AFTER
Copyright @ 2019 Oracle and/or its affiliates.
I want that my application to replicate data as close as possible to synchronous.
And of course I'm OK to pay the required price !
141 / 161
Consistency: BEFORE_AND_AFTER
Copyright @ 2019 Oracle and/or its affiliates.
142 / 161
Defining Consistency
The consistency level of the MySQL InnoDB Cluster can be de ned during the creation:
Copyright @ 2019 Oracle and/or its affiliates.
143 / 161
Defining Consistency (2)
The consistency can be globally changed in MySQL Shell:
It's also possible and recommended for the most restrictive levels to de ne it at
statement session level:
Copyright @ 2019 Oracle and/or its affiliates.
144 / 161
MySQL InnoDB Cluster
MySQL Router Specific Settings
Copyright @ 2019 Oracle and/or its affiliates.
145 / 161
MySQL Router : Metadata Cache
The se ings to access MySQL InnoDB Cluster's Metadata Cache is de ned in the
[metadata_cache:<name>] section of MySQL Router's con g le.
Copyright @ 2019 Oracle and/or its affiliates.
146 / 161
MySQL Router : Metadata Cache
The se ings to access MySQL InnoDB Cluster's Metadata Cache is de ned in the
[metadata_cache:<name>] section of MySQL Router's con g le.
The most interesting se ings is:
ttl: Time to live (in seconds) of information in the metadata cache. One milissecond is
0.001. To continously query the the metadata cache in a tight loop, set it to 0 (the
default value is 0.5).
Copyright @ 2019 Oracle and/or its affiliates.
147 / 161
MySQL Router : Metadata Cache (2)
As of MySQL Router 8.0.17, ttl is now obsolete as you can enable
use_gr_noti cations to refresh metadata on each of the four noti cations that Group
Replication sends:
group_replication/membership/quorum_loss
group_replication/membership/view
group_replication/status/role_change
group_replication/status/state_change
Copyright @ 2019 Oracle and/or its affiliates.
148 / 161
MySQL Router
By default, when bootstrapped, MySQL Router is con gured using rst-available as
routing strategy for R/W connections (to the Primary-Master).
For R/O connections (to Secondary-Masters), the strategy is round-robin-with-fallback.
This is actually setup for the MySQL Classic Protocol and MySQL X Protocol.
Copyright @ 2019 Oracle and/or its affiliates.
149 / 161
MySQL Router : Routing Strategy Methods
rst-available: the new connection is routed to the rst available server from the
destinations list. In case of failure, the next available server is used. This cycle continues
until all servers are unavailable.
round-robin-with-fallback: for load-balancing, each new connection is made to the
next available secondary server in a round-robin fashion. If a secondary server is not
available then servers from the primary list are used in round-robin fashion.
Copyright @ 2019 Oracle and/or its affiliates.
150 / 161
MySQL Router : Routing Strategy Methods
round-robin: for load-balancing, each new connection is made to the next available
server in a round-robin fashion.
next-available: like rst-available, in that the new connection is routed to the rst
available server from the destinations list. Unlike rst-available, if a server is marked as
unreachable then it gets discarded and is never used again as a destination
Copyright @ 2019 Oracle and/or its affiliates.
151 / 161
MySQL Router : Routing destinations
Now that we have de ned a strategy for the MySQL Router, we also need to de ne
destinations.
Destinations (desinations in [routing:<name>] section), can be a comma-separated list of
MySQL Servers, or a metadata-cache de nition:
Copyright @ 2019 Oracle and/or its affiliates.
152 / 161
MySQL Router : Routing destinations
Now that we have de ned a strategy for the MySQL Router, we also need to de ne
destinations.
Destinations (desinations in [routing:<name>] section), can be a comma-separated list of
MySQL Servers, or a metadata-cache de nition:
[routing:PerconaLiveEU_default_rw]
bind_address=0.0.0.0
bind_port=6446
destinations=metadata-cache://PerconaLiveEU/default?role=PRIMARY
routing_strategy= rst-available
protocol=classic
Copyright @ 2019 Oracle and/or its affiliates.
153 / 161
MySQL InnoDB Cluster
Configuration & Settings
Copyright @ 2019 Oracle and/or its affiliates.
154 / 161
Instance Specific Settings
Within the MySQL Shell you can de ne some Group Replication speci c se ings via the
Admin API that will only a ect the targeted instance:
exitStateAction: string value indicating the group replication exit state action.
(ABORT_SERVER, READ_ONLY and OFFLINE_MODE [8.0.18])
memberWeight: integer value with a percentage weight for automatic primary election
on failover.
autoRejoinTries: integer value to de ne the number of times an instance will a empt to
rejoin the cluster after being expelled.
Copyright @ 2019 Oracle and/or its affiliates.
155 / 161
Instance Specific Settings (2)
label: a string identi er of the instance.
Copyright @ 2019 Oracle and/or its affiliates.
156 / 161
Group Specific Settings
Within the MySQL Shell you have the possibility to de ne some Group speci c se ings via
the Admin API:
clusterName: string value to de ne the cluster name.
exitStateAction: string value indicating the group replication exit state action.
memberWeight: integer value with a percentage weight for automatic primary election
on failover.
consistency: string value indicating the consistency guarantees that the cluster
provides.
Copyright @ 2019 Oracle and/or its affiliates.
157 / 161
Group Specific Settings (2)
expelTimeout: integer value to de ne the time period in seconds that cluster members
should wait for a non-responding member before evicting it from the cluster.
autoRejoinTries: integer value to de ne the number of times an instance will a empt to
rejoin the cluster after being expelled.
disableClone: boolean value used to disable the clone usage on the cluster.
Copyright @ 2019 Oracle and/or its affiliates.
158 / 161
Upgrade to MySQL 8.0
It's time to upgrade to MySQL 8.0, the fastest MySQL adoption release ever !
Copyright @ 2019 Oracle and/or its affiliates.
159 / 161
Copyright @ 2019 Oracle and/or its affiliates.
160 / 161
Thank you !
Copyright @ 2019 Oracle and/or its affiliates.
161 / 161

More Related Content

What's hot (20)

PDF
MySQL Group Replication - Ready For Production? (2018-04)
Kenny Gryp
 
PDF
Group Replication in MySQL 8.0 ( A Walk Through )
Mydbops
 
PDF
InnoDb Vs NDB Cluster
Mark Swarbrick
 
PDF
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
PDF
MySQL Database Architectures - High Availability and Disaster Recovery Solution
Miguel Araújo
 
PDF
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
Kenny Gryp
 
PPTX
Query logging with proxysql
YoungHeon (Roy) Kim
 
PDF
State of the Dolphin - May 2022
Frederic Descamps
 
PPTX
Running MariaDB in multiple data centers
MariaDB plc
 
PDF
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
PDF
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
PDF
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
Jesmar Cannao'
 
PDF
MariaDB MaxScale
MariaDB plc
 
PDF
Automated master failover
Yoshinori Matsunobu
 
PDF
MySQLレプリケーションあれやこれや
yoku0825
 
PDF
MySQL Group Replication: Handling Network Glitches - Best Practices
Frederic Descamps
 
PDF
MySQL 8.0で憶えておいてほしいこと
yoku0825
 
PDF
Load Balancing MySQL with HAProxy - Slides
Severalnines
 
PPTX
Data Guard Architecture & Setup
Satishbabu Gunukula
 
PDF
MySQL Router REST API
Frederic Descamps
 
MySQL Group Replication - Ready For Production? (2018-04)
Kenny Gryp
 
Group Replication in MySQL 8.0 ( A Walk Through )
Mydbops
 
InnoDb Vs NDB Cluster
Mark Swarbrick
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
MySQL Database Architectures - High Availability and Disaster Recovery Solution
Miguel Araújo
 
MySQL InnoDB Cluster - New Features in 8.0 Releases - Best Practices
Kenny Gryp
 
Query logging with proxysql
YoungHeon (Roy) Kim
 
State of the Dolphin - May 2022
Frederic Descamps
 
Running MariaDB in multiple data centers
MariaDB plc
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
Jesmar Cannao'
 
MariaDB MaxScale
MariaDB plc
 
Automated master failover
Yoshinori Matsunobu
 
MySQLレプリケーションあれやこれや
yoku0825
 
MySQL Group Replication: Handling Network Glitches - Best Practices
Frederic Descamps
 
MySQL 8.0で憶えておいてほしいこと
yoku0825
 
Load Balancing MySQL with HAProxy - Slides
Severalnines
 
Data Guard Architecture & Setup
Satishbabu Gunukula
 
MySQL Router REST API
Frederic Descamps
 

Similar to MySQL InnoDB Cluster - Advanced Configuration & Operations (20)

PDF
MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as ...
Miguel Araújo
 
PDF
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
Miguel Araújo
 
PDF
From single MySQL instance to High Availability: the journey to MySQL InnoDB ...
Frederic Descamps
 
PDF
How to operate MySQL InnoDB Cluster with MySQL Shell
Frederic Descamps
 
PPTX
The State of the Dolphin, MySQL Keynote at Percona Live Europe 2019, Amsterda...
Geir Høydalsvik
 
PDF
MySQL highav Availability
Baruch Osoveskiy
 
PDF
Buytaert kris my_sql-pacemaker
kuchinskaya
 
PDF
Confoo 202 - MySQL Group Replication and ReplicaSet
Dave Stokes
 
PDF
MySQL InnoDB Cluster and Group Replication in a nutshell hands-on tutorial
Frederic Descamps
 
PDF
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
NETWAYS
 
PDF
MySQL 8.0 InnoDB Cluster - Easiest Tutorial
Frederic Descamps
 
PDF
MySQL High Availability with Group Replication
Nuno Carvalho
 
PDF
20190817 coscup-oracle my sql innodb cluster sharing
Ivan Ma
 
PDF
Introduction to MySQL InnoDB Cluster
Frederic Descamps
 
PDF
Oracle Open World 2018 / Code One : MySQL 8.0 High Availability with MySQL I...
Frederic Descamps
 
PDF
MySQL 5.6 Replication Webinar
Mark Swarbrick
 
PDF
MySQL InnoDB Cluster and Group Replication in a Nutshell
Frederic Descamps
 
PDF
Training Slides: 203 - Backup & Recovery
Continuent
 
ODP
MySQL HA with PaceMaker
Kris Buytaert
 
PDF
Best practices for MySQL High Availability
Colin Charles
 
MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as ...
Miguel Araújo
 
MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell
Miguel Araújo
 
From single MySQL instance to High Availability: the journey to MySQL InnoDB ...
Frederic Descamps
 
How to operate MySQL InnoDB Cluster with MySQL Shell
Frederic Descamps
 
The State of the Dolphin, MySQL Keynote at Percona Live Europe 2019, Amsterda...
Geir Høydalsvik
 
MySQL highav Availability
Baruch Osoveskiy
 
Buytaert kris my_sql-pacemaker
kuchinskaya
 
Confoo 202 - MySQL Group Replication and ReplicaSet
Dave Stokes
 
MySQL InnoDB Cluster and Group Replication in a nutshell hands-on tutorial
Frederic Descamps
 
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
NETWAYS
 
MySQL 8.0 InnoDB Cluster - Easiest Tutorial
Frederic Descamps
 
MySQL High Availability with Group Replication
Nuno Carvalho
 
20190817 coscup-oracle my sql innodb cluster sharing
Ivan Ma
 
Introduction to MySQL InnoDB Cluster
Frederic Descamps
 
Oracle Open World 2018 / Code One : MySQL 8.0 High Availability with MySQL I...
Frederic Descamps
 
MySQL 5.6 Replication Webinar
Mark Swarbrick
 
MySQL InnoDB Cluster and Group Replication in a Nutshell
Frederic Descamps
 
Training Slides: 203 - Backup & Recovery
Continuent
 
MySQL HA with PaceMaker
Kris Buytaert
 
Best practices for MySQL High Availability
Colin Charles
 
Ad

More from Frederic Descamps (20)

PDF
MySQL Innovation & Cloud Day - Document Store avec MySQL HeatWave Database Se...
Frederic Descamps
 
PDF
MySQL Day Roma - MySQL Shell and Visual Studio Code Extension
Frederic Descamps
 
PDF
RivieraJUG - MySQL Indexes and Histograms
Frederic Descamps
 
PDF
RivieraJUG - MySQL 8.0 - What's new for developers.pdf
Frederic Descamps
 
PDF
Percona Live 2022 - MySQL Shell for Visual Studio Code
Frederic Descamps
 
PDF
Percona Live 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
PDF
LinuxFest Northwest 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
PDF
Open Source 101 2022 - MySQL Indexes and Histograms
Frederic Descamps
 
PDF
Pi Day 2022 - from IoT to MySQL HeatWave Database Service
Frederic Descamps
 
PDF
Confoo 2022 - le cycle d'une instance MySQL
Frederic Descamps
 
PDF
FOSDEM 2022 MySQL Devroom: MySQL 8.0 - Logical Backups, Snapshots and Point-...
Frederic Descamps
 
PDF
Les nouveautés de MySQL 8.0
Frederic Descamps
 
PDF
Les nouveautés de MySQL 8.0
Frederic Descamps
 
PDF
State of The Dolphin - May 2021
Frederic Descamps
 
PDF
Deploying Magento on OCI with MDS
Frederic Descamps
 
PDF
Cloud native - Why to use MySQL 8.0 and how to use it on oci with MDS
Frederic Descamps
 
PDF
MySQL Database Service Webinar: Installing Drupal in oci with mds
Frederic Descamps
 
PDF
the State of the Dolphin - October 2020
Frederic Descamps
 
PDF
MySQL 8.0 Document Store - Discovery of a New World
Frederic Descamps
 
PDF
Oracle Developer Live: Deploying MySQL InnoDB Cluster on OCI with Terraform
Frederic Descamps
 
MySQL Innovation & Cloud Day - Document Store avec MySQL HeatWave Database Se...
Frederic Descamps
 
MySQL Day Roma - MySQL Shell and Visual Studio Code Extension
Frederic Descamps
 
RivieraJUG - MySQL Indexes and Histograms
Frederic Descamps
 
RivieraJUG - MySQL 8.0 - What's new for developers.pdf
Frederic Descamps
 
Percona Live 2022 - MySQL Shell for Visual Studio Code
Frederic Descamps
 
Percona Live 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
LinuxFest Northwest 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
Open Source 101 2022 - MySQL Indexes and Histograms
Frederic Descamps
 
Pi Day 2022 - from IoT to MySQL HeatWave Database Service
Frederic Descamps
 
Confoo 2022 - le cycle d'une instance MySQL
Frederic Descamps
 
FOSDEM 2022 MySQL Devroom: MySQL 8.0 - Logical Backups, Snapshots and Point-...
Frederic Descamps
 
Les nouveautés de MySQL 8.0
Frederic Descamps
 
Les nouveautés de MySQL 8.0
Frederic Descamps
 
State of The Dolphin - May 2021
Frederic Descamps
 
Deploying Magento on OCI with MDS
Frederic Descamps
 
Cloud native - Why to use MySQL 8.0 and how to use it on oci with MDS
Frederic Descamps
 
MySQL Database Service Webinar: Installing Drupal in oci with mds
Frederic Descamps
 
the State of the Dolphin - October 2020
Frederic Descamps
 
MySQL 8.0 Document Store - Discovery of a New World
Frederic Descamps
 
Oracle Developer Live: Deploying MySQL InnoDB Cluster on OCI with Terraform
Frederic Descamps
 
Ad

Recently uploaded (20)

PDF
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PDF
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
PPTX
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PPTX
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
PDF
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
Mastering Financial Management in Direct Selling
Epixel MLM Software
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 

MySQL InnoDB Cluster - Advanced Configuration & Operations

  • 1. Pedro Gomes MySQL Senior Software Developer Frédéric Descamps MySQL Community Manager EMEA & APAC MySQL InnoDB Cluster Advanced Configuration & Operations   2 / 161
  • 2.   Safe Harbor The following is intended to outline our general product direction. It is intended for information purpose only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any features or functionality described for Oracle´s product may change and remains at the sole discretion of Oracle Corporation. Statement in this presentation relating to Oracle´s future plans, expectations, beliefs, intentions and ptospects are "forward-looking statements" and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that a ect our business is contained in Oracle´s Securities and Exchange Commission (SEC) lings, including our most recent reports on Form 10-K and Form 10-Q under the heading "Risk Factors". These lings are available on the SEC´s website or on Oracle´s website at h p://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Copyright @ 2019 Oracle and/or its affiliates. 3 / 161
  • 3. about us Who are we ? Copyright @ 2019 Oracle and/or its affiliates. 4 / 161
  • 4. Pedro Gomes @distributedpete MySQL Replication Developer MySQL Developer since 2012 Portuguese born and raised 🇵🇹 h ps://mysqlhighavailability.com Copyright @ 2019 Oracle and/or its affiliates. 5 / 161
  • 5. Frédéric Descamps @lefred MySQL Evangelist Managing MySQL since 3.23 devops believer living in Belgium 🇧🇪 h ps://lefred.be Copyright @ 2019 Oracle and/or its affiliates. 6 / 161
  • 6. MySQL InnoDB Cluster New Member Provisioning & Recovery Copyright @ 2019 Oracle and/or its affiliates. 7 / 161
  • 7. Provisioning VS Recovery Provisioning a server is the process of con guring and adding data to a server, which can then potentially be used in a replication topology. Typically, this involves restoring a backup or snapshot into a given instance. The distributed recovery is the protocol that runs when a server is added to a cluster, so that the new server nds a state donor, fetches missing state and synchronizes with the rest of the cluster automatically. As of MySQL 8.0.17, the distributed recovery protocol for Group Replication is able to automatically orchestrate provisioning, in addition to using binary logs, in the process of synchronizing the new instance with the rest of the group Copyright @ 2019 Oracle and/or its affiliates. 8 / 161
  • 8. How to Provision a New Member Incremental Recovery New node for a recent Cluster From a Backup Clone (NEW 8.0.17) Copyright @ 2019 Oracle and/or its affiliates. 9 / 161
  • 9. Incremental Recovery Based on the good old Asynchronous replication Fetches only the missing data based on the member GTIDs info Also provides the member with group transactional information Copyright @ 2019 Oracle and/or its affiliates. 10 / 161
  • 10. Based on the brand new Clone plugin Fetches a full snapshot of the donor data Replaces the server data and requires its restart Still requires incremental recovery for group transactional information   Clone Copyright @ 2019 Oracle and/or its affiliates. 11 / 161
  • 11. MySQL InnoDB Cluster Provisioning binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 12 / 161
  • 12. MySQL InnoDB Cluster Provisioning binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 13 / 161
  • 13. MySQL InnoDB Cluster Provisioning binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 14 / 161
  • 14. MySQL InnoDB Cluster Provisioning binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 15 / 161
  • 15. MySQL InnoDB Cluster Provisioning binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 16 / 161
  • 16. MySQL InnoDB Cluster Provisioning binlog binlog hello, I would like to join ! Copyright @ 2019 Oracle and/or its affiliates. 17 / 161
  • 17. MySQL InnoDB Cluster Provisioning binlog binlog I have no data yet ! Copyright @ 2019 Oracle and/or its affiliates. 18 / 161
  • 18. MySQL InnoDB Cluster Provisioning binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 19 / 161
  • 19. MySQL InnoDB Cluster Provisioning binlog binlog Option 1: Incremental donor joiner binlog Copyright @ 2019 Oracle and/or its affiliates. 20 / 161
  • 20. MySQL InnoDB Cluster Provisioning binlog binlog Option 1: Incremental donor joiner binlog Copyright @ 2019 Oracle and/or its affiliates. 21 / 161
  • 21. MySQL InnoDB Cluster Provisioning binlog binlog Option 1: Incremental donor joiner binlog Copyright @ 2019 Oracle and/or its affiliates. 22 / 161
  • 22. MySQL InnoDB Cluster Provisioning binlog binlog Option 1: Incremental binlog Copyright @ 2019 Oracle and/or its affiliates. 23 / 161
  • 23. MySQL InnoDB Cluster Provisioning Copyright @ 2019 Oracle and/or its affiliates. 24 / 161
  • 24. MySQL InnoDB Cluster Provisioning binlog binlog Option 2: Clone binlog donor joiner clone Copyright @ 2019 Oracle and/or its affiliates. 25 / 161
  • 25. MySQL InnoDB Cluster Provisioning binlog binlog Option 2: Clone binlog donor joiner clone Copyright @ 2019 Oracle and/or its affiliates. 26 / 161
  • 26. MySQL InnoDB Cluster Provisioning binlog binlog hello, I would like to join ! binlog Copyright @ 2019 Oracle and/or its affiliates. 27 / 161
  • 27. MySQL InnoDB Cluster Provisioning binlog binlog hello, I would like to join ! binlog And I have no data yet ! Copyright @ 2019 Oracle and/or its affiliates. 28 / 161
  • 28. MySQL InnoDB Cluster Provisioning binlog binlog We are not able to provide you incremental data ! Use CLONE ! binlog Copyright @ 2019 Oracle and/or its affiliates. 29 / 161
  • 29. MySQL InnoDB Cluster Provisioning binlog binlog donor joiner clone binlog Copyright @ 2019 Oracle and/or its affiliates. 30 / 161
  • 30. MySQL InnoDB Cluster Provisioning binlog binlog binlog Copyright @ 2019 Oracle and/or its affiliates. 31 / 161
  • 31. MySQL InnoDB Cluster Provisioning Copyright @ 2019 Oracle and/or its affiliates. 32 / 161
  • 32. binlog binlog binlog restored backup MySQL InnoDB Cluster Provisioning What kind of provisioning will happen ? Copyright @ 2019 Oracle and/or its affiliates. 33 / 161
  • 33. binlog binlog binlog restored backup   Incremental Recovery ! MySQL InnoDB Cluster Provisioning What kind of provisioning will happen ? Copyright @ 2019 Oracle and/or its affiliates. 34 / 161
  • 34. MySQL InnoDB Cluster Provisioning Copyright @ 2019 Oracle and/or its affiliates. 35 / 161
  • 35. MySQL InnoDB Cluster Provisioning Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to force the use of Clone when adding an instance even if Incremental Recovery is possible. 36 / 161
  • 36. MySQL InnoDB Cluster Provisioning Just specify it in the MySQL Shell: Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to force the use of Clone when adding an instance even if Incremental Recovery is possible. 37 / 161
  • 37. Recovery Process The recovery process happens when a previous member joins back a Cluster/Group. By default and if possible (based on GTID set), the Incremental Recovery method is used. However, sometimes applying a large amount of binary log events might be slower than performing a Clone. You have the possibility to con gure the treshold: group_replication_clone_threshold (default is 9223372036854775807 !): Copyright @ 2019 Oracle and/or its affiliates. 38 / 161
  • 38. Recovery Process (2) Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to force the use of Clone when rejoining an instance even if Incremental Recovery is enough to recover. 39 / 161
  • 39. Recovery Process (2) It's possible ! You need to connect to the instance you want to rejoin and set group_replication_clone_threshold to 1. As Clone will be called and the change is not persisted, no need to set it back to its original value. Never persist it to 1 or to a really low value on a live Group, never! Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to force the use of Clone when rejoining an instance even if Incremental Recovery is enough to recover. 40 / 161
  • 40. Recovery Process (3) Copyright @ 2019 Oracle and/or its affiliates. 41 / 161
  • 41. Force a Donor during Clone Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to add a node (mysql3) but it should only clone from server mysql2 42 / 161
  • 42. Force a Donor during Clone This is again possible, you need to connect on the other members (in this case mysql1 and unload the clone plugin). Please note, that to be able to unload the plugin you need to be connected using the classic protocol (3306). Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to add a node (mysql3) but it should only clone from server mysql2 43 / 161
  • 43. We connect using the classic protocol and we uninstall the plugin: Force a Donor during Clone (2) Copyright @ 2019 Oracle and/or its affiliates. 44 / 161
  • 44. We connect using the classic protocol and we uninstall the plugin: Then we add the instance forcing Clone: Force a Donor during Clone (2) Copyright @ 2019 Oracle and/or its affiliates. 45 / 161
  • 45. Force a Donor during Clone (3) Copyright @ 2019 Oracle and/or its affiliates. 46 / 161
  • 46. Force a Donor during Clone (4) We can also verify this on the new joiner Performance_Schema.clone_status table: Copyright @ 2019 Oracle and/or its affiliates. 47 / 161
  • 47. Force a Donor during Clone (5) Don't forget to install the Clone plugin again: Copyright @ 2019 Oracle and/or its affiliates. 48 / 161
  • 48. Force a Donor during Clone (5) Don't forget to install the Clone plugin again: Copyright @ 2019 Oracle and/or its affiliates. 49 / 161
  • 49. MySQL InnoDB Cluster Monitoring Copyright @ 2019 Oracle and/or its affiliates. 50 / 161
  • 50. Monitoring Provisioning & Recovery process Currently, when the CLONE is triggered within MySQL Shell, you can follow its progress: Copyright @ 2019 Oracle and/or its affiliates. 51 / 161
  • 51. Monitoring Provisioning & Recovery process And we already saw that the status and the progress is also available in performance_schema tables on the joiner: Copyright @ 2019 Oracle and/or its affiliates. 52 / 161
  • 52. Monitoring Provisioning & Recovery process Copyright @ 2019 Oracle and/or its affiliates. 53 / 161
  • 53. Incremental Recovery process It's also possible to evaluate the amount of transactions a member needs to process before being in sync with the Group when joining. You can nd a User-De ned Report to see that amount : More Info: h ps://lefred.be/content/mysql-innodb-cluster-recovery-process-monitoring- with-the-mysql-shell-reporting-framework/ Copyright @ 2019 Oracle and/or its affiliates. 54 / 161
  • 54. By default the status() method disables the extended mode. It's possible to have much more information enabling it. MySQL InnoDB Cluster - status() Copyright @ 2019 Oracle and/or its affiliates. 55 / 161
  • 55. MySQL InnoDB Cluster - status({extended}) The extended option allows 4 values (or boolean): 0: disables the command verbosity (default) 1: includes information about the Group Protocol Version, Group name, cluster member UUIDs, cluster member roles and states as reported by Group Replication and the list of fenced system variables 2: includes information about transactions processed by connection and applier 3: includes more detailed stats about the replication machinery of each cluster member Copyright @ 2019 Oracle and/or its affiliates. 56 / 161
  • 56. MySQL InnoDB Cluster - status({extended:1}) Copyright @ 2019 Oracle and/or its affiliates. 57 / 161
  • 57. MySQL InnoDB Cluster - status({extended:2}) Copyright @ 2019 Oracle and/or its affiliates. 58 / 161
  • 58. MySQL InnoDB Cluster - status({extended:3}) Copyright @ 2019 Oracle and/or its affiliates. 59 / 161
  • 59. Monitoring MySQL InnoDB Cluster - Performance_Schema Other than the status() in the MySQL Shell, it's possible to get information from Performance_Schema tables: Copyright @ 2019 Oracle and/or its affiliates. 60 / 161
  • 60. Monitoring MySQL InnoDB Cluster - Performance_Schema (2) Copyright @ 2019 Oracle and/or its affiliates. 61 / 161
  • 61. Monitoring MySQL InnoDB Cluster - Performance_Schema (3) Copyright @ 2019 Oracle and/or its affiliates. 62 / 161
  • 62. Monitoring MySQL InnoDB Cluster - Performance_Schema (4) Copyright @ 2019 Oracle and/or its affiliates. 63 / 161
  • 63. Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I would like to know which of my Secondary-Master is slower ? 64 / 161
  • 64. Performance_Schema provides to us all what we need to discover which Secondary-Slave takes more time to "fully replicate" events. Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I would like to know which of my Secondary-Master is slower ? 65 / 161
  • 65. Let's run this bash code on all our Secondary-Masters (mysql1 and mysql2): for i in $(seq 1 10) do if [[ $i -eq 1 ]]; then echo "repl delay transport time time to RL apply time" mysql -BN -e " SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP 'rep delay (sec)', LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP 'transport time', LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP 'time RL', LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP 'apply time' FROM performance_schema.replication_applier_status_by_worker t1 JOIN performance_schema.replication_connection_status t2 ON t2.channel_name=t1.channel_name WHERE t1.channel_name='group_replication_applier'"; sleep 2 done Copyright @ 2019 Oracle and/or its affiliates. 66 / 161
  • 66. Let's run this bash code on all our Secondary-Masters (mysql1 and mysql2): for i in $(seq 1 10) do if [[ $i -eq 1 ]]; then echo "repl delay transport time time to RL apply time" mysql -BN -e " SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP 'rep delay (sec)', LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP 'transport time', LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP 'time RL', LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP 'apply time' FROM performance_schema.replication_applier_status_by_worker t1 JOIN performance_schema.replication_connection_status t2 ON t2.channel_name=t1.channel_name WHERE t1.channel_name='group_replication_applier'"; sleep 2 done This script will query some performance_schema tables 10 times every 2 seconds (we write on our cluster every 2 secs), and all members have their time in sync: ntpd. Copyright @ 2019 Oracle and/or its affiliates. 67 / 161
  • 67. Monitoring MySQL InnoDB Cluster - Performance_Schema (5)   Copyright @ 2019 Oracle and/or its affiliates. 68 / 161
  • 68. Monitoring MySQL InnoDB Cluster - Performance_Schema (5)   Copyright @ 2019 Oracle and/or its affiliates. 69 / 161
  • 69. Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I would like to estimate how many transactions per second could my cluster process with the current configuration. 70 / 161
  • 70. To perform this estimation, we will block all writes to a member of our cluster. As a result, the apply queue will grow on that server. When the queue is large enough, we will enable writes again on the node and see how many transactions and how much time the node needed to process everything and be back in sync with the cluster. Currently, the application is performing 100 tx/sec. Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I would like to estimate how many transactions per second could my cluster process with the current configuration. 71 / 161
  • 71. On the node that we will lock, let's run this small bash command: while true do mysql -BNe "select now(), transactions_behind, count_transactions_remote_applied from sys.gr_member_routing_candidate_status, performance_schema.replication_group_member_stats where member_id=@@server_uuid;" sleep 1 done Copyright @ 2019 Oracle and/or its affiliates. 72 / 161
  • 72. On the node that we will lock, let's run this small bash command: while true do mysql -BNe "select now(), transactions_behind, count_transactions_remote_applied from sys.gr_member_routing_candidate_status, performance_schema.replication_group_member_stats where member_id=@@server_uuid;" sleep 1 done Copyright @ 2019 Oracle and/or its affiliates. 73 / 161
  • 73. Copyright @ 2019 Oracle and/or its affiliates. 74 / 161
  • 74. Copyright @ 2019 Oracle and/or its affiliates. 75 / 161
  • 75. So we can estimate that our MySQL InnoDB Cluster is able to process approximately 350 tx/sec. So what will happen if we increase our workload to 360 tx/sec ? Copyright @ 2019 Oracle and/or its affiliates. 76 / 161
  • 76.   Copyright @ 2019 Oracle and/or its affiliates. 77 / 161
  • 77. The cluster will start slowing down and this might have a major impact depending of the consistency level.   Copyright @ 2019 Oracle and/or its affiliates. 78 / 161
  • 78. More Info: h ps://lefred.be/content/using-the-new-mysql- shell-reporting-framework-to-monitor-innodb-cluster/ Monitoring MySQL InnoDB Cluster - MySQL Shell UDR It's also possible to constantly monitor the status of your MySQL InnoDB Cluster using User-De ned Reports in MySQL Shell: Copyright @ 2019 Oracle and/or its affiliates. 79 / 161
  • 80. Monitoring MySQL Router (2) You can then query the REST API using curl: $ curl -s -u fred:fred http://192.168.91.2:8080/api/20190715/routes/myCluster_default_rw/destinations {"items":[{"address":"mysql2","port":3306}]} Or use/create a plugin for the MySQL Shell. More Info: h ps://lefred.be/content/mysqlrouter-8-0-17-and-the-rest-api/ h ps://lefred.be/content/mysql-router-8-0-17s-rest-api-mysql-shell-extensions/ Copyright @ 2019 Oracle and/or its affiliates. 81 / 161
  • 81.   Monitoring MySQL Router (3) Copyright @ 2019 Oracle and/or its affiliates. 82 / 161
  • 82. Monitoring MySQL Router (4) Copyright @ 2019 Oracle and/or its affiliates. 83 / 161
  • 83. MySQL InnoDB Cluster Network Monitoring Copyright @ 2019 Oracle and/or its affiliates. 84 / 161
  • 84.   All good ;) Network monitoring - the basics Copyright @ 2019 Oracle and/or its affiliates. 85 / 161
  • 85.   When members crash or are expelled Network monitoring - the basics Copyright @ 2019 Oracle and/or its affiliates. 86 / 161
  • 86.   When a majority of the members becomes unreachable Network monitoring - the basics Copyright @ 2019 Oracle and/or its affiliates. 87 / 161
  • 87. Network monitoring - Adapt to faulty networks To avoid frequent member expels due to faulty networks make use of: group_replication_member_expel_timeout Copyright @ 2019 Oracle and/or its affiliates. 88 / 161
  • 88. Network monitoring - Deal with failure scenarios Con gure your rejoin a empts for each member (this method persists the change) Copyright @ 2019 Oracle and/or its affiliates. 89 / 161
  • 89. Network monitoring - Deal with failure scenarios Con gure your rejoin a empts for each member (this method persists the change) If you prefer to not persist it: Copyright @ 2019 Oracle and/or its affiliates. 90 / 161
  • 90. Network monitoring - Deal with failure scenarios Con gure your timeout during loss of quorum Copyright @ 2019 Oracle and/or its affiliates. 91 / 161
  • 91. Network monitoring - Deal with failure scenarios Be prepared: whom shall be the primary when failure strikes Copyright @ 2019 Oracle and/or its affiliates. 92 / 161
  • 92. Network monitoring - Adapt to higher latencies Copyright @ 2019 Oracle and/or its affiliates. 93 / 161
  • 93. Network monitoring - Adapt to higher latencies group replication write concurrency Copyright @ 2019 Oracle and/or its affiliates. 94 / 161
  • 94. Network monitoring - Adapt to higher latencies group replication write concurrency Copyright @ 2019 Oracle and/or its affiliates. 95 / 161
  • 95. Network monitoring - Adapt to higher latencies group replication write concurrency Copyright @ 2019 Oracle and/or its affiliates. 96 / 161
  • 96. Network monitoring - Adapt to higher latencies Copyright @ 2019 Oracle and/or its affiliates. 97 / 161
  • 97. Network monitoring - Adapt to higher latencies Copyright @ 2019 Oracle and/or its affiliates. 98 / 161
  • 98. Network monitoring - Adapt to higher latencies Copyright @ 2019 Oracle and/or its affiliates. 99 / 161
  • 99. Network monitoring - Adapt to higher latencies Copyright @ 2019 Oracle and/or its affiliates. 100 / 161
  • 100. Con gure your write concurrency in the group Network monitoring - Adapt to higher latencies   Copyright @ 2019 Oracle and/or its affiliates. 101 / 161
  • 101. MySQL InnoDB Cluster Operations Copyright @ 2019 Oracle and/or its affiliates. 102 / 161
  • 102. Upgrading to a newer version Before upgrading, don't forget to use util.upgradeChecker() in MySQL Shell. This operation must be performed on the Primary Master: Copyright @ 2019 Oracle and/or its affiliates. 103 / 161
  • 103. Upgrading Process PM SM SM 8.0.16 8.0.16 8.0.16 Copyright @ 2019 Oracle and/or its affiliates. 104 / 161
  • 104. Upgrading Process PM SM SM 8.0.16 8.0.16 8.0.16 util.checkForServerUpgrade( )PM Copyright @ 2019 Oracle and/or its affiliates. 105 / 161
  • 105. Upgrading Process PM SM 8.0.16 8.0.16 8.0.16 stop mysqld on a SM Copyright @ 2019 Oracle and/or its affiliates. 106 / 161
  • 106. Upgrading Process PM SM 8.0.16 8.0.16 8.0.17 upgrade MySQL binaries Copyright @ 2019 Oracle and/or its affiliates. 107 / 161
  • 107. Upgrading Process PM SM 8.0.16 8.0.16 8.0.17 start mysqld Copyright @ 2019 Oracle and/or its affiliates. 108 / 161
  • 108. Upgrading Process PM SM 8.0.16 8.0.16 8.0.17 join the group Copyright @ 2019 Oracle and/or its affiliates. 109 / 161
  • 109. Upgrading Process PM SM SM 8.0.16 8.0.16 8.0.17 Copyright @ 2019 Oracle and/or its affiliates. 110 / 161
  • 110. Upgrading Process PM SM 8.0.16 8.0.16 8.0.17 stop mysqld on the other SM Copyright @ 2019 Oracle and/or its affiliates. 111 / 161
  • 111. Upgrading Process PM SM 8.0.16 8.0.17 8.0.17 upgrade the MySQL binaries Copyright @ 2019 Oracle and/or its affiliates. 112 / 161
  • 112. Upgrading Process PM SM 8.0.16 8.0.17 8.0.17 start mysqld Copyright @ 2019 Oracle and/or its affiliates. 113 / 161
  • 113. Upgrading Process PM SM 8.0.16 8.0.17 8.0.17 join the group join the group Copyright @ 2019 Oracle and/or its affiliates. 114 / 161
  • 114. Upgrading Process PM SM SM 8.0.16 8.0.17 8.0.17 Copyright @ 2019 Oracle and/or its affiliates. 115 / 161
  • 115. Upgrading Process PM SM 8.0.16 8.0.17 8.0.17 stop mysqld on the PM Copyright @ 2019 Oracle and/or its affiliates. 116 / 161
  • 116. Upgrading Process PM SM 8.0.17 8.0.17 8.0.17 upgrade MySQL binaries Copyright @ 2019 Oracle and/or its affiliates. 117 / 161
  • 117. Upgrading Process PM SM 8.0.17 8.0.17 8.0.17 start myslqd Copyright @ 2019 Oracle and/or its affiliates. 118 / 161
  • 118. Upgrading Process PM SM 8.0.17 8.0.17 8.0.17 join the group Copyright @ 2019 Oracle and/or its affiliates. 119 / 161
  • 119. Upgrading Process PMSM SM 8.0.17 8.0.17 8.0.17 Copyright @ 2019 Oracle and/or its affiliates. 120 / 161
  • 120. Upgrading Process Info The lowest version of a Group must always be the Primary Master If ejected from the Group, the lowest version instance won't be able to rejoin the Group again Copyright @ 2019 Oracle and/or its affiliates. 121 / 161
  • 121. Downgrading Downgrading an existing instance won't work as downgrading MySQL Server is not supported: [ERROR] [MY-013171] [InnoDB] Cannot boot server version 80016 on data directory built by version 80017. Downgrade is not supported. However it's possible to downgrade your MySQL InnoDB Cluster without downtime, by using a previous backup. Before 8.0.17, any 8.0.x could join a MySQL InnoDB Cluster 8.0. Since 8.0.17, the minor version is also checked. Copyright @ 2019 Oracle and/or its affiliates. 122 / 161
  • 122. Downgrading (2) So if you want to have a lower version to join a cluster, you need to set group_replication_allow_local_lower_version_join to ON on the joiner. But don't forget that this instance running with a lower version should be promoted to be the Primary-Master as soon as possible to avoid any possible issues related to protocol and variable di erences. Copyright @ 2019 Oracle and/or its affiliates. 123 / 161
  • 123. Backup Taking a backup of your MySQL InnoDB Cluster is trivial. You can use exactly the same method as you are using for a standalone MySQL Server: Physical (MySQL Enterperise Backup (MEB), Xtrabackup) Logical (mysqldump, mysqlpump, mydumper) Snapshot (lvm, MySQL Clone) Copyright @ 2019 Oracle and/or its affiliates. 124 / 161
  • 124. Backup (2) Copyright @ 2019 Oracle and/or its affiliates. As a DBA using MySQL InnoDB Cluster retains data consistency, I only want to backup one member of the Group and preferably a Secondary Master, no need to waste disk space with multiple backups of the same data. 125 / 161
  • 125. Backup (2) This is possible ! Let's de ne the best backup solution. Copyright @ 2019 Oracle and/or its affiliates. As a DBA using MySQL InnoDB Cluster retains data consistency, I only want to backup one member of the Group and preferably a Secondary Master, no need to waste disk space with multiple backups of the same data. 126 / 161
  • 126. Backup (3) The perfect backup solution for you MySQL InnoDB Cluster, should then perform the following steps: check if the node where the script run is part of the cluster check if the node is indeed a secondary master eventually check if the node is lagging behind (large apply queue) ensure that the backup is not running on another member Copyright @ 2019 Oracle and/or its affiliates. 127 / 161
  • 127. Backup (3) The perfect backup solution for you MySQL InnoDB Cluster, should then perform the following steps: check if the node where the script run is part of the cluster check if the node is indeed a secondary master eventually check if the node is lagging behind (large apply queue) ensure that the backup is not running on another member Here, you can nd a working script calling MEB to use as a cron job : h ps://lefred.be/content/how-to-backup-your-innodb-cluster/ Copyright @ 2019 Oracle and/or its affiliates. 128 / 161
  • 128. DDLs There is nothing special about DDLs. When DDLs are performed, the cluster is not blocked completely. Of course, concurent DDLs and DMLs are not a good idea when using a cluster in Multi- Primary mode. If you use a Multi-Primary Cluster, you must swicth to Single-Primary mode the time of the DDL. Don't forget that MySQL 8.0 supports also Instant DDLs like adding a column. Copyright @ 2019 Oracle and/or its affiliates. 129 / 161
  • 129. DDLs on Multi-Primary Clusters Copyright @ 2019 Oracle and/or its affiliates. 130 / 161
  • 130. DDLs on Multi-Primary Clusters Copyright @ 2019 Oracle and/or its affiliates. 131 / 161
  • 131. Configurable Consistency Guarantees Consistency Levels Copyright @ 2019 Oracle and/or its affiliates. 132 / 161
  • 132. mysql> show variables like 'group_replication_consistency'; +-------------------------------+----------+ | Variable_name | Value | +-------------------------------+----------+ | group_replication_consistency | EVENTUAL | +-------------------------------+----------+   Consistency: EVENTUAL (default) By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read the same data on another node, it is eventually there. Copyright @ 2019 Oracle and/or its affiliates. 133 / 161
  • 133. mysql> show variables like 'group_replication_consistency'; +-------------------------------+----------+ | Variable_name | Value | +-------------------------------+----------+ | group_replication_consistency | EVENTUAL | +-------------------------------+----------+   Consistency: EVENTUAL (default) By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read the same data on another node, it is eventually there. Since MySQL 8.0.16, we have the possibility to set the synchronization point at read or at write or both (globally or for a session). Copyright @ 2019 Oracle and/or its affiliates. 134 / 161
  • 134. Consistency: BEFORE (READ) Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to load balance my reads without deploying additional restrictions on which server I read from to avoid reading stale data, my group writes are much more than my group reads. 135 / 161
  • 135. Consistency: BEFORE (READ) Copyright @ 2019 Oracle and/or its affiliates. As a DBA, I want to load balance my reads without deploying additional restrictions on which server I read from to avoid reading stale data, my group writes are much more than my group reads. As a developer, I want specific transactions in my workload to always read up- to-date data from the group, so that whenever that sensitive data is updated, I will enforce that reads shall read the most up to date value. 136 / 161
  • 136. Consistency: BEFORE (READ) Copyright @ 2019 Oracle and/or its affiliates. 137 / 161
  • 137. Consistency: AFTER (WRITE) Copyright @ 2019 Oracle and/or its affiliates. I want to load balance my reads without deploying additional restrictions on which server I read from to avoid reading stale data, my group writes are much less than my group reads. 138 / 161
  • 138. Consistency: AFTER (WRITE) Copyright @ 2019 Oracle and/or its affiliates. I want to load balance my reads without deploying additional restrictions on which server I read from to avoid reading stale data, my group writes are much less than my group reads. I have a group that mostly does reads-only, I want my read-write transactions to be applied everywhere once they commit, so that subsequent reads are done on up-to-date data that includes my latest write. Without paying at reads. 139 / 161
  • 139. Consistency: AFTER (WRITE) Copyright @ 2019 Oracle and/or its affiliates. 140 / 161
  • 140. Consistency: BEFORE_AND_AFTER Copyright @ 2019 Oracle and/or its affiliates. I want that my application to replicate data as close as possible to synchronous. And of course I'm OK to pay the required price ! 141 / 161
  • 141. Consistency: BEFORE_AND_AFTER Copyright @ 2019 Oracle and/or its affiliates. 142 / 161
  • 142. Defining Consistency The consistency level of the MySQL InnoDB Cluster can be de ned during the creation: Copyright @ 2019 Oracle and/or its affiliates. 143 / 161
  • 143. Defining Consistency (2) The consistency can be globally changed in MySQL Shell: It's also possible and recommended for the most restrictive levels to de ne it at statement session level: Copyright @ 2019 Oracle and/or its affiliates. 144 / 161
  • 144. MySQL InnoDB Cluster MySQL Router Specific Settings Copyright @ 2019 Oracle and/or its affiliates. 145 / 161
  • 145. MySQL Router : Metadata Cache The se ings to access MySQL InnoDB Cluster's Metadata Cache is de ned in the [metadata_cache:<name>] section of MySQL Router's con g le. Copyright @ 2019 Oracle and/or its affiliates. 146 / 161
  • 146. MySQL Router : Metadata Cache The se ings to access MySQL InnoDB Cluster's Metadata Cache is de ned in the [metadata_cache:<name>] section of MySQL Router's con g le. The most interesting se ings is: ttl: Time to live (in seconds) of information in the metadata cache. One milissecond is 0.001. To continously query the the metadata cache in a tight loop, set it to 0 (the default value is 0.5). Copyright @ 2019 Oracle and/or its affiliates. 147 / 161
  • 147. MySQL Router : Metadata Cache (2) As of MySQL Router 8.0.17, ttl is now obsolete as you can enable use_gr_noti cations to refresh metadata on each of the four noti cations that Group Replication sends: group_replication/membership/quorum_loss group_replication/membership/view group_replication/status/role_change group_replication/status/state_change Copyright @ 2019 Oracle and/or its affiliates. 148 / 161
  • 148. MySQL Router By default, when bootstrapped, MySQL Router is con gured using rst-available as routing strategy for R/W connections (to the Primary-Master). For R/O connections (to Secondary-Masters), the strategy is round-robin-with-fallback. This is actually setup for the MySQL Classic Protocol and MySQL X Protocol. Copyright @ 2019 Oracle and/or its affiliates. 149 / 161
  • 149. MySQL Router : Routing Strategy Methods rst-available: the new connection is routed to the rst available server from the destinations list. In case of failure, the next available server is used. This cycle continues until all servers are unavailable. round-robin-with-fallback: for load-balancing, each new connection is made to the next available secondary server in a round-robin fashion. If a secondary server is not available then servers from the primary list are used in round-robin fashion. Copyright @ 2019 Oracle and/or its affiliates. 150 / 161
  • 150. MySQL Router : Routing Strategy Methods round-robin: for load-balancing, each new connection is made to the next available server in a round-robin fashion. next-available: like rst-available, in that the new connection is routed to the rst available server from the destinations list. Unlike rst-available, if a server is marked as unreachable then it gets discarded and is never used again as a destination Copyright @ 2019 Oracle and/or its affiliates. 151 / 161
  • 151. MySQL Router : Routing destinations Now that we have de ned a strategy for the MySQL Router, we also need to de ne destinations. Destinations (desinations in [routing:<name>] section), can be a comma-separated list of MySQL Servers, or a metadata-cache de nition: Copyright @ 2019 Oracle and/or its affiliates. 152 / 161
  • 152. MySQL Router : Routing destinations Now that we have de ned a strategy for the MySQL Router, we also need to de ne destinations. Destinations (desinations in [routing:<name>] section), can be a comma-separated list of MySQL Servers, or a metadata-cache de nition: [routing:PerconaLiveEU_default_rw] bind_address=0.0.0.0 bind_port=6446 destinations=metadata-cache://PerconaLiveEU/default?role=PRIMARY routing_strategy= rst-available protocol=classic Copyright @ 2019 Oracle and/or its affiliates. 153 / 161
  • 153. MySQL InnoDB Cluster Configuration & Settings Copyright @ 2019 Oracle and/or its affiliates. 154 / 161
  • 154. Instance Specific Settings Within the MySQL Shell you can de ne some Group Replication speci c se ings via the Admin API that will only a ect the targeted instance: exitStateAction: string value indicating the group replication exit state action. (ABORT_SERVER, READ_ONLY and OFFLINE_MODE [8.0.18]) memberWeight: integer value with a percentage weight for automatic primary election on failover. autoRejoinTries: integer value to de ne the number of times an instance will a empt to rejoin the cluster after being expelled. Copyright @ 2019 Oracle and/or its affiliates. 155 / 161
  • 155. Instance Specific Settings (2) label: a string identi er of the instance. Copyright @ 2019 Oracle and/or its affiliates. 156 / 161
  • 156. Group Specific Settings Within the MySQL Shell you have the possibility to de ne some Group speci c se ings via the Admin API: clusterName: string value to de ne the cluster name. exitStateAction: string value indicating the group replication exit state action. memberWeight: integer value with a percentage weight for automatic primary election on failover. consistency: string value indicating the consistency guarantees that the cluster provides. Copyright @ 2019 Oracle and/or its affiliates. 157 / 161
  • 157. Group Specific Settings (2) expelTimeout: integer value to de ne the time period in seconds that cluster members should wait for a non-responding member before evicting it from the cluster. autoRejoinTries: integer value to de ne the number of times an instance will a empt to rejoin the cluster after being expelled. disableClone: boolean value used to disable the clone usage on the cluster. Copyright @ 2019 Oracle and/or its affiliates. 158 / 161
  • 158. Upgrade to MySQL 8.0 It's time to upgrade to MySQL 8.0, the fastest MySQL adoption release ever ! Copyright @ 2019 Oracle and/or its affiliates. 159 / 161
  • 159. Copyright @ 2019 Oracle and/or its affiliates. 160 / 161
  • 160. Thank you ! Copyright @ 2019 Oracle and/or its affiliates. 161 / 161