SlideShare a Scribd company logo
MySQL Group Replication: Handling Network Glitches - Best Practices
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Group Replication: Handling Network Glitches
Best Practices
Pedro Gomes
Software Developer
MySQL Replication Team Censored Dolphins
January 30, 2020
Confidential – Oracle Internal/Restricted/Highly Restricted
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a 
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
Confidential – Oracle Internal/Restricted/Highly Restricted 3
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
●@distributedpete
●MySQL Replication Developer
●MySQL Developer since 2012
●Portuguese born and raised 🇵🇹
Pedro Gomes
Confidential – Oracle Internal/Restricted/Highly Restricted 4
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
What you can expect
Checking the status
Diagnostics
Adapt
Deal with Failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 5
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Diagnostics
Adapt
Deal with Failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 6
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
• Where to look?
– The usual suspect: Performance_Schema
• MySQL Shell is your friend
The first step in solving an issue is knowing what the issue is
Confidential – Oracle Internal/Restricted/Highly Restricted 7
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Performance Schema
Confidential – Oracle Internal/Restricted/Highly Restricted 8
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Performance Schema
Confidential – Oracle Internal/Restricted/Highly Restricted 9
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
Performance Schema
Confidential – Oracle Internal/Restricted/Highly Restricted 10
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
MySQL Shell
Confidential – Oracle Internal/Restricted/Highly Restricted 11
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Checking the status
MySQL Shell – extended information
Confidential – Oracle Internal/Restricted/Highly Restricted 12
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Check the status
Diagnostics
Adapt
Deal with failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 13
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
• Identify the scenario
• Identify the cause
What do I do with this diagnostic data?
Confidential – Oracle Internal/Restricted/Highly Restricted 14
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Easily identified scenarios
Confidential – Oracle Internal/Restricted/Highly Restricted 15
• Loosing a single server
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Easily identified
scenarios
Confidential – Oracle Internal/Restricted/Highly Restricted 16
• No Quorum
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
• There is a delay or network latency fluctuations on the secondaries
– What stats can I check?
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 17
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
• From replication_applier_status_by_worker and replication_connection_status
• Replication delay - 'rep delay (sec)':
– SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP -
LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
• Transport time - 'transport time':
– LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP -
LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP
• Time to queue on the relay log - 'time RL' :
– LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP -
LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP
• Time for application - 'apply time' :
– LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP -
LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 18
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 19
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics
Evaluating delays
Confidential – Oracle Internal/Restricted/Highly Restricted 20
• You can code this as a shell plugin
• Check it at https://github.com/lefred/mysqlshell-plugins
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Diagnostics (non network issues)
• Can your workload be executed in parallel?
– Check if your parallel applier workers are idle
• Can the group handle the load?
– COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE
– COUNT_TRANSACTIONS_IN_QUEUE
What if I see no delays?
Confidential – Oracle Internal/Restricted/Highly Restricted 21
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Check the status
Diagnostics
Adapt
Deal with the failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 22
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
• To avoid frequent member expels due to faulty networks make use of:
– group_replication_member_expel_timeout
Adapt to faulty networks
Confidential – Oracle Internal/Restricted/Highly Restricted 23
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 24
group replication write concurrency
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 25
group replication write concurrency
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 26
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 27
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 28
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Adapt
Adapt to higher latencies
Confidential – Oracle Internal/Restricted/Highly Restricted 29
• Also be aware of big transactions:
– group_replication_communication_max_message_size
●
●
●
●
Alternative strategy
– group_replication_transaction_size_limit
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Check the status
Diagnostics
Adapt
Deal with failures
1
2
3
4
Confidential – Oracle Internal/Restricted/Highly Restricted 30
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• Configure your rejoin attempts for each member
– group_replication_autorejoin_tries
• If you don’t want to persist it
Retry retry retry
Confidential – Oracle Internal/Restricted/Highly Restricted 31
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• Configure whom shall the next primary be
Whom shall be next
Confidential – Oracle Internal/Restricted/Highly Restricted 32
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• Configure your timeout when quorum is not achievable
Prevent other members from rising from the dead
Confidential – Oracle Internal/Restricted/Highly Restricted 33
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Deal with failures
• What is the behavior when a member goes to error?
– Read only mode : no writes, can still read from it
– Offline mode: no reads or writes, can still access the server with
admin privileges (CONNECTION_ADMIN or SUPER)
– Abort server: the name is self explanatory
Update your exit state action
Confidential – Oracle Internal/Restricted/Highly Restricted 34
Copyright © 2020, Oracle and/or its affiliates. All rights reserved. |
Thank you!
Questions?
Confidential – Oracle Internal/Restricted/Highly Restricted 35

More Related Content

What's hot (20)

PDF
InnoDb Vs NDB Cluster
Mark Swarbrick
 
PDF
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
PDF
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
PDF
MySQL InnoDB Cluster and Group Replication in a nutshell hands-on tutorial
Frederic Descamps
 
PDF
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
Miguel Araújo
 
PDF
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
Severalnines
 
PDF
How to operate MySQL InnoDB Cluster with MySQL Shell
Frederic Descamps
 
PDF
MySQL Database Architectures - 2020-10
Kenny Gryp
 
PDF
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
Jesmar Cannao'
 
PDF
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
ODP
MySQL Group Replication
Ulf Wendel
 
PDF
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
PDF
MySQL Group Replication
Kenny Gryp
 
PDF
Open Source 101 2022 - MySQL Indexes and Histograms
Frederic Descamps
 
PDF
MySQL 8.0で憶えておいてほしいこと
yoku0825
 
PDF
binary log と 2PC と Group Commit
Takanori Sejima
 
PDF
MariaDB MaxScale
MariaDB plc
 
PDF
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
Continuent
 
PDF
Introduction to MySQL InnoDB Cluster
Frederic Descamps
 
PPTX
My sql failover test using orchestrator
YoungHeon (Roy) Kim
 
InnoDb Vs NDB Cluster
Mark Swarbrick
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
MySQL InnoDB Cluster and Group Replication in a nutshell hands-on tutorial
Frederic Descamps
 
Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?
Miguel Araújo
 
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...
Severalnines
 
How to operate MySQL InnoDB Cluster with MySQL Shell
Frederic Descamps
 
MySQL Database Architectures - 2020-10
Kenny Gryp
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
Jesmar Cannao'
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
Olivier DASINI
 
MySQL Group Replication
Ulf Wendel
 
Percona Live 2022 - MySQL Architectures
Frederic Descamps
 
MySQL Group Replication
Kenny Gryp
 
Open Source 101 2022 - MySQL Indexes and Histograms
Frederic Descamps
 
MySQL 8.0で憶えておいてほしいこと
yoku0825
 
binary log と 2PC と Group Commit
Takanori Sejima
 
MariaDB MaxScale
MariaDB plc
 
MySQL High Availability and Disaster Recovery with Continuent, a VMware company
Continuent
 
Introduction to MySQL InnoDB Cluster
Frederic Descamps
 
My sql failover test using orchestrator
YoungHeon (Roy) Kim
 

Similar to MySQL Group Replication: Handling Network Glitches - Best Practices (20)

PDF
Confoo 202 - MySQL Group Replication and ReplicaSet
Dave Stokes
 
PPTX
Fatkulin presentation
Enkitec
 
PDF
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
Markus Michalewicz
 
PDF
MySQL Group Replication - an Overview
Matt Lord
 
PDF
Everything You Need to Know About MySQL Group Replication
Nuno Carvalho
 
PDF
Database failover from client perspective
Priit Piipuu
 
PDF
20190817 coscup-oracle my sql innodb cluster sharing
Ivan Ma
 
PDF
MySQL Database Architectures - High Availability and Disaster Recovery Solution
Miguel Araújo
 
PPTX
Finding an unusual cause of max_user_connections in MySQL
Olivier Doucet
 
PDF
Why MySQL Replication Fails, and How to Get it Back
Sveta Smirnova
 
PDF
InnoDB Cluster Experience (MySQL User Camp)
Mydbops
 
PDF
Customer Experience: InnoDB Cluster Implementation by PR Karthik
Mysql User Camp
 
PDF
New Generation Oracle RAC Performance
Anil Nair
 
PDF
MySQL Database Architectures - 2022-08
Kenny Gryp
 
PDF
20191001 bkk-secret-of inno-db_clusterv1
Ivan Ma
 
PDF
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
Sujatha Sivakumar
 
PPT
Oracle Wait Events That Everyone Should Know.ppt
TricantinoLopezPerez
 
PPTX
Enterprise grade deployment and security with PostgreSQL
Himanchali -
 
PDF
Deep Dive into MySQL InnoDB Cluster Read Scale-out Capabilities.pdf
Miguel Araújo
 
PDF
High performance Infrastructure Oct 2013
Server Density
 
Confoo 202 - MySQL Group Replication and ReplicaSet
Dave Stokes
 
Fatkulin presentation
Enkitec
 
The Top 5 Reasons to Deploy Your Applications on Oracle RAC
Markus Michalewicz
 
MySQL Group Replication - an Overview
Matt Lord
 
Everything You Need to Know About MySQL Group Replication
Nuno Carvalho
 
Database failover from client perspective
Priit Piipuu
 
20190817 coscup-oracle my sql innodb cluster sharing
Ivan Ma
 
MySQL Database Architectures - High Availability and Disaster Recovery Solution
Miguel Araújo
 
Finding an unusual cause of max_user_connections in MySQL
Olivier Doucet
 
Why MySQL Replication Fails, and How to Get it Back
Sveta Smirnova
 
InnoDB Cluster Experience (MySQL User Camp)
Mydbops
 
Customer Experience: InnoDB Cluster Implementation by PR Karthik
Mysql User Camp
 
New Generation Oracle RAC Performance
Anil Nair
 
MySQL Database Architectures - 2022-08
Kenny Gryp
 
20191001 bkk-secret-of inno-db_clusterv1
Ivan Ma
 
MySQL InnoDB Cluster and Group Replication - OSI 2017 Bangalore
Sujatha Sivakumar
 
Oracle Wait Events That Everyone Should Know.ppt
TricantinoLopezPerez
 
Enterprise grade deployment and security with PostgreSQL
Himanchali -
 
Deep Dive into MySQL InnoDB Cluster Read Scale-out Capabilities.pdf
Miguel Araújo
 
High performance Infrastructure Oct 2013
Server Density
 
Ad

More from Frederic Descamps (20)

PDF
MySQL Innovation & Cloud Day - Document Store avec MySQL HeatWave Database Se...
Frederic Descamps
 
PDF
MySQL Day Roma - MySQL Shell and Visual Studio Code Extension
Frederic Descamps
 
PDF
RivieraJUG - MySQL Indexes and Histograms
Frederic Descamps
 
PDF
RivieraJUG - MySQL 8.0 - What's new for developers.pdf
Frederic Descamps
 
PDF
MySQL User Group NL - MySQL 8
Frederic Descamps
 
PDF
State of the Dolphin - May 2022
Frederic Descamps
 
PDF
Percona Live 2022 - MySQL Shell for Visual Studio Code
Frederic Descamps
 
PDF
Percona Live 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
PDF
LinuxFest Northwest 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
PDF
Pi Day 2022 - from IoT to MySQL HeatWave Database Service
Frederic Descamps
 
PDF
Confoo 2022 - le cycle d'une instance MySQL
Frederic Descamps
 
PDF
FOSDEM 2022 MySQL Devroom: MySQL 8.0 - Logical Backups, Snapshots and Point-...
Frederic Descamps
 
PDF
Les nouveautés de MySQL 8.0
Frederic Descamps
 
PDF
Les nouveautés de MySQL 8.0
Frederic Descamps
 
PDF
State of The Dolphin - May 2021
Frederic Descamps
 
PDF
Deploying Magento on OCI with MDS
Frederic Descamps
 
PDF
MySQL Router REST API
Frederic Descamps
 
PDF
From single MySQL instance to High Availability: the journey to MySQL InnoDB ...
Frederic Descamps
 
PDF
Cloud native - Why to use MySQL 8.0 and how to use it on oci with MDS
Frederic Descamps
 
PDF
MySQL Database Service Webinar: Installing Drupal in oci with mds
Frederic Descamps
 
MySQL Innovation & Cloud Day - Document Store avec MySQL HeatWave Database Se...
Frederic Descamps
 
MySQL Day Roma - MySQL Shell and Visual Studio Code Extension
Frederic Descamps
 
RivieraJUG - MySQL Indexes and Histograms
Frederic Descamps
 
RivieraJUG - MySQL 8.0 - What's new for developers.pdf
Frederic Descamps
 
MySQL User Group NL - MySQL 8
Frederic Descamps
 
State of the Dolphin - May 2022
Frederic Descamps
 
Percona Live 2022 - MySQL Shell for Visual Studio Code
Frederic Descamps
 
Percona Live 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
LinuxFest Northwest 2022 - The Evolution of a MySQL Database System
Frederic Descamps
 
Pi Day 2022 - from IoT to MySQL HeatWave Database Service
Frederic Descamps
 
Confoo 2022 - le cycle d'une instance MySQL
Frederic Descamps
 
FOSDEM 2022 MySQL Devroom: MySQL 8.0 - Logical Backups, Snapshots and Point-...
Frederic Descamps
 
Les nouveautés de MySQL 8.0
Frederic Descamps
 
Les nouveautés de MySQL 8.0
Frederic Descamps
 
State of The Dolphin - May 2021
Frederic Descamps
 
Deploying Magento on OCI with MDS
Frederic Descamps
 
MySQL Router REST API
Frederic Descamps
 
From single MySQL instance to High Availability: the journey to MySQL InnoDB ...
Frederic Descamps
 
Cloud native - Why to use MySQL 8.0 and how to use it on oci with MDS
Frederic Descamps
 
MySQL Database Service Webinar: Installing Drupal in oci with mds
Frederic Descamps
 
Ad

Recently uploaded (20)

PDF
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PPTX
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
Automating Feature Enrichment and Station Creation in Natural Gas Utility Net...
Safe Software
 
PDF
NLJUG Speaker academy 2025 - first session
Bert Jan Schrijver
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
DOCX
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
Automating Feature Enrichment and Station Creation in Natural Gas Utility Net...
Safe Software
 
NLJUG Speaker academy 2025 - first session
Bert Jan Schrijver
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 

MySQL Group Replication: Handling Network Glitches - Best Practices

  • 2. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Group Replication: Handling Network Glitches Best Practices Pedro Gomes Software Developer MySQL Replication Team Censored Dolphins January 30, 2020 Confidential – Oracle Internal/Restricted/Highly Restricted
  • 3. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a  commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Confidential – Oracle Internal/Restricted/Highly Restricted 3
  • 4. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | ●@distributedpete ●MySQL Replication Developer ●MySQL Developer since 2012 ●Portuguese born and raised 🇵🇹 Pedro Gomes Confidential – Oracle Internal/Restricted/Highly Restricted 4
  • 5. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | What you can expect Checking the status Diagnostics Adapt Deal with Failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 5
  • 6. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Diagnostics Adapt Deal with Failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 6
  • 7. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status • Where to look? – The usual suspect: Performance_Schema • MySQL Shell is your friend The first step in solving an issue is knowing what the issue is Confidential – Oracle Internal/Restricted/Highly Restricted 7
  • 8. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Performance Schema Confidential – Oracle Internal/Restricted/Highly Restricted 8
  • 9. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Performance Schema Confidential – Oracle Internal/Restricted/Highly Restricted 9
  • 10. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status Performance Schema Confidential – Oracle Internal/Restricted/Highly Restricted 10
  • 11. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status MySQL Shell Confidential – Oracle Internal/Restricted/Highly Restricted 11
  • 12. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Checking the status MySQL Shell – extended information Confidential – Oracle Internal/Restricted/Highly Restricted 12
  • 13. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Check the status Diagnostics Adapt Deal with failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 13
  • 14. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics • Identify the scenario • Identify the cause What do I do with this diagnostic data? Confidential – Oracle Internal/Restricted/Highly Restricted 14
  • 15. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Easily identified scenarios Confidential – Oracle Internal/Restricted/Highly Restricted 15 • Loosing a single server
  • 16. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Easily identified scenarios Confidential – Oracle Internal/Restricted/Highly Restricted 16 • No Quorum
  • 17. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics • There is a delay or network latency fluctuations on the secondaries – What stats can I check? Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 17
  • 18. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics • From replication_applier_status_by_worker and replication_connection_status • Replication delay - 'rep delay (sec)': – SELECT LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP • Transport time - 'transport time': – LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_ORIGINAL_COMMIT_TIMESTAMP • Time to queue on the relay log - 'time RL' : – LAST_QUEUED_TRANSACTION_END_QUEUE_TIMESTAMP - LAST_QUEUED_TRANSACTION_START_QUEUE_TIMESTAMP • Time for application - 'apply time' : – LAST_APPLIED_TRANSACTION_END_APPLY_TIMESTAMP - LAST_APPLIED_TRANSACTION_START_APPLY_TIMESTAMP Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 18
  • 19. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 19
  • 20. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics Evaluating delays Confidential – Oracle Internal/Restricted/Highly Restricted 20 • You can code this as a shell plugin • Check it at https://github.com/lefred/mysqlshell-plugins
  • 21. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Diagnostics (non network issues) • Can your workload be executed in parallel? – Check if your parallel applier workers are idle • Can the group handle the load? – COUNT_TRANSACTIONS_REMOTE_IN_APPLIER_QUEUE – COUNT_TRANSACTIONS_IN_QUEUE What if I see no delays? Confidential – Oracle Internal/Restricted/Highly Restricted 21
  • 22. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Check the status Diagnostics Adapt Deal with the failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 22
  • 23. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt • To avoid frequent member expels due to faulty networks make use of: – group_replication_member_expel_timeout Adapt to faulty networks Confidential – Oracle Internal/Restricted/Highly Restricted 23
  • 24. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 24 group replication write concurrency
  • 25. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 25 group replication write concurrency
  • 26. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 26
  • 27. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 27
  • 28. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 28
  • 29. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Adapt Adapt to higher latencies Confidential – Oracle Internal/Restricted/Highly Restricted 29 • Also be aware of big transactions: – group_replication_communication_max_message_size ● ● ● ● Alternative strategy – group_replication_transaction_size_limit
  • 30. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Check the status Diagnostics Adapt Deal with failures 1 2 3 4 Confidential – Oracle Internal/Restricted/Highly Restricted 30
  • 31. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • Configure your rejoin attempts for each member – group_replication_autorejoin_tries • If you don’t want to persist it Retry retry retry Confidential – Oracle Internal/Restricted/Highly Restricted 31
  • 32. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • Configure whom shall the next primary be Whom shall be next Confidential – Oracle Internal/Restricted/Highly Restricted 32
  • 33. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • Configure your timeout when quorum is not achievable Prevent other members from rising from the dead Confidential – Oracle Internal/Restricted/Highly Restricted 33
  • 34. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Deal with failures • What is the behavior when a member goes to error? – Read only mode : no writes, can still read from it – Offline mode: no reads or writes, can still access the server with admin privileges (CONNECTION_ADMIN or SUPER) – Abort server: the name is self explanatory Update your exit state action Confidential – Oracle Internal/Restricted/Highly Restricted 34
  • 35. Copyright © 2020, Oracle and/or its affiliates. All rights reserved. | Thank you! Questions? Confidential – Oracle Internal/Restricted/Highly Restricted 35