More Related Content
Similar to MariaDB MaxScale: an Intelligent Database Proxy (20)
MariaDB MaxScale: an Intelligent Database Proxy
- 1. MariaDB MaxScale: An Intelligent Database Proxy Why we should abstract the database cluster and how to do it
- 2. Overview • Part 1 – What is a database cluster? – Why is database abstraction important for clusters? – Few Examples of simple cluster abstraction • Part 2 – Deep dive into MariaDB MaxScale: a Database Proxy
- 3. Part 1 What is Database Cluster Abstraction and How it can be Done
- 4. What is a Database Cluster? ● Two or more nodes ● Logically connected ○ Shared storage ○ Logical replication ● All nodes are active ○ Hot standby ○ Multi-master cluster
- 5. Two Types of Database Clusters Master-Slave Cluster ● Logical replication ○ MariaDB/MySQL ● Shared storage ○ Amazon Aurora ● One authoritative node ○ Write to one, read from many ● Conflict-free Master SlaveSlave
- 6. Synchronous Cluster ● Data synchronization ○ MariaDB Galera ○ MySQL Group Replication ○ MySQL NDB Cluster ● No authoritative nodes ○ Write to all, read from all ● Conflicts can occur Two Types of Database Clusters Master MasterMaster
- 7. The Perfect Database: ● Behavior of a single database ○ Simple to use ○ Easy to manage ● Characteristics of a cluster ○ Robust and failure tolerant ○ Near-linear scalability What is Database Abstraction for Clusters? The Database
- 8. Why is Database Abstraction Important for Clusters? ● Isolates the complexity of the cluster ○ One logical database, multiple physical databases ○ Build simpler applications ● Highly Available Database ○ Fault tolerant ○ Easier maintenance ● Improved performance ○ Load balancing ○ Read/Write splitting Database Abstraction Layer
- 9. ● Readily available ○ Corosync/Pacemaker ○ Keepalived ● Well integrated in cloud ○ Amazon ELB ○ Azure Load Balancer ● Not really intended for databases Example: Floating/Virtual IP 10.0.0.1 192.168.0.180 192.168.0.30
- 10. ● Built into application ○ No external dependencies ● Simple setup ○ e.g. JDBC connection string ● Not in all connectors ● Increases application complexity Example: Connectors/Drivers 2. 3.1.
- 11. ● Specialized software ○ Isolates complexity ● Provides a service ○ Encapsulates resources ● One point of entry ○ Minimizes external footprint Example: Proxy Proxy
- 12. ● Replacement of failed nodes ○ Activate standby nodes ○ Query interruptions ■ Retry failed queries ■ Replay transactions ● Changes in topology ● Read/Write splitting ○ Classification of queries Why Simple Abstraction is Not Enough: Problems Involved in Database Abstraction ● Session state ○ Session and user variables ○ Dependent queries ■ Last generated ID ■ Deleted/updated row count ○ Transactions ○ Temporary tables ○ Multi-statement queries
- 13. Problem: Node Failure Failure of a single node should not be visible to the client Steps involved: ● Detect failed node ● Replace with standby node ● Retry interrupted query ● Replay interrupted transaction Database Abstraction Layer
- 14. Problem: Changes in Topology 1. Initial state 2. Node type changes 3. New node is added 4. Node goes into maintenance Master SlaveSlave Slave SlaveMaster Slave SlaveMaster Slave Slave SlaveMaster Slave 2 3 41 Cluster abstraction must understand and handle all cases
- 15. START TRANSACTION; SELECT name FROM accounts WHERE id = 1; INSERT INTO logins VALUES (‘john doe’); COMMIT; Problem: Transactions Transactional behavior must be kept intact ● Executed only on one node ● Replay transaction only when it is safe
- 16. SELECT name FROM accounts WHERE id = 1; INSERT INTO logins VALUES (‘john doe’); SELECT LAST_INSERT_ID(); SET @@character_set_client=cp850; Problem: Query classification Read Write Dependent Query Session State Different queries require different behavior ● Writes to master ● Reads to slaves ● Dependent queries to previous server ● Session state modifications to all
- 17. ● Non-transactional ○ Work on a single node ○ Fail when load balanced ● Depend on previous queries ○ Read inserted value Problem: Critical Reads INSERT INTO accounts VALUES (‘john doe’); SELECT name FROM accounts WHERE name = ’john doe’; ● Not compatible with load balancing ○ Can return a result without the inserted value ● Not the “correct way” to do it ○ Legacy application → hard to modify ○ Framework →impossible to modify
- 18. Part 1: Summary ● Database clusters ○ Improve performance ○ Highly available ○ Not easy to handle ● Database cluster abstraction ○ Hide complexity ○ Simplifies application development ○ Easier node maintenance Not an easy problem to solve
- 19. Part 2 MariaDB MaxScale: Intelligent Database Proxy
- 20. MaxScale Overview ● Modular Database Proxy ○ Only use what is needed ○ Extendable ● Content-aware ○ Understands routed traffic ● Cluster-aware ○ Active cluster monitoring ○ Understands different cluster types
- 21. MaxScale Core Components Service ● A “virtual” database ● Collection of modules Router ● Main routing logic ● Handles load balancing Query Classifier ● Parses SQL ● Provides information Monitor ● Cluster-specific ● Actively monitors nodes
- 22. SELECT WHERE id = 1; ● Provides both abstract and detailed information ○ Read or write ■ Does the query modify the database? ○ Query components ■ Is the table `t1` used in this query? ■ What are the values for the functions in the query? ○ Query characteristics ■ Does the query have a WHERE clause? ○ State changes ■ Was the default character set changed? ■ Is there an open transaction? The Brains: Query Classifier Read-only query
- 23. ● Read/write splitting ○ Write to master, read from slaves ○ Major performance improvement ○ Prevents conflicts ● Session state tracking ○ Consistent session state ● Failure tolerant ○ Hides slave failures ● Multiple backend connections ○ Must-have for read/write splitting ○ Speeds up node failover The Brawn: ReadWriteSplit
- 24. Looks at the cluster ● Tracks node status ○ Replication configuration →topology detection ○ Relevant global variables ○ Overall status →detect broken slaves ● Actively detects failures ○ Unresponsive servers ○ Out-of-sync nodes (mainly Galera) The Eyes: Monitors
- 25. Monitors Abstracting the Cluster Concept
- 26. ● Classify servers ○ Up or Down? ○ Master or Slave? ○ In sync or not? ● Information used by routers ○ Masters used for writes ○ Slaves used for reads ● Detects events ○ Server went down ○ Slave is disconnected Overview: Monitors
- 27. ● Detects topology ○ Builds the replication tree ● Assigns correct labels ○ Root node for writes ○ Other nodes for reads ● Detects replication lag ○ Write timestamp on master ○ Read from slave Master-Slave Monitor Master SlaveSlave This is a master This is a slave
- 28. ● Galera Clusters ○ Synchronous Cluster ○ Globally conflict free ○ Conflicting transaction → Error on commit ● Abstracted in MaxScale ○ One “master” node ■ Prevents conflicts ○ Rest labeled as “slaves” ■ Good for scaleout Galera Cluster Monitor Master MasterMaster Use this for all writes... …and these two for reads
- 29. ReadWriteSplit The Main Router in MaxScale
- 30. 1. Classify query ○ Write, read, session state modification etc. ○ Uses Query Classifier 2. Choose routing target based on query attributes ○ Modifies the database → master ○ Read-only → slave ○ Session stateme is modified → all servers 3. Pick a target server ○ Pick most valid node ■ Governed by load balancing algorithm 4. Write query and read result ReadWriteSplit: Read/Write Splitting
- 31. Based on server score ● Multiple algorithms ○ Active operation count → Default ■ MIN(operations) ○ Connection count ■ MIN(connections) ○ Replication delay ■ MIN(delay) ● Manually adjustable ○ Weight each server differently ■ MIN(score * weight) ReadWriteSplit: Load Balancing
- 32. Read retry ● Hides “trivial” failures ○ SELECT statement ○ autocommit=1 ○ No open transaction ● Guaranteed reply ○ Try slaves first ○ Use master as last resort ReadWriteSplit: Hiding Node Failures
- 33. ● Triggered on master failure ○ Master server down ○ Lost connection to master ● Read-only queries and transactions allowed ○ For read-heavy traffic ● Configurable behavior ○ Close connection on master failure ○ Close connection on first write ○ Send error on all writes ReadWriteSplit: Read-only Mode
- 34. ● Consistent state for all connections ○ State modifications propagated ○ Truly abstracted reads ● State modification history ○ Node replacement ReadWriteSplit: Session State SET SQL_MODE=’ANSI’;
- 36. ● Between client and router module ○ Pre-processing ○ Analytics ○ Target hinting ● Chainable ○ Output pipes to input ● Easy to write ○ First community contribution ■ tpmfilter Filter Overview
- 37. ● Detects data modification ○ Writes “pin” the session to master ● Tags the query with a hint ○ Route to master ● Configurable ○ Number of queries ○ Time interval CCRFilter: Consistent Critical Reads INSERT INTO accounts VALUES (‘john doe’); SELECT name FROM accounts WHERE name = ’john doe’; Route this to the master!
- 38. ● Match-replace functionality ○ PCRE2 regular expressions ● Fix broken SQL ○ “Patching” after release ● Allows neat tricks ○ Append a LIMIT clause ○ Add optimizer hints ○ Change storage engine Regexfilter: sed for SQL
- 39. ● Monitors ○ Provide abstract information ○ Assign labels to nodes ● Routers ○ Hide node failures ○ Detect dependencies ○ Load balance queries ● Filters ○ Extend functionality ○ Tailored behavior Part 2: Summary
- 40. Solution: Use the right tool. Work smart, not hard. Wrapping Up Problem: Database clusters are essential for performance and HA but are also hard to use properly.
- 41. Wrapping Up MaxScale: A Toolbox for the Database. ● Abstracts database clusters into services ● Truly understands traffic and environment ● Makes database clusters easy to use efficiently
- 42. Thank you Get Started with MaxScale SlideShare