ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
Download as PPTX, PDF2 likes1,623 views
The document discusses the challenges of scaling MySQL and proposes Scalebase as a solution for effective sharding that enhances performance without the need for extensive application rewrites. It outlines MySQL 5.6's limitations and the advantages of Scalebase, which provides a unified database experience, fault tolerance, and simplified administration while ensuring ACID compliance. Scalebase allows for seamless scaling in cloud environments and supports various deployment options, making it a flexible choice for managing database growth.
ScaleBase Webinar: Scaling MySQL - Sharding Made Easy!
- 1. Scaling MySQL – Sharding Made Easy
- 2. 2 Agenda • Scalability Issues • MySQL 5.6 • Why Do-It-Yourself (DIY) Sharding Sucks • ScaleBase Data Distribution: – Successful sharding on Amazon and private clouds – Single vs. multiple shards per server – Eliminating data silos – Creating a redundant, fault-tolerant architecture – Re-balancing and splitting shards • Q & A
- 3. 3 Doron Levari, Founder & CTO Doron Levari, Founder & CTO A technologist and long- time veteran of the database industry. Prior to founding ScaleBase, Doron was CEO to Aluna.
- 4. 4 What We Do Simply and cost-effectively scale MySQL to support an infinite number of users, transactions and data with NO disruption to the existing infrastructure
- 5. Scalability Issues and MySQL 5.6
- 6. 6 MySQL Scalability Challenges • Too many transactions • Too many users • Too much data • Too many writes • Capacity • Throughput • Performance inconsistencies
- 7. 7 Improvements in MySQL 5.6 – Single Box Partitioning Improvements – Explicit Partition Selection: SELECT * FROM employees PARTITION (p0, p2); – Import / Export for Partitioned Tables: Bring a new data set into a partitioned table, or export a partition to manage it as a regular table ALTER TABLE e EXCHANGE PARTITION p0 WITH TABLE e2; http://dev.mysql.co/tech-resources/articles/whats-new-in-mysql-5.6.html Replication Improvements – Optimizations to Row-Based Replication – Multi-Threaded Slaves – Improvements to Data Integrity – Crash-Safe Slaves – Replication Checksums SCALABILITY issues remain due to the limitations of a single box: To ensure ACID, you still face limitations with: - Memory management - Thread management - Semaphores - Locking - Recovery tasks No new functionality for sharing workloads across multiple boxes
- 8. 8 What are my Options 1. More/Bigger Hardware? – Temporary fix…you will need new hardware again – More memory…helps mostly with “reads,” but not with “writes” – Every write operation is at least 4 write operations in database, plus multiple activities in the database engine memory 2. Application re-architecture? – Steer workload away from the database – Example: introduce a caching layer – Force application re-writes; new test & QA cycles 3. Do it Yourself Sharding? 4. Migrate to new database architecture – Other RDBMS/NewSQL / NoSQL? – Force application re-writes; new test & QA cycles – ACID/Durability Issues
- 9. 9 Scale Out your Existing MySQL • Keep your MySQL - keep your InnoDB • Ecosystem compatibility, preserve skills • 100% application compatibility • Smoother migration, no down-time, no forklift • Your data is safe • No “in-memory” magic • No “in-memory” size limit Don’t throw out the baby with the bath water!
- 10. Why Do It Yourself Sharding “Sucks”
- 11. 11 What is Sharding? Wikipedia - Shard (database architecture) http://en.wikipedia.org/wiki/Shard_(database_architecture) A database shard is a horizontal partition in a database or search engine. Each individual partition is referred to as a shard. Horizontal partitioning is a database design principle whereby rows of a database table are held separately, rather than being split into columns. Each partition forms part of a shard, which may in turn be located on a separate database server or physical location.
- 12. 12 DIY Sharding Challenges Applications must be modified to support multiple shards
- 13. 13 • Maintaining DB ops and IPs in the app • Non-optimized sharding strategies – No good way to maintain global tables replicated across all database • Sacrifices development agility, additional administrative complexity • Results in database silos • Database ecosystem breaks because the application “conceals” sharding strategies internally • Risks for data inconsistency • Adding and removing databases is not supported…overprovisioning… • Jeopardizes high availability, backups & disaster recovery • Demands custom application code that can fail ACID compliance DIY Sharding Challenges Challenges exist because application code changes are required to support multiple database instances.
- 14. ScaleBase Data Distribution Overview
- 15. 15 Data Distribution: Application Experience Without ScaleBase: App must be customized to support shards With ScaleBase: App sees ONE database… …and doesn’t require any customization ScaleBase acts as a proxy between the app and the database, virtualizing the database environment
- 16. 16 Manual Sharding versus ScaleBase Sharding Limitations: • Major app rewrite, maintaining code • Maintaining DB ops & IPs in the app • Administration/3rd party tools are broken • DB silos/Database ecosystem is blind – Application “hides” sharding strategies • Non-optimized data distribution policy – No good way to maintain global tables, replicated across all database • Sacrifices development agility • Adding/removing DBs is not supported • Risks for data inconsistency • Demands custom application code that can fail ACID compliance • Jeopardizes high availability, backups, and disaster recovery ScaleBase Benefits: • No hard-coding application re-writes • Unlimited scalability • Improve performance • Real time elasticity • ACID compliance • Verified data consistency • Real time monitoring, traffic analysis • Carefully analyze distribution policy • Enable system upgrades and updates • Simplified, centralized admin – Adding users – Changing schemas – Maintenance scripts – Management queries
- 17. 17 Typical ScaleBase Data Traffic Manager Deployment Application Servers BI Management Database A Replica A Database B Replica B Database C Replica C Database D Replica D Unlimited Scale ScaleBase Architecture is Fault Tolerant
- 18. ScaleBase Data Distribution – In Detail
- 19. 19 ScaleBase Enables MySQL Scale Out without Re- writing Apps • Data distribution and scale-out is part of the database architecture, not the application • One IP to connect to, and “see a unified database” – The application – Entire ecosystem (ETL, mysqldump, PHPMyAdmin) – No special sharding wizard developer – No app re-design, re-dev, re-QA, re-test, re-deploy – No hard-coded variables lost in the code – No special documentation
- 20. 20 ScaleBase Enable Scale Out on AWS and Private Clouds • A virtualized DB environment makes it easy to change real infrastructure, because it’s decoupled from the application • No cloud makes your database elastic • ScaleBase enables elasticity of MySQL in the cloud (EC2, RDS, etc.) Scale-up hits AWS’s tiered configuration limits fast Scale-out is unlimited and gives cloud flexibility
- 21. 21 ScaleBase Supports Scale Out on Single & Multiple Machines Advantages of several shards on one machine: – Several smaller MySQL instances better utilize cores, memory – When data grows, each instance can later on migrate to a bigger machine of its own Advantages of several shards on multiple machines – Leverage commodity hardware – When reaches machine limits - ScaleBase enables online data redistribution (resharding) and shard-split
- 22. 22 ScaleBase Enables Splitting Shards • ScaleBase also redistributes data across the array to eliminate hot spots, splitting the hot spot into two databases
- 23. 23 ScaleBase Re-balances Shards • Special analysis and alerts about approaching limits • ScaleBase dynamically redistributes data (resharding) - moving the data across the array from the over-utilized to the under-utilized
- 24. 24 ScaleBase Provides Optimal Data Distribution Policies A good data distribution policy ensures that a specific transaction is directed to a specific database 1,000 transactions 250 transactions 250 transactions 250 transactions 250 transactions 1,000 transactions
- 25. 25 ScaleBase Eliminates Data Silos When a query needs data from several databases, ScaleBase: – Runs the query in parallel on all databases – Aggregates results into one meaningful result-set to be returned to the client – the same result-set that would have been returned from a single DB! – Including cross-db GROUP BY, ORDER BY, aggregate functions – Including cross-db JOIN operations – Enables 2-phase commit for transactions spanning multiple databases
- 26. 26 ScaleBase Provides a Fault Tolerant Architecture Application Servers BI Management Database A Replica A Database B Replica B Database C Replica C Database D Replica D Fully Redundant Resilience to failures Scheduled maintenance without downtime
- 27. Summary
- 28. 28 ScaleBase Delivers Scalability Scale to Unlimited Throughput No Specialized Hardware No Re-architecture No Application Rewrites
- 30. 30 Detailed Scale Out Case Studies Large Chip Co • Scalability • Multiple Apps • Multiple growing users • Availability • MySQL DB Solar Edge • Next Gen Monitoring App • Massive Scale • Monitors real time data from thousands of distributed systems Mozilla • New Product/ Next Gen App/ AppStore • Scalability • Geo-clustering AppDynamics • Next gen APM company • Scalability for the Netflix implementation
- 31. 31 ScaleBase Deployment Environments – Public Cloud – AWS, Rackspace, any – Private cloud – Hosted / on-premise Databases Supported – MySQL 5.1, 5.5, 5.6 (under certification) – AWS RDS MySQL 5.1, 5.5 – Maria DB 10.0 (under certification) Path to Scale-Out: 1. Data Distribution Policy Analysis 2. Functional Test 3. Load Test 4. Production Migration (safe, online)
- 32. 32 Summary ScaleBase provides cost-effective Scale-Out solutions • Scale to an infinite number of users, data and transactions • Improve performance • No application rewrites • Real-time elasticity • ACID Compliant • Expert analysis and simple deployment • Leverage existing MySQL ecosystem/skills • Improve database visibility with real-time monitoring • Simplified, centralized administration
- 33. 33 Questions (please enter directly into the GTW side panel) [email protected] [email protected] www.ScaleBase.com 617.630.2800 Additional Resources http://www.scalebase.com/blog/ http://www.scalebase.com/resources/ @scalebase
- 34. 34 Thank You