Managing MySQL Scale Through Consolidation

Editor's Notes

• #3: When you’re using MySQL in production, at scale in the Enterprise, you’re usually not dealing with 100MB databases (Unless you’re a service provider of some sort, maybe SaaS such as WordPress). When datasets get large, into the 100s of GB or multi-TB range, you end up managing a new class of challenges which broadly include 1) planning for the future (or, scaling your MySQL platform resources over time), and 2) Quickly responding to the needs of the present (or, scaling and re-allocating your system resources in real-time)
• #4: Requires understanding your application data storage profile, Being able to predict, as accurately as possible, the growth trajectory for your MySQL systems This requires knowing your resource consumption profiles for Compute, Memory, Storage
• #5: * Compute resources * What is contributing to CPU consumption? (query processing, i/o wait, virtualized steal?, etc) * Is CPU consumption is truly from mysqld churning and processing data (rather than wait, steal, network wait, etc)? * What is the CPU growth rate for your systems?
• #6: Are you memory bound? Does the hotset of data fit in memory? Are your application queries optimized to accurately target the required information, rather than pulling unnecessary data into memory? (select * vs select 1) How about indexes? Are you over indexed? Under indexed? Given data growth rate, when is the next RAM increase required to avoid serious performance degradation?
• #7: Are you disk resource heavy? Are you IOPS bound? What is the max IOPS ceiling for your disk configuration? Are disk latencies within acceptable range for the required query and application response time? Does your application lean on sequential reads, such as range queries or large chunk processing? Is the query profile more random in nature? If so, are your disk heads thrashing (if on spinning disk) Is flash memory indicated for your dataset and application needs? What is your disk usage consumption rate? When is the next storage resource addition required? How will you add that capacity? Are you going to extend the existing filesystem via LVM? Can you extend the existing filesystem natively? (xfs_grow) Do you use a filesystem-per-database strategy (symlinks)?
• #8: At a higher level, do you have growth rate and and topology scaling plan for: * Master cluster servers (via technologies such as Percona Cluster or Galera) * Master shard servers (via homegrown or mysql extension technologies such as ClusterixDB) * Slave servers (used in read-heavy environments) * DevTest instances that require copies of production datasets * Reporting instances for consolidated warehouse querying by various departments * Even if you’re running in the Public Cloud, such as AWS, there comes a point in the growth curve where you have to evaluate the costs of not bringing servers back in-house and creating your own Private Cloud to control costs. True dedicated performance in the public cloud understandably costs a considerable amount more than oversubscribed best-effort shared resource pools (example: pIOPS database storage volumes)
• #9: All of these things require you to understand your systems current resource consumption profiles, and be able to quantify the change rate over time, and in relation to other growth factors, such as application end-users (for every million users of our platform we add, we’ll require X additional resources for the MySQL layer), or in relation to internal company growth (for every 10 developers we add, we’ll require Y additional MySQL resource pools). Gathering and accessing this data absolutely requires the ability to monitor every layer of your platform stack, and there are a great number of monitoring tools that you can use to instrument your environment in order to collect, graph, and analyze this data over time. Nagios + Graphite, Zabbix, Zenoss, Munin, Cacti, and many more. The most important point is that you capture this data over time and reserve time to analyze and understand growth trends. These tools also are invaluable for real-time troubleshooting, allowing you to zero in and better ascertain and pinpoint root cause during incidents. Incident data can also be used to set up predictive and preventative alerting to avoid outages before they happen. This type of tooling, data, and insight, in turn, enables one to confidently plan and to be responsive to the needs of the present, but as datasets get larger, so do the operational challenges.
• #10: DBAs and DevOps teams are faced with a specific class of problems when MySQL datasets move toward the TB and multi-TB range, or when the scale of various copies of the system exceeds the cost-benefit of manual operations. Automation becomes an increasingly important dimension of the MySQL landscape. When very large databases (VLDBs) are in play, the time it takes to copy or manipulate the corresponding data volumes becomes a major operational hinderance, and a whole host of strategies and tactics need to be employed to reduce or eliminate performance impact and downtime that would otherwise be required during maintenance operations. Operations staff touch-time requirements can also be minimized with various tools and techniques. * Quickly respond to requirements to increase performance * * In virtualized environments, CPU and RAM allocations can be added quickly (though a restart is likely required) * Storage capacity can be added, or storage allocations can be reconfigured to provide additional throughput * In advanced storage systems, you can even dial up the IOPS allocations for a MySQL data volume on the fly, allowing more throughput to the database in real time, no restart required * Deploy additional read-slaves for copies quickly * Refresh Dev/Test, QA, and business copies of databases * Improve backup and restore times As your enterprise grows, your data gravity increases. This comes along with additional customers, end-user, and organizational growth. This increased data mass can mean decreasing agility and drag on your business in the form of increased MySQL platform management complexity, longer response times for provisioning database systems, delays in refreshing of business reporting and data warehouse systems, among other challenges.
• #11: As compute platforms get more flexible, both in the public cloud, and in private clouds, it becomes increasingly possible to turn the traditional model of database platform management on its head. Rather than seeing the data layer as the child or property of the compute layer, Make the compute layer an attribute of the dataset. With data gravity and ‘stickiness’ being the operational bottleneck, you can pivot on your storage layer and use the capabilities of modern shared storage systems, combined with virtualization, allow you to flip that model. I refer to this as the technique of Storage Jujitsu. Some of the capabilities include instant snapshots and quick volume cloning, or writable snapshots. Data de-duplication for very efficient use of high performance shared storage platforms. QoS or Quality of Service allocation to segregate performance resources between storage volumes, allowing you to put production MySQL database systems on the same storage array as DevTest and other secondary systems without fear of resource conflict (commonly known as the noisy neighbor effect). Scale-out architectures are also a great development in storage, allowing for seamless storage growth without the need for huge migrations and massive capital poured in to system refreshes. These systems are all designed to make efficient use of storage resources and avoid the need to over provision. The key here is to avoid the use of the network to copy and transfer data, but instead, lean on the native capabilities of the underlying storage layer.
• #12: Let’s take a look at some examples of how this can be done, in practice. Here are some the operational models we use at SolidFire for managing our Percona Server MySQL platforms that have been completely reimagined and retooled to leverage next-generation storage capabilities. - Dev/test secondary copies - Replication slave creation - Backup windows - Restore scenarios