Using MySQL in a web-scale environment
0 likes751 views
David Landgren's presentation at Percona Live Europe discusses the use of MySQL in a large-scale video hosting platform with significant traffic and unique visitors. He outlines the architecture, database management practices including optimization techniques, handling slow queries, and schema changes. The presentation also touches on performance strategies and future directions such as global transaction IDs and parallel replication.
Using MySQL in a web-scale environment
- 1. Using MySQL in a web-scale environment David Landgren (@dlandgren, [email protected]) Percona Live Europe Amsterdam September 21-23, 2015
- 3. Some facts and figures Video hosting platform 10 years old 300 billion+ video views per month 400 million+ unique visitors per month 34th largest site world wide (comScore) Largest European site with global reach
- 4. Classic LAMP Architecture Linux (1400 servers) Apache (250 servers) Mysql (70 servers) PHP
- 5. 12 clusters Limited use of horizontal partitioning Percona 5.6 used (nearly) everywhere InnoDB used (nearly) everywhere No foreign keys or referential integrity
- 7. Primary cluster 2 master servers for writes 25 slaves for reads Additional slaves for special purposes (backups, devs) Vertical partitioning, no sharding No query cache
- 9. Managing the farm Apache ZooKeeper / zkfarmer Slaves announce their presence in the farm PHP framework is automatically aware (on next request) https://github.com/rs/zkfarmer
- 11. Performance Memcache deflects 10x more queries Nearly all reads on PK Organic growth in "slow" queries count(*) still hurts us Blindly kill long running queries
- 13. All your base are optimized How do we avoid fragmentation? Perform rolling optimize table <foo> across the schema we care about Take an ordinary slave out of production, copy the latest backup over, and send it back into production Perform regular backups on a dedicated slave (Percona Xtrabackup)
- 14. Spread the love
- 15. Spread the love^Wload Slaves have a nominal weight (probability to be selected for a given HTTP connection) 0 50 0.25 lag weight p 0 50 0.25 0 50 0.25 0 50 0.25 ∑=200
- 16. Spread the load Reduce production pressure on slaves that are not keeping up with the replication from master 0 50 0.3125 lag weight p 0 50 0.3125 0 50 0.3125 120 10 0.0625 ∑=160
- 17. Slow queries Percona has some interesting parameters for managing slow logs § log_slow_filter § log_slow_rate_limit pt-query-digest <logfile> Read the results regularly
- 18. Worst. Query. Ever. SELECT foo.foo_id AS id FROM foo INNER JOIN ( SELECT foo.foo_id FROM foo INNER JOIN bar_quux bar_gonk ON (bar_gonk.label = 'splak' AND bar_gonk.value = 'skroom') INNER JOIN foo_quux foo_gonk ON (foo_gonk.label = 'fwask' AND foo_gonk.value = 'florf') INNER JOIN bar_has_quux is_bar_gonk ON (is_bar_gonk.bar_quux_id = bar_gonk.bar_quux_id AND is_bar_gonk.bar_id = foo.bar_id) INNER JOIN foo_has_quux is_foo_gonk ON (is_foo_gonk.foo_quux_id = foo_gonk.foo_quux_id AND is_foo_gonk.foo_id = foo.foo_id) UNION SELECT baz.foo_id FROM baz WHERE baz.value > 0 ) X ON (X.foo_id = foo.foo_id) LEFT JOIN foo_has_fliff zwot ON (zwot.foo_id = foo.foo_id AND zwot.foo_fliff_id = 246) INNER JOIN foo_quux bleep ON (bleep.label = 'blort-‐allow' AND bleep.value = 'thlip') LEFT JOIN foo_quux bap ON (bap.label = 'blort' AND bap.value = 'ba') LEFT JOIN foo_has_quux blif ON (blif.foo_id = foo.foo_id AND blif.foo_quux_id = bap.foo_quux_id) LEFT JOIN foo_has_quux fweep ON (fweep.foo_id = foo.foo_id AND fweep.foo_quux_id = bleep.foo_quux_id) WHERE foo.a = 1 AND foo.b = 0 AND foo.c = 0 AND foo.d = 0 AND foo.e = 0 AND foo.f = 0 AND foo.g = 1 AND foo.bar_id = 12345 AND zwot.foo_fliff_id IS NULL AND NOT (blif.foo_id IS NULL XOR fweep.foo_id IS NULL) ORDER BY foo.tink DESC
- 19. Slow queries Keep an eye on their evolution
- 20. https://github.com/box/Anemometer Analyze slow query logs to identify problematic queries Slow queries
- 22. REALLY slow queries Save your platform, kill them github.com/dailymotion/mysql-‐genocide Graph the effects
- 24. Watching technical debt Information_schema.table_statistics SELECT t.table_name, s.rows_read FROM information_schema.tables t LEFT JOIN information_schema.table_statistics s ON (t.table_schema = s.table_schema and t.table_name = s.table_name) WHERE t.table_schema = 'myschema' AND coalesce(rows_read, 0) = 0 ORDER BY 1;
- 25. Watching technical debt Unused indices SELECT DISTINCT t.table_name, t.index_name, s.rows_read FROM statistics t LEFT JOIN index_statistics s USING (table_schema, table_name, index_name) WHERE t.table_schema = 'myschema' AND coalesce(s.rows_read, 0) = 0 ORDER BY 1, 2;
- 27. Managing change https://github.com/outbrain/propagator All schema changes are logged (and recorded in git)
- 28. Managing change (2) Percona Online Schema Change https://www.percona.com/doc/percona-‐toolkit/2.1/ pt-‐online-‐schema-‐change.html
- 29. Future Directions Global Transaction ID (GTID) Parallel replication Multiple source replication
- 30. Thank you! David Landgren (dland@github, @dlandgren) [email protected] Acknowledgements All photos by Gézelin Grée gezelingree.com / 10000lux.blogspot.com Comic drawn by Randall Munroe / xkcd.com