![Graphite’s capabilities
• Each metric is in its own bucket
– Periods make folders
– prod.syseng.mmm.<hostname>.admin_offline
• Metric types
– Counters
– Gauge
• Retention can be set using a regex
– [mysql]
– pattern = ^prod.syseng.mysql..*$
– retentions = 2s:1d,1m:3d,5m:7d,1h:5y
11](https://image.slidesharecdn.com/percona-live-europe-amsterdam-mysql-statds-150920201210-lva1-app6892/85/MySQL-performance-monitoring-using-Statsd-and-Graphite-11-320.jpg)
![Example configuration
[daemon]
logfile = /var/log/mysql_statsd/daemon.log
pidfile = /var/run/mysql_statsd.pid
[statsd]
host = localhost
port = 8125
prefix = prd.mysql
include_hostname = true
[mysql]
host = localhost
username = mysqlstatsd
password =ub3rs3cr3tp@ss!
stats_types = status,variables,innodb,commit
query_variables = SHOW GLOBAL VARIABLES
interval_variables = 10000
query_status = SHOW GLOBAL STATUS
interval_status = 500
query_innodb = SHOW ENGINE INNODB STATUS
interval_innodb = 10000
query_commit = COMMIT
interval_commit = 5000
sleep_interval = 500
[metrics]
variables.max_connections = g
status.max_used_connections = g
status.connections = c
innodb.spin_waits = c
29](https://image.slidesharecdn.com/percona-live-europe-amsterdam-mysql-statds-150920201210-lva1-app6892/85/MySQL-performance-monitoring-using-Statsd-and-Graphite-29-320.jpg)
![Adding a custom query
[mysql]
host = localhost
username = mysqlstatsd
password =ub3rs3cr3tp@ss!
stats_types = status,variables,innodb,commit,myapp
query_variables = SHOW GLOBAL VARIABLES
interval_variables = 10000
query_status = SHOW GLOBAL STATUS
interval_status = 500
query_innodb = SHOW ENGINE INNODB STATUS
interval_innodb = 10000
query_commit = COMMIT
interval_commit = 5000
query_myapp = SELECT productname, sum(order_total) FROM myapp.orders
WHERE orderdate >= DATESUB(NOW(), INTERVAL 300 second) GROUP BY productid
interval_myapp = 300000
sleep_interval = 500
[metrics]
myapp.product_a = c
myapp.product_b = c
variables.max_connections = g
status.max_used_connections = g
status.connections = c
innodb.spin_waits = c
30](https://image.slidesharecdn.com/percona-live-europe-amsterdam-mysql-statds-150920201210-lva1-app6892/85/MySQL-performance-monitoring-using-Statsd-and-Graphite-30-320.jpg)
MySQL performance monitoring using Statsd and Graphite
- 1. MySQL-StatsD MySQL Performance monitoring using Statsd, Graphite and Grafana Art van Scheppingen Freelance DBA @ DBArt
- 2. Overview • Who am I? • What monitoring tools are out there? • What are StatsD, Collectd and Graphite? • How to use StatsD? • MySQL+StatsD = MySQL-StatsD • Graphing examples • Grafana • Challenges • Questions?
- 3. Who am I? • Freelance DBA • Worked >10 years in the gaming industry • Worked for several start ups • HA and scaling specialist
- 4. What monitoring systems are out there?
- 5. Clarifying monitoring methods • Pull – Nagios (NRP) – SNMP • Push – Shinken – StatsD / CollectD
- 6. Monitoring storage systems • File based – RRD – Carbon (single host) • Database based – zabbix • Cluster based – OpenTSDB – InfluxDB – Carbon (clustered)
- 7. Popular monitoring systems • Cacti – Pull + RRD • Zabbix – Pull + SQL • Munin – Pull + RRD • Shinken – Pull + Push + RRD • Monyog – Pull + proprietary storage • MRTG – Pull + RRD
- 8. Hosted solutions • VividCortex • Severalnines • Percona cloud tools • Cloudwatch (AWS) • Google Cloud Monitoring (Google)
- 9. Statsd + Collectd + Graphite What are they?
- 10. What is Graphite? • Highly scalable real-time graphing system • Collects numeric time-series • Backend daemon Carbon – Carbon-cache: receives data – Carbon-aggregator: aggregates data – Carbon-relay: replication and sharding • RRD or Whisper database 10
- 11. Graphite’s capabilities • Each metric is in its own bucket – Periods make folders – prod.syseng.mmm.<hostname>.admin_offline • Metric types – Counters – Gauge • Retention can be set using a regex – [mysql] – pattern = ^prod.syseng.mysql..*$ – retentions = 2s:1d,1m:3d,5m:7d,1h:5y 11
- 12. Graphite server Graphite server 12 Client requesting graphs Graphite (port 443) Server- 1 Server- 2 Server- n Carbon (port 2003) Graphite Rendering Carbon relay
- 13. Scaling a Graphite environment 13 Client requesting graphs Graphite Rendering Cluster Carbon relay Loadbalancer (port 443) DEV TEST PRD DC1 PRD DC2 Server- 1 Server- 2 Server- n Loadbalancer (port 2003) Skyline 24h retention 1 node
- 15. What is Collectd? • Unix daemon that gathers system statistics • Over 90 (input/output) plugins • Plugin to send metrics to Graphite/Carbon • Very useful for system metrics
- 16. Collectd 16 Collectd Gather data plugins CPU DISK LOAD …. CarbonTCP 30 second interval
- 17. What is StatsD? • Front-end proxy for Graphite/Carbon (by Etsy) • NodeJS daemon (also other languages) • Receives UDP (on localhost) • Buffers metrics locally • Flushes periodically to Graphite/Carbon (TCP) • Client libraries available in many languages • Send any metric you like! 17
- 18. StatsD functions • StatsD functions – update_stats (counter) • Difference between previous and current value • metric = <value> per second – increment/decrement • metric++ / metric-- – Set • Counts the number of unique items (e.g. userids, email) • metric = unique(value1, value2, value3, value4) – gauge • What is the current value (e.g. 240.87 volts) and persist • metric = <value> – timers • How much time did it take in seconds? • metric = <value> in seconds 18
- 19. StatsD 19 StatsD Application Level # OF LOGINS CACHE HIT/MISS STATUS INNODB STATUS CarbonTCP 2 second interval MySQL_Statsd localhost:8125 UDP
- 20. Using StatsD
- 21. StatsD Bash examples echo ”some.metric:1|c" | nc -w 1 -u graphite.host 8125 echo ”some.metric:1|c" > /dev/udp/localhost/8125 bash-3.2# netstat -s | grep "listen" 26 times the listen queue of a socket overflowed netstat -s | grep "listen" | awk '{print "hostname.listen.queue.overflowed:"$1"|c"}’ > /dev/udp/localhost/8125 hostname.listen.queue.overflowed:26|c echo "show global status" | mysql -u root | awk '{print "hostname.mysql.status."$1":"$2"|c"}' 21
- 22. Other metrics • Deployments • User initiated actions – Logins – High scores – Comments / ratings – Images uploaded – Payments • Application metrics – Error counts – Cache statistics (cache hit/miss) – Request timers – Image sizes 22
- 23. Common mistakes 1 • Using an update instead of a gauge – com_select vs select queries since last flush prod.host.mysql.status.com_select:100000|c ---- flush (every 5 seconds) prod.host.mysql.status.com_select:100010|c prod.host.mysql.status.com_select:100015|c ---- flush (every 5 seconds) Theory: 10 + 5 / 5 seconds = 3 queries per second Reality: 100010 + 100015 / 5 seconds = 40005 per second
- 24. Common mistakes 2 • Misaligned flush interval with update (counter) – StatsD flush interval 10 seconds – Application interval 15 seconds – Results in every second StatsD flush to be empty – Change to gauge ---- flush (10 seconds) prod.host.mysql.innodb.history_list:4|c ---- flush (10 seconds) ---- flush (10 seconds) prod.host.mysql.innodb.history_list:8|c ---- flush (10 seconds) ---- flush (10 seconds) prod.host.mysql.innodb.history_list:6|c ---- flush (10 seconds)
- 25. MySQL + StatsD = MySQL-StatsD
- 26. Why use StatsD over Collectd? • MySQL plugin for Collectd – Sends SHOW STATUS – No INNODB STATUS – Plugin not flexible • DBI plugin for Collectd – Metrics based on columns • Different granularity needed • Separate daemon (with persistent connection) • StatsD is easy as ABC 26
- 27. MySQL StatsD daemon • Written in Python • Rewritten and open sourced during a hackday • Gathers data every 0.5 seconds • Sends to StatsD (localhost) after every run • Easy configuration (Unix type config) • Persistent connection • Baron Schwartz’ InnoDB status parser (cacti poller) – Multi bufferpool support • Other interesting metrics and counters – Information Schema – Performance Schema – MariaDB specific – Galera specific – If you can query it, you can use it as a metric! 27
- 28. MySQL StatsD overview 28 StatsD MySQL Thread MySQL MySQL StatsD daemon StatsD thread SHOW GLOBAL VARIABLES SHOW GLOBAL STATUS SHOW ENGINE INNODB STATUS SHOW SLAVE STATUS Preparsers InnoDB Columnar
- 29. Example configuration [daemon] logfile = /var/log/mysql_statsd/daemon.log pidfile = /var/run/mysql_statsd.pid [statsd] host = localhost port = 8125 prefix = prd.mysql include_hostname = true [mysql] host = localhost username = mysqlstatsd password =ub3rs3cr3tp@ss! stats_types = status,variables,innodb,commit query_variables = SHOW GLOBAL VARIABLES interval_variables = 10000 query_status = SHOW GLOBAL STATUS interval_status = 500 query_innodb = SHOW ENGINE INNODB STATUS interval_innodb = 10000 query_commit = COMMIT interval_commit = 5000 sleep_interval = 500 [metrics] variables.max_connections = g status.max_used_connections = g status.connections = c innodb.spin_waits = c 29
- 30. Adding a custom query [mysql] host = localhost username = mysqlstatsd password =ub3rs3cr3tp@ss! stats_types = status,variables,innodb,commit,myapp query_variables = SHOW GLOBAL VARIABLES interval_variables = 10000 query_status = SHOW GLOBAL STATUS interval_status = 500 query_innodb = SHOW ENGINE INNODB STATUS interval_innodb = 10000 query_commit = COMMIT interval_commit = 5000 query_myapp = SELECT productname, sum(order_total) FROM myapp.orders WHERE orderdate >= DATESUB(NOW(), INTERVAL 300 second) GROUP BY productid interval_myapp = 300000 sleep_interval = 500 [metrics] myapp.product_a = c myapp.product_b = c variables.max_connections = g status.max_used_connections = g status.connections = c innodb.spin_waits = c 30
- 31. Start graphing!
- 32. What is important for you? – Identify your KPIs – Don’t graph everything • More graphs == less overview – Combine metrics – Stack clusters 32
- 33. Correlate! – Include other metrics into your graphs • Deployments • Failover(s) – Combine application metrics with your database – Other influences • Launch of a new game 33
- 34. Graphing • Graphite Graphing Engine – DIY – Giraffe • Readily available dashboards/tools – Grafana (Kibana fork) – Skyline (Etsy) – Dashing (Shopify) 34
- 35. Graphite Graphing Engine – URI based rendering API – Support for wildcards – stats.prod.syseng.mysql.*.status.com_select – sumSeries (stats.prod.syseng.mysql.*.status.com_select) – aliasByNode(stats.prod.syseng.mysql.*.status.com_select, 4) • Many functions – Nth percentile – Holt-Winters Forecast – Timeshift 35
- 38. Graphing with Grafana I know Kibana hence I know Grafana
- 39. What is Grafana? • Fork of the Kibana interface – Flexible and intuitive GUI – Easy to create graphs • Supports Graphite, InfluxDB and OpenTSDB • Clientside JavaScript rendering (no gif/png) • Templated dashboards • Scripted dashboards • Supports CMDB apis
- 40. Demonstration
- 41. Challenges Sounds great, can I use it?
- 42. MySQL-Statsd repo Current version: 1.4.0 https://github.com/db-art/mysql-statsd
- 43. MySQL-Statsd roadmap – Most teething problems have been resolved – Install via pip • vitrual-env advised – Support custom columnar queries – Whitelisting of wildcard paths not supported yet – Merge with Mongo-Collector? 43
- 44. What lessons have we learned? – Persistent connections + repeatable read • History list skyrocketed – Requesting too many metrics slows down graphing – Sending too many metrics can kill a host • EstatsD for Erlang 44
- 45. Questions?
- 46. Practical links – Graphite: http://graphite.readthedocs.org/en/latest/ – Grafana: http://grafana.org/ – Collectd: https://collectd.org/ – StatsD on Github by Etsy: https://github.com/etsy/statsd/wiki – Etsy on StatsD: http://codeascraft.etsy.com/2011/02/15/measure- anything-measure-everything/ 46
- 47. Thank you! • Presentation can be found at: http://goo.gl/AR3OCf MySQL Statsd can be found at: https://github.com/db-art/mysql-statsd • If you wish to contact me: [email protected] 47
Editor's Notes
- #46: ----- Meeting Notes (30-11-12 12:00) ----- Abbreviations (try to pronounce) Theory too long, second part too brief. High Availability -> HA What do we do? Games! 180M+ Query numbers on DBs Some examples of portal names SSP is abstraction layer SSP query example Explain why horizontal instead of vertical Functional sharding slide! Explain why sattelite DC Introduction to sattelite data centers (moving data to caching) but explain they do not own the data Instead of example of migrating users, example of adding a new DC Slide 23: leave out slide Why we chose erlang: remove pattern matching. Adds productivity: simpler Add another example for buckets with a different backend Slide 22: partition on users, bucket and GIDs. It is not a mess in LAMP stack: the backend is just not scalables