How (not) to kill your MySQL infrastructure
Download as PPTX, PDF0 likes434 views
The document discusses common pitfalls in managing MySQL infrastructure, sharing experiences from the author's background in backend development. Key issues include version mismatches, configuration discrepancies, and the importance of proper user connection management and backups. It emphasizes the use of monitoring tools and best practices to maintain a healthy MySQL environment.
![8
• Wait_timeout 28800 by default
„Warning: mysql_connect() [function.mysql-connect]: User hircsardasql
already has more than 'max_user_connections' active connections in
/web/hircsarda/hircsarda.hu/wp-includes/wp-db.php on line 1515””
• USE max_user_connections, this will prevent a single app from using all
threads(max_connections)
gdb -p $pid -ex "set max_connections=3000 --batch
4. MY.CNF – WAIT_TIMEOUT](https://image.slidesharecdn.com/budapestmysqlmeetupmiklosszel-161202080048/85/How-not-to-kill-your-MySQL-infrastructure-8-320.jpg)
![17
• Disable atime on the data partition
• Disable swap
• Allocate most of the RAM to MySQL
• This will make it the perfect candidate to kill for the OOM-killer
• tmp files goes to /tmp by default
• DDLs on big tables can consume it!
[mysqld]
open_files_limit = 10000
/etc/security/limits.conf:
* hard nofile 10000
* soft nofile 10000
7. OS](https://image.slidesharecdn.com/budapestmysqlmeetupmiklosszel-161202080048/85/How-not-to-kill-your-MySQL-infrastructure-17-320.jpg)
![20
“In a Web environment where the clients are connecting from a Web server, a user
could use LOAD DATA LOCAL to read any files that the Web server process has read
access to (assuming that a user could run any command against the SQL server).”
http://dev.mysql.com/doc/refman/5.6/en/load-data-local.html
[root@hostdb~]# mysql -e "SHOW GLOBAL VARIABLES LIKE 'local_infile'"
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| local_infile | ON |
+---------------+-------+
[root@hostdb ~]# mysql -e "show grants for wordpress@%”
GRANT USAGE ON *.* TO ’wordpress@'localhost' IDENTIFIED BY PASSWORD ‘xxx’
GRANT SELECT, INSERT ON `wordpress`.* TO ’wordpress'@’%’
10 LOAD DATA LOCAL INFILE](https://image.slidesharecdn.com/budapestmysqlmeetupmiklosszel-161202080048/85/How-not-to-kill-your-MySQL-infrastructure-20-320.jpg)
![21
mysql> LOAD DATA LOCAL INFILE '/root/.aws/config' INTO TABLE `wp_comments`
(comment_content) ;
Query OK, 5 rows affected, 2 warnings (0.00 sec)
mysql> LOAD DATA LOCAL INFILE '/root/.my.cnf' INTO TABLE `wp_comments`
(comment_content) ;
Query OK, 3 rows affected, 2 warnings (0.00 sec)
10 LOAD DATA LOCAL INFILE
select comment_content from wp_comments;
[..]
| [default]
| output = json
| region = us-west-2
| aws_access_key_id = AKIAxxxxxxxxx
| aws_secret_access_key = S4xxxxxxxxxxxxxxxxxx
| [client]
| user=root
| password=mysqlpassword
|](https://image.slidesharecdn.com/budapestmysqlmeetupmiklosszel-161202080048/85/How-not-to-kill-your-MySQL-infrastructure-21-320.jpg)
How (not) to kill your MySQL infrastructure
- 1. HOW (NOT) TO KILL YOUR MYSQL INFRASTRUCTURE Based on true stories
- 2. 2 • Miklos ‘Mukka’ Szel • From Hungary • Started using Linux in 1997 and MySQL in 2000, used to be a backend developer • Worked for Own Startup, ISP, Walt Disney(ESPN,Cricinfo), PalominoDB/Pythian • Freelance Consultant since 2015 - Edmodo /ME
- 3. 3 Ansible: - name: Install MySQL-Oracle Server yum: pkg={{ item }} state=present with_items: - MySQL-server - MySQL-client Centos/Redhat: yum install MySQL-server Debian/Ubuntu: apt-get install percona-server-server-5.6 1. VERSION MISMATCH - SOFTWARE
- 4. 4 ./pt-slave-find --host 10.77.3.181 -u root--ask-pass Enter password: 10.77.24.181 Version 5.6.32-78.1-log Server ID 134 Uptime 318+06:01:55 (started 2015-11-19T02:42:03) Replication Is not a slave, has 1 slaves connected, is not read_only Filters Binary logging MIXED Slave status Slave mode STRICT InnoDB version 5.6.32-78.1 +- 10.77.25.17 Version 5.6.27-75.0-log Server ID 143 Uptime 167+18:21:03 (started 2016 -04-17T14:22:55) Replication Is a slave, has 0 slaves connected, is read_only Filters Binary logging MIXED Slave status 0 seconds behind, running, no errors Slave mode STRICT Auto-increment increment 2, offset 1 InnoDB version 5.6.27-75.0 1. VERSION MISMATCH - SOFTWARE
- 5. 5 • Monitoring • Check the process list on every server • Scan for port nmap -p 3306 –sV 10.77.3.0/24 Nmap scan report for 10.77.3.238 Host is up (0.00087s latency). PORT STATE SERVICE VERSION 3306/tcp open mysql MySQL 5.6.27-75.0-log 1. VERSION MISMATCH - SOFTWARE
- 6. 6 • Between config file and runtime variables • Between servers in the same replication chain ./pt-config-diff /etc/my.cnf h=localhost 3 config differences Variable /etc/my.cnf c1-adhoc4-i.us-west2 ======================================== innodb_thread_concurrency 0 4 wait_timeout 600 3600 read_only ON OFF https://www.percona.com/doc/percona-toolkit/2.2/pt-config-diff.html 2. VERSION MISMATCH - CONFIG
- 7. 7 • Don’t use methods like removing dots from ip addresses • 10.77.12.3 (1077123) • 10.77.1.23 (1077123) • MySQL truncates the server_id if it’s bigger than the max value for an INT(4294967295) • 192.168.111.222 (192168111222 -> 4294967295) • 192.168.222.222 (192168222222 -> 4294967295) • 4508354421957495439 -> 4294967295 • 12354356476576 -> 4294967295 3. MY.CNF – SERVER_ID
- 8. 8 • Wait_timeout 28800 by default „Warning: mysql_connect() [function.mysql-connect]: User hircsardasql already has more than 'max_user_connections' active connections in /web/hircsarda/hircsarda.hu/wp-includes/wp-db.php on line 1515”” • USE max_user_connections, this will prevent a single app from using all threads(max_connections) gdb -p $pid -ex "set max_connections=3000 --batch 4. MY.CNF – WAIT_TIMEOUT
- 9. 9 “mysqld actually permits max_connections+1 clients to connect. The extra connection is reserved for use by accounts that have the SUPER privilege.” https://dev.mysql.com/doc/refman/5.7/en/too-many-connections.html • GRANT ALL permission comes with SUPER which can write on instances with read_only = ON • You can use super-read-only in Percona Server >5.6 • https://www.percona.com/doc/percona-server/5.6/management/super_read_only.html 5. USER AUDIT
- 10. 10 Logical • mysqldump • mydumper Cold • stop mysql, archive data files Snapshots • LVM(performance overhead) • EBS snapshots Hot or Online • MySQL Enterprise backup(expensive) • Percona XtraBackup 6. BACKUP
- 11. 11 Why EBS snapshots are great for backups/restore: • easy to implement • EC2 uses incremental snapshots • Easy to provision multiple machines Cons • EBSs are network drives, check the throughput limit for your instance • (n disks * throughput)+other nw activity< bandwidth limit • volumes are COLD in the beginning, penalty! http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs -ec2-config.html 6. BACKUP - EBS SNAPSHOTS
- 12. 12 Warming up new 100GB EBS disks - m3.2xlarge EBS optimized instance http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs -prewarm.html 6. BACKUP - EBS SNAPSHOTS Type Warmup Time Warmup Speed SSD ‘io1’ PIOPS 3000 869s 124 MB/s generic purpose ‘gp2’ SSD 300/3000 864s 124 MB/s SSD ‘io1’ PIOPS 1500 1604s 66.8 MB/s Magnetic ‘standard’ no iops 4884s 22.0 MB/s
- 13. 13 “If you access a piece of data that hasn't been loaded yet, the volume immediately downloads the requested data from Amazon S3, and then continues loading the rest of the volume's data in the background. “ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSSnapshots.html EBS types: • gp2 1TB 3.000 IOPS • io1 1TB 20.000 IOPS • magnetic 1TB Warming up the data on 2 threads: 355467264 bytes (355 MB) copied, 159.721 s, 2.2 MB/s 475004928 bytes (475 MB) copied, 159.601 s, 3.0 MB/s Warming up the data on 3 threads: 360710144 bytes (361 MB) copied, 195.47 s, 1.8 MB/s 363855872 bytes (364 MB) copied, 195.47 s, 1.9 MB/s 360710144 bytes (361 MB) copied, 195.471 s, 1.8 MB/s 6. BACKUP - EBS SNAPSHOTS
- 14. 14 Warming up 20GB (or restore from a backup kept on a snapshot): 4302.73 s, 5.0 MB/s = 71 minutes If we want to warm up a 500GB volume = ~ 30 hours!!! 6. BACKUP - RESTORE FROM EBS
- 15. 15 Online open source backup tool from Percona • Parallel • Compression • Enryption • Streaming • Throttle • Doesn’t work with streaming! • Having multiple MyISAM needs longer period when Flush Tables With Read Lock S3 upload/download speed is pretty good: innobackupex --user=root --password=password --stream=xbstream -- encrypt=AES256 --encrypt-key-file=keyfile --parallel=10 /tmp | pigz –p 2|pv –l 20M | aws s3 cp - s3://bucket/filename.gz Do you have (encrypted) offsite backup ? 6. BACKUP - PERCONA XRABACKUP
- 16. 16 • Backups taken with Xtrabackup are consistent across databases • Backups are not consistent across clusters (end time matters) • Consistent restore requires point in time recovery • mysqlbinlog supports --read-from-remote-server starting from MySQL 5.5 • expire_logs_days = 10 by default 6. BACKUP - MYSQLBINLOG
- 17. 17 • Disable atime on the data partition • Disable swap • Allocate most of the RAM to MySQL • This will make it the perfect candidate to kill for the OOM-killer • tmp files goes to /tmp by default • DDLs on big tables can consume it! [mysqld] open_files_limit = 10000 /etc/security/limits.conf: * hard nofile 10000 * soft nofile 10000 7. OS
- 18. 18 • “1 pageload = 1 bad query” can be deadly when aggregating a lot of data even on a small table Monitor: • pt-query-digest • Anemometer • Percona Monitoring and Management Query Analyzer • Vividcortex Handle: • pt-kill • ProxySQL • Memcached 8 - QUERIES
- 19. 19 • INT: Most typical column types for Primary Keys • UNSIGNED NOT NULL • A rolled back TX still increases the AUTO_INCREMENT https://github.com/RickPizzi/pztools/blob/master/findmax.sh 9 SCHEMA – MAX INTEGER CHECK
- 20. 20 “In a Web environment where the clients are connecting from a Web server, a user could use LOAD DATA LOCAL to read any files that the Web server process has read access to (assuming that a user could run any command against the SQL server).” http://dev.mysql.com/doc/refman/5.6/en/load-data-local.html [root@hostdb~]# mysql -e "SHOW GLOBAL VARIABLES LIKE 'local_infile'" +---------------+-------+ | Variable_name | Value | +---------------+-------+ | local_infile | ON | +---------------+-------+ [root@hostdb ~]# mysql -e "show grants for wordpress@%” GRANT USAGE ON *.* TO ’wordpress@'localhost' IDENTIFIED BY PASSWORD ‘xxx’ GRANT SELECT, INSERT ON `wordpress`.* TO ’wordpress'@’%’ 10 LOAD DATA LOCAL INFILE
- 21. 21 mysql> LOAD DATA LOCAL INFILE '/root/.aws/config' INTO TABLE `wp_comments` (comment_content) ; Query OK, 5 rows affected, 2 warnings (0.00 sec) mysql> LOAD DATA LOCAL INFILE '/root/.my.cnf' INTO TABLE `wp_comments` (comment_content) ; Query OK, 3 rows affected, 2 warnings (0.00 sec) 10 LOAD DATA LOCAL INFILE select comment_content from wp_comments; [..] | [default] | output = json | region = us-west-2 | aws_access_key_id = AKIAxxxxxxxxx | aws_secret_access_key = S4xxxxxxxxxxxxxxxxxx | [client] | user=root | password=mysqlpassword |
- 22. 22 10 LOAD DATA LOCAL INFILE
- 23. 23 QUESTIONS?