MySQL At Mastercard - 2018 MySQL Days

MySQL At Mastercard - 2018 MySQL Days

• 1.
  James Sturrock, OperationsManager February 15, 2018 MySQL at Mastercard
• 2.
  ©2018Mastercard.ProprietaryandConfidential. 2JANUARY 19, 2018 • Weemploy over 13,000 people worldwide • One of the most recognizable brands in the world • Our vision is “a World Beyond Cash™“ • Our mission is: Every day, everywhere, we use our technology and expertise to make payments safe, simple and smart About us
• 3.
  ©2018Mastercard.ProprietaryandConfidential. 3JANUARY 19, 2018 • JamesSturrock • Operations Manager • With Mastercard for over 7 years • Part of the Payment Gateway Services division Who am I?
• 4.
  ©2018Mastercard.ProprietaryandConfidential. 4JANUARY 19, 2018 • OurPayment Gateways processes financial transitions for merchants globally, across a variety of sectors such as: – Ecommerce – major online brands – Airlines – Cardholder Present – pub and restaurant chains, high-street stores etc • We bridge the gap between your bank authorizing a payment and the merchant receiving the funds • Due to the nature of our business, operationally we must focus on maintaining three key objectives: 1.  Security – we handle peoples personal data as well as cardholder data 2.  Stability – huge financial and reputational cost to merchants if people can’t buy things 3.  Scalability – we need to ensure we can always cope with unexpected surges in traffic (Black Friday, Sporting Events etc) What we do
• 5.
  ©2018Mastercard.ProprietaryandConfidential. 5JANUARY 19, 2018 • MySQLwas a good fit for our Linux environment and open source approach • Flexibility to use it in whatever way you need to • Stability, MySQL is almost never the problem! • Simplicity, MySQL can be used as simply or complex as you want Why MySQL
• 6.
  ©2018Mastercard.ProprietaryandConfidential. 6JANUARY 19, 2018 – EnterpriseMonitor – Enterprise Authentication – Enterprise Scalability • These are all products which we are now using or evaluating! ensure that 3rd party vendor releases patches for security vulnerabilities in a timely manner. This can’t be guaranteed from the open source community. • Traditionally we have failed to take advantage of the full suite of Enterprise tools such as: – Enterprise Monitor – Enterprise Authentication – Enterprise Scalability • These are all products which we are now using or evaluating! Why MySQL Enterprise
• 7.
  ©2018Mastercard.ProprietaryandConfidential. 7JANUARY 19, 2018 • Around40 servers running MySQL • Anywhere between 1 and 12 running instances of MySQL on a single machine • Vast majority are running MySQL Enterprise Edition • All running on Red Hat Enterprise Linux (64 bit) General Overview Hardware Presentation Operating System Database
• 8.
  ©2018Mastercard.ProprietaryandConfidential. 8JANUARY 19, 2018 • Weuse MySQL Enterprise Monitor along with some legacy in house log monitoring tools • Nagios is used for system level monitoring as well as basic MySQL checks (such as are instances running, is replication stalled, how far behind is replication) • Grafana used for monitoring “user experience” of the platform, often the best indicator if there is an actual problem Monitoring
• 9.
  ©2018Mastercard.ProprietaryandConfidential. 9JANUARY 19, 2018 • Classic“upside down tree” replication chain, A single read/ write master replicates down the chain one by one • Having too many slaves replicating off one master can slow down the master! • When carrying out a failover, there is much less remastering to be done • Allows for us to carry out major schema upgrades on all slaves then failover with no downtime Replication Host 1 Host 2 Host 3 Host 4 Host X
• 10.
  ©2018Mastercard.ProprietaryandConfidential. 10JANUARY 19, 2018 • Ourreplication structure and database design doesn’t give us high availability out of the box, there is still a single read/write master • Red Hat Cluster Suite layered on top of MySQL to provide automated failure detection and failover • Built in clustering and quorum functionality • Essentially manages a VIP and ensures it is running on the correct host • Custom health checks are run by the cluster software to determine if a MySQL instance or the entire host has crashed High Availability
• 11.
  ©2018Mastercard.ProprietaryandConfidential. 11JANUARY 19, 2018 • Replicationlag during peak processing periods – Potentially could be fixed by parallel replication • Cumbersome process to isolate databases for Kernel patching – Potentially could be fixed by using GTID replication – Potentially could be fixed by using tools like salt/fabric to automate • Length of time for a cold started database to become “hot” and fast enough to use – Potentially could be fixed by migrating to InnoDB Current Challenges…
• 12.
  ©2018Mastercard.ProprietaryandConfidential. 12JANUARY 19, 2018 • Complianceconsiderations: – MySQL 8? – RHEL 7? • Performance/usability improvements: – Implement GTID replication – Test parallel replication • Tighter integration of MySQL into our “DevOps” toolkit – Puppet – Fabric/salt The Future…..
• 13.
  ©2018Mastercard.ProprietaryandConfidential. 13JANUARY 19, 2018 • UseSSD disks where possible! • Always test schema changes on a dataset equivalent to production (and test the rollback as well as the rollout) • You can never have too many monitoring metrics across your platform • Having a production like stress test environment is invaluable • Historically MySQL has not been the problem, hardware and software bottlenecks are more common • Disconnect database connections when reaching out to 3rd party services (avoids rapidly reaching the max_connection limit) Lessons Learned…