SlideShare a Scribd company logo

Zendcon scaling magento

Download as PPTX, PDF
Mathew Beane
Mathew Beane

Magento is an open-source e-commerce platform built on PHP. The document discusses various ways to optimize and scale a Magento application, including: - Optimizing server settings like PHP-FPM, Nginx, Redis, and MySQL configurations - Ensuring the Magento application code is clean and optimized before scaling - Having a rigorous development process in place including continuous integration, testing, code reviews, and documentation

Internet
1 of 122
Downloaded 41 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
Scaling Magento Reaching Peak Performance
2 Mathew Beane @aepod Director of Systems Engineering - Robofirm Zend Z-Team Volunteer – Magento Division Family member – 3 Kids and a Wife Magento Certified Developer
3 Joshua Warren: Magento 2 - An Introduction to a Modern PHP-Based System Catch this talk Wednesday at 4pm in The Joint • Open-source e-commerce platform • PHP based application • Core utilizes Zend Framework 1 • Very flexible, built to modify • Extremely scalable, supports huge stores • Market leader and still growing • Magento 2 is right around the corner • It’s is in merchant beta now • http://magento.com/developers/magento2 Magento: What is it? Mathew Beane: Magento101: Getting Started with Magento Development Catch this talk Tuesday at 11:15am in Artist 4
4 Magento is the market leader in E-Commerce
5 E-Commerce Growth
6 Shopping Is Moving Online Be prepared to scale your clients Magento websites because growth is the norm.
7 E-Commerce Growth Is Worldwide
8 https://github.com/aepod/magento-digitalocean-sandbox • Uses Puppet to build out cluster • To setup the initial test checkout readme in project • Initial release standard (n)webnodes + single db config • Monolith server has Web, PHP, DB, Redis all-in-one • We will come back to this later Cluster Configuration Overview
9 • Introduction • Magento Application: Preparing and Optimizing • Magento Cluster Architecture: Examine Typical Layouts • Magento Digital Ocean Sandbox: Demonstrating Clustering • Magento Performance Toolkit: Measuring Performance Benchmarks • Magento Digital Ocean Sandbox: Guided experiments • Conclusion: Open Q & A Todays Plan
10 Preparing and Optimizing
11 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
12 • Optimize sysctl.conf: Increase limits for the servers based on Memory and CPU. • Modify Nginx settings:  Setup PHP FPM with TCP properly  Optimize Worker Connections  Tune up caching and other settings • Optimize PHP FPM  Process Manager Optimization  APC / Zend Opcode Cache  Tune up other settings • Optimize Redis • Optimize MySQL Optimizing Overview Thijs Feryn: Making Magento Go Fast Catch this talk Tuesday at 2:45PM in Artist 2
13 • Sysctl is an interface for configuring the Linux Kernel. • The following examples are some of the settings you may change in the sysctl.conf. • Remember to backup your configurations. • After changing you run: sysctl –p to update your kernel settings. • Always test after changing values, quantify your results against previous tests. See also: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system- tuning.html Optimizing Hints – Linux Kernel
14 # Controls the maximum shared segment size, in bytes (64gb) kernel.shmmax = 68719476736 # Controls the maximum number of shared memory segments, in pages (16gb) kernel.shmall = 4294967296 #controls the number of process identifers that can be created kernel.pid_max = 2097152 # Number of open files handles fs.file-max = 2097152 # Defaults to 65536 – Deals with malloc maps. vm.max_map_count = 262144 • You can find a lot more tuning information by looking up the parameters and learning more about them. Optimizing Hints – Linux Kernel
15 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.rmem_default = 1048576 net.core.wmem_default = 1048576 net.core.netdev_max_backlog = 65536 net.core.somaxconn = 65536 net.ipv4.tcp_rmem = 4096 1048576 16777216 net.ipv4.tcp_wmem = 4096 1048576 16777216 net.ipv4.tcp_max_syn_backlog = 65536 net.core.somaxconn is the least trivial, and typically set very low causing the following: “apr_socket_recv: Connection reset by peer” and “connect() … failed“. Optimizing Hints – Linux Kernel
16 location ~ .php$ { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /var/www/html/$fastcgi_script_name; fastcgi_param MAGE_IS_DEVELOPER_MODE 0; fastcgi_param PHP_VALUE error_log=/var/log/php-errors.log; include fastcgi_params; fastcgi_cache off; fastcgi_buffer_size 128k; fastcgi_buffers 256 16k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_read_timeout 900s; } Optimizing Hints – Nginx
17 Battle Hardened Magento Nginx Config: https://gist.github.com/gwillem/cd5ae6845fa33aa0d481 • worker considerations:  Worker Processes = Total # CPU Cores  Worker Connections: ½ of ulimit –n  Max Connections = worker processes * worker connections  Worker_rlimit_nofile safe value = 2 * Max Connections  keepalive_timeout mitigates overage by trading for latency Optimizing Hints – Nginx
18 # Media: images, icons, video, audio, HTC location ~* .(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ { expires 1M; access_log off; add_header Cache-Control "public"; } •Set long expires dates on media assets. Optimizing Hints – Nginx
19 curl -X GET -I www.idealphp.com:80/skin/frontend/rwd/default/images/logo.gif HTTP/1.1 200 OK Server: nginx/1.0.15 Date: Tue, 14 Apr 2015 17:56:32 GMT Content-Type: image/gif Content-Length: 2320 Last-Modified: Tue, 14 Apr 2015 00:30:08 GMT X-Robots-Tag: noindex, nofollow, nosnippet, noarchive Accept-Ranges: bytes Cache-Control: max-age=31536000 Cache-control: public • Test your headers using curl or other tools Optimizing Hints – Nginx
20 • PHP 5.4 - Minimum • PHP 5.5 – Great, better performance if you have compatibility • PHP 5.6/5.7 - Even Better, although not supported • PHP 7 / HHVM - Not until Magento 2 – Unless your Daniel Slooth • Use TCP instead of sockets • Test with Zend OpCache instead of APC • Turn off APC Stat/Zend Revalidate  apc.stat = 0  opcache.revalidate_freq=0 Optimizing Hints – PHP-FPM
21 • Tune Process Manager  pm: ondemand  pm.max_children: about 6-8 per CPU *more later  pm.max_requests: 10000 *php threads should die for garbage collection  listen.backlog: set to 64000 (typically set to -1 or 128) Listen backlog is limited by net.ipv4.tcp_max_syn_backlog *only do after sytsctl.conf optimization Optimizing Hints – PHP-FPM
22 pm.max_children will limit the amount of memory php uses overall  Calculate the memory a thread is using while under load: ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%sn", sum/NR/1024,"M") }'  Use the total amount of memory you want php to consume: pm.max_children = Total RAM dedicated to the web server / Max child process size Based on this, you can set your max_children based on memory instead of the CPU metric used earlier. This will allow for more memory consumption in production environment. Be sure to test with higher values, more is not always better. Further Reading: http://myshell.co.uk/index.php/adjusting-child-processes-for-php-fpm-nginx/ Optimizing Hints – PHP-FPM
23 • Redis is the preferred cache for Magento  sysctl ulimit values affect the redis maxclients  Use three separate Redis instances with a single DB in each, on different ports  tcp-backlog can be increased, adjust sysctl somaxconn and tcp_max_syn_backlog  maxmemory can be set very low typically. • Using Sentinel and twemproxy you can horizontally scale  Clustering redis to eliminate all single threaded bottlenecks and single points of failure  Requires orchestration, and Magento application changes • Don’t forget to update Cm_Redis to newest version (Magento lags behind) • At Rackspace or AWS: Use Object Rocket https://objectrocket.com/ for Redis SaaS Optimizing Hints – Redis
24 • Turn off MySQL Query Caching – This is single threaded and not of use with Magento (* Before MySQL 5.7) • Innodb settings can be tweaked to consume more memory Selected Sample Configuration Settings for a 32GB Dual Hex-Core Server myisam_sort_buffer_size = 64M key_buffer_size = 256M join_buffer_size = 4M read_buffer_size = 4M read_rnd_buffer_size = 4M sort_buffer_size = 8M table_open_cache = 8192 thread_cache_size = 512 tmp_table_size = 384M max_heap_table_size = 384M max_allowed_packet = 1024M query_cache_type=0 query_cache_size=0 Optimizing Hints – MySQL innodb_thread_concurrency = 24 innodb_buffer_pool_size = 16G innodb_log_file_size = 384M innodb_log_buffer_size=24M innodb_additional_mem_pool_size = 16M innodb_io_capacity = 800 innodb_concurrency_tickets = 900 innodb_flush_neighbor_pages=cont innodb_lock_wait_timeout=75 innodb_flush_method=O_DIRECT
25 • Clustering MySQL for Magento Requires all tables to be converted to InnoDB • catalog_full_text_search requires MyISAM, unless your running MySQL 5.7 • When clustering always use identical hardware for each server • Master/Master is not supported by Magento • You can specify read servers and even load balance them • It is unusual for Magento to require a true MySQL Cluster, as it is typically not the bottleneck Optimizing Hints – MySQL
26 • Varnish is great, however your application must be suitable  Adding Varnish to a slow site is like putting a bandage on a neck wound  Varnish complicates logging and request tracking throughout the application • CDNs can have a great impact on end-user performance  Reduction in requests is also beneficial to the web server  Typically one of the cheapest/best moves to make • Always test any changes you make Optimizing Hints – Other
27 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
28 What You Don’t Want To Scale A very good un-cached response, good ready to scale. A typical bad response, scaling this will only cause you heartache.
29 • DOUBLE CHECK YOUR PATCHES:  https://www.magereport.com/ • Triple check your patches:  Patch Grid: https://goo.gl/X6iu8O by @knowj Magento – Apply Security Patches
30 Keep a clean core. If you inherit a project, one of the first things you should do is clean up core. This means you have not changed any default files including templates, layouts and code. Magento – “Clean Core” What is a clean core: • No changes to any file provided by Magento • No changes to app/code/core • No changes to any other core location • … even the index.php • Exception: PATCHES! Ben Marks, Magento Community Evangelist @benmarks (Protector of the Core)
31 • Find and eliminate your core changes with: https://github.com/AOEpeople/mpmd Magento Project Mess Detector by Fabrizio Branca Magento Project Mess Detector Fabrizio Branca: How to Build a Pure Evil Magento Module Catch this talk Tuesday at 4pm in Artist 3
32 Fabrizio Branca’s work on cleaning up Magento projects: http://fbrnc.net/blog/2014/12/magento-update In this article he goes into depth on the process of separating the core files, and using Modman and composer to package everything. Tegan Snyder's work on building modman configurations for extensions out of existing source: https://github.com/tegansnyder/meff This will allow you to generate modman files for your extensions, which may be handy in the process of cleaning up. Refactoring Bad Magento Projects – Further Study
33 • Use Z-Ray to profile and tune your Magento Application • Z-Ray allows you to see every request, and many details about the request. • It’s easy to install, free to try and really amazing. • It can be used to debug Mobile browsers through the Zend Server backend • Z-Ray No longer requires Zend Server: http://www.zend.com/en/products/z-ray/z-ray-preview Optimizing Your Application with Zend Z-Ray Mathew Beane: Z-Ray & Magento: A Customizable Development Tool Belt Catch this talk Wednesday at 11am in Artist 4
34 • Demo: http://serverdemo.zend.com/magento/ • Blog Article: http://aepod.com/debug-magento-using-zend-zray/ Optimizing Your Application with Zend Z-Ray
35 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
36 • Should apply equally to the hardware architecture as to the software application. • Every project should have a continuous development cycle in place, and ensure that it is well documented and easy to follow for all of the people involved. Development Cycle
37 • Design: Plan our your sprint, stick to Agile. • Develop: Toward a release on a schedule  Test: Build PHPUnit and Selenium tests into your process  Code Review: Pull requests and code review EVERYTHING.  User Acceptance Tests: Make sure the client is happy first. • Deploy: Package, test and deploy.  Deployments: Automate, and make the production servers hands off. Development Cycle - Simplified
38  Goal based system that encourages teamwork.  Daily communications with all involved in project.  Flexible and focused on delivery.  Encourages Teamwork and Accountability. Development Cycle – Agile / Scrum
39 • Keep a clean core  Never edit core files. Ensure that you can see any core changes in production. • Release / Feature Branching  Choose a branching methodology and build around it. • Use Pull Request  Code review increases the quality of your code. Make pull requests part of your workflow. Development Cycle – Branching
40 http://nvie.com/posts/a-successful-git-branching-model/ Successful Git Branching Model • Use Tagging as you release to master • Types of branches: • Master: Tagged versions, what's on production. • Develop: What will be merged into master with the next release. • Release: Preparing for release. Allows for last minute fixes, testing and prepping. • Feature: Usually checked out by developers to work on features. • Hotfix: Fixes aimed for unplanned production releases.
41 Build testing into all of your development cycles. Development Cycle – Testing Developers are working on testing with Magento 1 BDD: https://speakerdeck.com/alistairstead/bdd-with-magento BeHat: https://github.com/MageTest/BehatMage PHPSpec: https://github.com/MageTest/MageSpec Magento 2 has a built in Test Automation Framework ( PHPUnit,Selenium ) https://wiki.magento.com/display/MAGE2DOC/Magento+Automated+Testing+Guide https://github.com/magento/taf https://alankent.wordpress.com/2014/06/28/magento-2-test-automation/
42 • Use a deployment tool, and LOCK DOWN production. No one should ever have to touch an application file manually for any reason. • If you can extend this practice to your OS and server stack you now have fully engaged DevOps. Do Not Edit on Production
43 Development Cycle – Testing (Example of Fail) After introducing SOLR indexing, testing showed serious issues as it indexed.
44 • You must be able to package your application in order to deploy it to a cluster.  Deployment infrastructure may determine what packaging you will need to use, for instance one of the following: • Zend Server offers Application Packaging and Deployment • A simple “golden copy” (tgz or flat) that is pushed to web nodes. • RPM or other proprietary packaging tool.  Use Composer and/or modman to help maintain your application packages.  Composer can even be used to install Magento Core  Make sure your build process is repeatable and testable. Development Cycle – Application Packaging
45 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
46 • When deploying an application to a cluster of application servers, a deployment tool is a requirement. • There are many choice, here are a couple typical deployment tools.  Capistrano: Written in Ruby but well accepted and great to work with  Jenkins: Another example of Deployment Automation  Bamboo: Part of the Atlassian stack also includes testing and other features.  Roll Your Own: This is more common, using bash scripts and other tools you can build a project deployment tool fairly easily. Deployment Tools – Typical Software
47 • Integrated with code versioning • Supports Multi-Server • Supports the Application  Handles Maintenance Mode automatically  Runs Installers  Clears Caches • Low Downtime • Rollback Procedure (a/b switch) • Trigger events such as shell scripts, or CDN clearing Nice to have – May not be required • Integrated testing • Packaging • Integration to GUI Interfaces Deployment Tools – Some Requirements
48 I highly suggest researching Fabrizio Branca’s work on the subject: http://www.slideshare.net/aoepeople/rock-solid-magento Fabrizio has a ton of articles and slideshares on the subject. Also check out Joshua Warren’s slides, he has several really informative presentations on the subject. http://www.slideshare.net/joshuaswarren/ Deployment Tools – Further Study Fabrizio Branca: Rock-solid Magento Development and Deployment Workflows Catch this talk Wednesday at 2:45PM in Artist 5
49 • Automate server deployments:  Puppet  Chef  Ansible  Salt • Centralize configurations • Automated updates and server onlining process • Take out the human factor! Deployment Tools - Server Deployments
50 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
51 Load Testing and Metrics "Don't rate potential over performance." - Jim Fassel Blaze Meter Using Blazemeter you can easily build repeatable tests, with very nice graphs. (based on JMeter) Gatling http://gatling.io/ On par with Blazemeter. JMeter Very effective, without having to purchase a SaaS Siege Can be used minimally to simulate some types of load.
52 https://github.com/magento/magento-performance-toolkit The Magento Performance Toolkit is a set of automated tools that enables you to quickly and consistently measure the application performance of Magento using Apache JMeter. The toolkit includes(sic requires), a PHP script for generating a test database and a JMX script for the performance measurement. From: Magento Performance Toolkit Users Guide More Reading: http://aepod.com/using-the-magento-performance-toolkit/ Magento Performance Toolkit
53 The Magento Performance Toolkit can be crafted to pull data from production sites, although its complicated and you may want to cover your site using your own JMeter tests. Gathering the steps for a client site typically takes a day or more depending on the complexity of the site design, and the checkout. Developing and debugging this can be a real time sink depending on how far down the rabbit hole you want to go. JMeter Tests for “Normal Sites”
54 siege –t 5M –b –c 5 http://www.idealphp.com/ This will start up 5 threads and hammer the URL for 5 minutes. Siege is very useful, but limited. It’s easy to put a lot of traffic on a box with it, but the traffic will be limited to GET requests. You can pull a list of URL’s from the Magento sitemap and run through random requests, this can even be used to do some pseudo-warming of the cache. Using Siege
55 • Analyze software in Real-time • Collect Data  Application Data  Server Data  Browser Data  Magento Data * • Synthetics provides NrSQL • Alerts * Magento data provided via Magento Extension: http://www.magentocommerce.com/magento-connect/new-relic-reporting-29133.html New Relic - Introduction
56 New Relic – Production
57 New Relic - Testing
58 1. Prepare a downtime:  Forward site to a non-magento maintenance page  Exclude Testing IP – So tests can still proceed  Disable Cron 2. Copy Production Database 3. Point Magento at the new copied DB 4. Modify Magento Admin Settings in copied DB  Use Test Payment Gateway  Disable Transaction Emails 5. Test against Copied DB 6. After Testing is complete:  Point site at original DB  Put site back up on live  Enable Cron Using this method, you can test against production on production. You will need to bring the site down during the time period, but if this is done at a low point in traffic it causes little to no stress to the customers. Load Testing Production
59 Your ready to scale Magento Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place • Next Up: Discussion about typical cluster architecture and components
60 Magento Cluster Architecture
61 • Load Balanced – Front End Webservers • Admin / Backend Server Separated • Single MySQL Database Master, with Slaves Magento Cluster Architecture – Typical Layout
62 • Web Servers  Apache / Nginx  PHP 5.4 + (depends on Magento Version)  Multiple Webservers – Easy to cluster  Varnish Can be used as a Reverse Proxy • Nexcess Turpentine is the standard Magento Extension • Magento2 has Varnish support out of the box Magento Cluster Architecture - Components
63 • Database  MySQL / Percona  Master / Slave • Slaves are typically used for reporting or Hotswap for failure • Can be set to read servers for application, but it requires some work  Solr / Elastic Search • Typically run on the Database or Backend Server • Can power Search or other Magento business logic  Magento 2 introduces Partitioned Databases Magento Cluster Architecture - Components
64 • Other Components  File Server • NFS – This is by far the most common • NAS – Can be used on high end clients  Redis / Memcache  Deployment Tools  Monitoring  ELK – Log aggregation across servers  Puppet / Chef Magento Cluster Architecture - Components
65 Breaking Points, where you need to adjust first. 1. PHP – More web nodes, easy to do and very performant. 2. Filesystem – Use a CDN to offset static asset traffic. 3. Redis – Cluster Redis using sentinel/nutcracker. 4. Database – Look at read server clustering At some point adding Varnish should be considered as well. Varnish requires a clean application with some thought as to which blocks can be loaded dynamically via AJAX. Magento Cluster Architecture - Bottlenecks Thijs Feryn: Making Magento Go Fast Catch this talk Tuesday at 2:45PM in Artist 2
66 Scaling Implementation – A Scoring System Concern Score (1-5) What is the simplicity of the solution? How difficult is it to support? Is it extensible, how broadly does it apply? Will it break often? How easy is it to update? Is it Expensive? Concern Yes / No Does it require specialized services? (os level services, not typically installed) Does it require Magento application changes? Does it create a single point of failure?
67 • Simple to Load Balance, most sites start here • Challenges include the following:  Session Management: should be a no brainer if your using redis for sessions  Shared file System: Use NFS for media, keep all code local to web servers  Fencing: easily solved with either software or hardware  Log Collection: ELK, Rsyslog or Splunk • Keep the system builds simple, repeatable and automate if possible Scaling – Web Server Concerns
68 • How to automate:  Create an complete image to work from - include puppet so it can pull from a puppet master  The puppet master spins up your webserver stack  You have your deployment process in place, so plant the application and spin up the node • Be prepared to lose nodes, the more you have the more likely failure is • When a node runs amok, you must be prepared to kill it dead Scaling – Web Server Concerns (Automation)
69 Best practice for sharing files: Use NFS • Most other options are not viable. (glusterfs, iscsi, and so on) • NFS will not be the bottleneck if your application is configured correctly • Share only media/ to avoid bottlenecks • Client requirements may drive cluster toward a NAS Network Attached Storage (NAS) • Typically only on very large sites • Expensive but very reliable and robust • Entry level has very high bar (typically 20+ drives) Scaling - File systems
70 • Share specific folders via symbolic links  Example: [docroot]/media -> /var/magento/shared/media • Mount /var/magento/shared/[directory] via NFS, typically from backend server  Maintaining these when deploying is easy, just remake symbolic links or copy them • Can also be used for local content  Example: [docroot]/var -> /var/media/local/var  Not mounted via NFS, writes local var files for this server only Scaling – File Systems and Symbolic Links
71 Scaling – Redis Clustering “Redis clustering using sentinel is easy to set up. Adding twemproxy allows for a highly scalable Redis cluster and you get auto fail over and a ton of other benefits with this configuration. This arrangement can also remove your Redis single point of failure.” http://aepod.com/clustering-magento-redis-caching-with-sentinel-keepalived-twemproxy-and-twemproxy-agent/ • Easy to setup… well not really • Very Complex, requires orchestration • Sentinel Monitors Redis clusters • twemproxy handles sharding • twemproxy agent monitors sentinel • Very robust when setup, nothing is single threaded, everything is HA and the speed…. • Can be transparent to Magento despite the complexity
72 • Requires several layers of applications  Load Balancer: Load balances redis traffic to the twemproxy servers  Twemproxy: proxy server, typically setup as a cluster of 2 servers (can be mixed on other servers)  Nutcracker Agent: Connects twemproxy to sentinel to keep track of shards of servers  Sentinel: Monitors the redis instances and maintains availability  Redis: Dozens of redis servers, sharded, maintained and fenced by Sentinel and nutcracker  VERY Complex Redis (Sentinel / twemproxy) Thijs Feryn: Making Magento Go Fast Catch this talk Tuesday at 2:45PM in Artist 2
73 • Should be considered a low value target, except for High Availability concerns • Magento typically is not bottlenecked by MySQL • Using Percona XtraDB you can add read slaves • Magento only supports a single Master/Write server, with multiple read servers • Setting up load balanced READ db servers is not overly difficult, but doesn’t offer much if any performance benefits • Cluster DB will require more advanced fencing • Typical production sites do not use load balanced or clustered MySQL Scaling - MySQL
74 • Percona XtraDB to cluster MySQL • Percona XtraBackup for duplicating and rebuilding nodes • Percona Toolkit to help debug any issues your running into • Difficult to scale Write Servers • Scale out your read servers as needed, but MySQL reads are rarely the bottleneck • Typically Slave server is used for backup and hot swap, NOT clustering. A couple quick tips: • Not all tables in Magento are InnoDB, converting the MyISAM and Memory tables is OK • You will need to be able to kill read servers and refresh • Use your Master server as a read server in the load balancer pool, when you kill all your read servers, it can fall back to master. Database Servers (Percona)
75 • Load Balancer Components  Hardware Load Balancer  HAProxy Software Load Balancer • Database Components  Sentinel / Nutcracker  Percona XtraDB Tools  Clusterix or ScaleARC for High End Magento Cluster Architecture – Additional Tools
76 • HAProxy: a free fast and reliable solution featuring high availability, load balancing and proxy for TCP and HTTP-based applications. • HAProxy can be used for web servers, database servers, Redis and any other TCP based application. • Easy to configure using a puppet module • Can be clustered easily for high availability concerns • Handles L4 Load Balancing if you want to do fancy multi- application environments • Built in stats interface Load Balancers – Software / HAProxy SOFTWARE
77 Hardware Load Balancers • F5 Hardware load balancers are a standard • Brocade is another common brand • Rackspace offers a very easy to use web interface to maintain a hybrid infrastructure • Hardware load balancers offer a turn-key mature solution. • Typically Managed Service, with very hands off approach. HYBRID Hybrid Load Balancers • AWS Elastic Load Balancers • Rackspace Cloud Load Balancers • Cloudflare • Many others… Load Balancers – Hardware & Hybrid
78 Load Balancers – Making a choice • Budget concerns will drive this decision • Hosting Choices will affect availability, costs and toolsets • Start locally with HAProxy and build test clusters using vagrant • HAProxy can still be used, even with a hardware load balancer in place.
79 • Reverse Proxy: Varnish or Nginx you can and should cluster these • Puppet likes to be clustered: Mastering Puppet by Thomas Uphill is a great reference • Monitoring Tools: You’re running these single threaded, what if the monitoring server fails? • Warehouse Integrations, ERP Layer Stuff, everything may need to be multi-threaded • Search back ends such as solr or elasticsearch • Consider all of the parts of the business that surrounds the Magento Application and try to encapsulate them Scaling - Others
80 • Turpentine is the standard go to extension for Magento to tie to Varnish  Handles ESI  Handles Dealing with updating VCL  http://www.magentocommerce.com/magento-connect/turpentine-varnish-cache.html • If on Enterprise disable FPC, CE disable lesti_fpc • https://github.com/nexcess/magento-turpentine/wiki/Installation • Typically Varnish will run on each render box, and proxy to the local nginx. • Issues will typically show up as over caching of blocks or other bits of front end page • SSL is NOT supported by vagrant, terminate it at your load balancer Reverse Proxy (Varnish) Scaling Concerns
81 • NGINX is a great reverse proxy and load balancer • http://nginx.com/resources/admin-guide/reverse-proxy/ • Can be used to effectively buffer and split requests to different cluster servers Reverse Proxy (Nginx) Scaling Concerns
82 • Insert puzzle building analogy joke here: http://www.wikihow.com/Assemble-Jigsaw-Puzzles • Each hosting environment has its own quirks and add on top of that the business logic requirements you will almost always have a unique infrastructure for every client • Build small pieces and work them into the larger picture, you can get a lot of performance with a few minor changes. • Test everything you do, keep detailed notes on the configurations and compare against the previous tests Auto-Scaling Strategy
83 Magento Performance Toolkit
84 https://github.com/magento/magento-performance-toolkit The Magento Performance Toolkit is a set of automated tools that enables you to quickly and consistently measure the application performance of Magento using Apache JMeter. The toolkit includes(sic requires), a PHP script for generating a test database and a JMX script for the performance measurement. From: Magento Performance Toolkit Users Guide More Reading: http://aepod.com/using-the-magento-performance-toolkit/ Magento CE 1.9.1.0 Ready Version: https://github.com/aepod/magento-performance-toolkit Magento Performance Toolkit
85 • Importing can take a long time, and may require changes to php and MySQL settings • If it fails, it fails hard. Restart with a fresh DB • You can modify the profiles to better suite your testing requirements • Using the toolkit requires putting it in the dev/tools/performance_toolkit/ directory in the Magento install you want to use it on • The scripts requires this exact directory, due to includes and autoloading • Oftentimes will require you to remove the tmp dir inside the toolkit Magento Performance Toolkit – generate.php
86 Magento Performance Toolkit – Generate Profiles • You can create your own profiles by copying and modifying the XML in profiles/ • Large and X-Large take extremely long times to import and require changes to the PHP and MySQL settings and it still likes to fail hard. • After it completes import, double check the following • Front end still up and running and shows the framework categories • Admin and Frontend logs in correctly • Indexes and Caches rebuild correctly
87 Modify the tests, learn how they work and dig into errors you may be having running them. • Easy to use • Debugging requirement • Load/Save XML • Can crash on windows Magento Performance Toolkit – jMeter GUI
88 ./jmeter -n -t ~/benchmark-default.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=30 - Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_user="admin" - Jadmin_password=“adminpass"; • Host: host to check against • Base Path: for building the url • User: Roughly the # of threads (more on next slid) • Ramp Period: How long to take starting up • Report Save Path: Where it saves the summary • Loops: How many times it will try to loop through (except if it runs out of customers) • Admin User/Pass: The admin user and pass for the admin site.  This is expected to be at http://www.idealphp.com/admin/ • Any JMeter Attribute from top level can be set from command line using –Jattribute= Magento Performance Toolkit – JMeter Configuration
89 Magento Performance Toolkit – JMeter Test Results
90 The JMeter tests use 4 thread groups to accommodate the testing. • Category Product Browsing(30%): Opens the homepage, goes to a category, opens up 2 simple products and then a configurable. • View Product Add To Cart(62%): Same as Category browsing however it adds each of the items to the cart in addition to viewing them. • Guest Checkout (4%): Same as View/Add to Cart, in addition it runs through the checkout as a guest user. • Customer Checkout (4%): Same as View/Add to Cart, in addition it runs through the checkout as a logged in customer. Thread groups create threads based on the number of users and the above percentages(by default). users * group_percent / 100 • This result is rounded down, and can be 0 for checkouts, making no checkout threads. • You can set EACH group independently, and they do not need to add up to 100% or less in totality. Magento Performance Toolkit –Users Demystified
91 Magento Performance Toolkit – More Attributes Any JMeter Attribute from top level can be set from command line using –Jattribute= Customer and Admin Password must match what was used during the generate. Orders are tricky and used to calculate “users” which are used within the thread groups. See jmeter test setup Thread Group -> validate properties and count users
92 1. Setup Thread Groups 1. Extracts Categories from Top Menu 2. Searches for Simple Products, builds list 3. Searches for Config Products, builds list 4. Logs in on the Admin 5. Grabs a list of customers from the customer grid 6. Builds users count which is used by the threads. 2. Loops through Each Thread Group 1. Category Product Browsing 2. Product Browsing and add items to cart 3. Guest Checkout 4. Customer Checkout 3. Teardown 1. Just removes some stats, but you could extend this spot for some killer stats. Magento Performance Toolkit –JMeter Test Plan
93 Using Newrelic During JMeter tests
94 Using Newrelic During JMeter tests
95 Magento Online Customers
96 ./jmeter -n -t ~/benchmark-default.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=10 -Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_password="5678password" -Jadmin_user=admin -Jview_product_add_to_cart_percent=0 -Jview_catalog_percent=0 -Jcustomer_checkout_percent=0 -Jguest_checkout_percent=100 (This can be done in the JMX as well) Non-Trivial Settings: View Catalog : 0% View Product Add to Cart : 0% Customer Checkout: 0% Guest Checkout: 100% This test will slam orders through as fast as it can. Using it is as simple as recording the number of orders at the start and finish of a test, and then calculating the number of orders per hour on that. Orders Per Hour = (3600 / Time to Complete) * (Ending Orders – Starting Orders) Magento Performance Toolkit – Maxed Orders
97 To use the Magento Performance Toolkit on a existing built out production website: 1.Follow the testing on a production site plan from earlier. 2.Ensure your top-menu matches the JMeter tests search for top_menu, you can switch the jmeter test or try to modify the template to match. 3.Modify the generate script to not create categories 4.Generate your data 5.Run your tests Other Issues: • Not a trivial issue, it will take some time to setup. • Checkout and cart parts probably will fail and require changes to tests • Not the cleanest methodology, but it does work if you make the effort Magento Performance Toolkit – In Production
98 Magento Performance Toolkit – Benchmarking • Find metrics to base any decisions you make, and quantify the results • 30 Threads per web head is a good spot to start (except in the VM) • Orders Per Hour • Collect test start and end orders • (end orders – start orders) * (3600 / time to complete) • Customers Per Day • (86400 / time to complete) * Online Customers
99 Magento Performance Toolkit – Benchmarking • Max Response Time: Should stay below 10 seconds in a good test • Average Response Time: Should be below 500ms in a good test • Error Rate: 0.00% is what your looking for, some glitches may happen • Collect New Relic Stats or other stats and use them to supplement the tests
100 Clustering Demo Magento Digital Ocean Sandbox
101 • Vagrant  https://www.vagrantup.com/ • Digital Ocean Provider  https://github.com/smdahlen/vagrant-digitalocean • Magento Vagrant Puppet Sandbox  https://github.com/aepod/magento-digitalocean-sandboxCentos 6.6 1) Install Vagrant 2) Install Digital Ocean provider 3) Grab the project source from github git clone https://github.com/aepod/Magento-Vagrant-Puppet-Sandbox.git 4) Add your Token to Vagrantfile 5) Ensure you have a ssh key 6) Stand up some boxes. Using the Magento Vagrant Puppet Sandbox
102 • Newrelic has provided the following signup:  http://newrelic.com/robofirm/  Free 30 day lite account • Add your newrelic key to the Vagrantfile • Puppet will handle the rest, you will see test results in New Relic Using the Magento Vagrant Puppet Sandbox - Newrelic
103 • All nodes run Centos 6.5 and have firewalls disabled (NOT SECURED) • Puppet does a lot of the base configuration • Some shell scripting triggered from vagrant happens before/after puppet.  Available cluster nodes: • Puppet: All other boxes require it to stand up. • Monolith: Fully functional single webserver • DB: MySQL(Percona), Redis and Shared Filesystem(NFS) • Web: The public facing haproxy server. • Render[1-4]: Nginx, PHP-fpm. These can be added/removed and scaled. • jMeter: Used for testing, simple java enabled box. Sandbox – Stand Up
104 1. vagrant up puppet 2. Add puppet IP from output to Vagrantfile 3. vagrant up monolith Sandbox – vagrant up (monolith)
105 • To use the tests locally you need to “poison your DNS” • On Mac: /private/etc/hosts http://www.tekrevue.com/tip/edit-hosts-file-mac-os-x/ • On Windows: c:windowsSystem32driversetchosts http://www.thewindowsclub.com/hosts-file-in-windows Add the following for monolith: 192.168.200.21 web www.idealphp.com Config - Switching Hosts (monolith)
106 • Login on JMeter box • Switch to the jmeter directory:  cd /usr/local/apache-jmeter-2.13/bin/ ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com DON’T FORGET TO NOTE THE RESULTS Test -Monolith
107 1. vagrant up puppet 2. Add puppet IP from output to Vagrantfile 3. Vagrant up db 4. Vagrant up web 5. Vagrant up render1 * * render2,render3,render4 can also be brought up at anytime. Sandbox – vagrant up (cluster)
108 • Remove or change the entry for monolith  On Mac: /private/etc/hosts  On Windows: c:windowsSystem32driversetchosts • Change the hosts on the JMeter box, in /etc/hosts Modify the hosts file: 192.168.200.12 web www.idealphp.com Config – Switching Hosts (cluster/web)
109 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com • Note Results Test – Cluster (1 web)
110 • Vagrant up render2 • Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats Config – Add a Web Node
111 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com • Note Results Test – Cluster (2 web)
112 Break Q/A
113 • Vagrant up render3 • Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats Config – Add a Web Node
114 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com • Note Results • You can start to experiment with more threads if you wish. Test – Cluster (3 web)
115 • Vagrant up render4 • Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats Config – Add Another Web Node
116 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com - Jramp_period=30 • Note Results • Try more threads Test – Cluster (4 web)
117 • We are testing on small cloud servers, the results will not be good. • Doing some compare between the setups you should see throughput going up as you add boxes. • In production you will start to find bottlenecks easily using this method • Results – What does it all mean?
118 ./jmeter -n -t ../tests/benchmark.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=10 -Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_password="5678password" -Jadmin_user=admin -Jview_product_add_to_cart_percent=0 -Jview_catalog_percent=0 -Jcustomer_checkout_percent=0 -Jguest_checkout_percent=100 This test will slam orders through as fast as it can. Using it is as simple as recording the number of orders at the start and finish of a test, and then calculating the number of orders per hour on that. Orders Per Hour = (3600 / Time to Complete) * (Ending Orders – Starting Orders) Try against less web nodes and compare results Test – Cluster (Max Orders)
119 • Adjust /etc/php-fpm.d/www.conf change max_children from 8 to 16  Edit on puppet /etc/puppet/modules/daemons/manifest/nginx/php.pp  Line 42: Change to 16  Run on web: /root/deploy_application.sh • Retest with original 4 server test: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com - Jramp_period=30 Config – PHP FPM Tuning
120 Magento Vagrant Puppet Sandbox Questions / Feedback
121 Thank You! • Mathew Beane <mbeane@robofirm.com> • Twitter: @aepod • Blog: http://aepod.com/ Rate this talk: http://joind.in/talk/view/15501 (Slides will be available)Thanks to : My Family The Magento Community Robofirm Fabrizo Branca (deployments) Thjis Feryn (sentinel) Ben Rose (puppet) Digital Ocean New Relic Zend and the PHP Community
122 • Logo graphics:  http://payload.cargocollective.com/1/3/107799/1402304/Puppet%20Han d.jpeg  http://images.clipartpanda.com/sandbox-clipart-black-and-white- lemmling_sandbox_black_white_line_art_scalable_vector_graphics_svg_c lip_art.svg  Digital Ocean and Magento Logo

More Related Content

What's hot (18)

PPTX
Lonestar php scalingmagento
Mathew Beane
 
PPTX
cache concepts and varnish-cache
Marc Cortinas Val
 
PDF
Modern PHP Ch7 Provisioning Guide 導讀
Chen Cheng-Wei
 
PDF
Running and Scaling Magento on AWS
AOE
 
PDF
Cache hcm-topdev
Thanh Chau
 
PDF
How to scale PHP applications
Enrico Zimuel
 
PPTX
Nginx
Dhrubaji Mandal ♛
 
PDF
Varnish Configuration Step by Step
Kim Stefan Lindholm
 
ODP
Improving PHP Application Performance with APC
vortexau
 
PPTX
PHP Performance with APC + Memcached
Ford AntiTrust
 
PPTX
Anthony Somerset - Site Speed = Success!
WordCamp Cape Town
 
PPTX
Joomla! Performance on Steroids
SiteGround.com
 
PDF
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
ColdFusionConference
 
PPT
WE18_Performance_Up.ppt
webhostingguy
 
PDF
[MathWorks] Versioning Infrastructure
Perforce
 
PPT
are available here
webhostingguy
 
PDF
HAProxy tech talk
icebourg
 
PDF
Converting Your Dev Environment to a Docker Stack - php[world]
Dana Luther
 
Lonestar php scalingmagento
Mathew Beane
 
cache concepts and varnish-cache
Marc Cortinas Val
 
Modern PHP Ch7 Provisioning Guide 導讀
Chen Cheng-Wei
 
Running and Scaling Magento on AWS
AOE
 
Cache hcm-topdev
Thanh Chau
 
How to scale PHP applications
Enrico Zimuel
 
Nginx
Dhrubaji Mandal ♛
 
Varnish Configuration Step by Step
Kim Stefan Lindholm
 
Improving PHP Application Performance with APC
vortexau
 
PHP Performance with APC + Memcached
Ford AntiTrust
 
Anthony Somerset - Site Speed = Success!
WordCamp Cape Town
 
Joomla! Performance on Steroids
SiteGround.com
 
Scale ColdFusion with Terracotta Distributed Caching for Ehchache
ColdFusionConference
 
WE18_Performance_Up.ppt
webhostingguy
 
[MathWorks] Versioning Infrastructure
Perforce
 
are available here
webhostingguy
 
HAProxy tech talk
icebourg
 
Converting Your Dev Environment to a Docker Stack - php[world]
Dana Luther
 

Similar to Zendcon scaling magento (20)

PPTX
Phpworld.2015 scaling magento
Mathew Beane
 
PDF
Host and Boast: Best Practices for Magento Hosting | Imagine 2013 Technolog…
Atwix
 
PDF
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
MagentoImagine
 
PDF
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
varien
 
PDF
Oleh Kobchenko - Configure Magento 2 to get maximum performance
Meet Magento Italy
 
PPTX
Magento for-performance- v01
Rajneesh Maurya
 
PDF
Optimizing Magento Performance with Zend Server
varien
 
PPTX
Nexcess Magento Imagine 2014 Performance Breakout
Nexcess.net LLC
 
PPTX
Midwest PHP - Scaling Magento
Mathew Beane
 
PDF
Magento Performance Optimization 101
Angus Li
 
PDF
Magento performance & optimisation best practices
Philippe Humeau
 
PDF
Eugene Shaksuvarov - Tuning Magento 2 for Maximum Performance
Meet Magento Italy
 
PPTX
Improving Performance on Magento 1*
David Z. Lerner
 
PDF
Magento e commerce performance optimization
Ranosys Technologies Pte Ltd
 
PPTX
Magento performances 2015 best practices
NBS System
 
PDF
High-Performance Magento in the Cloud
AOE
 
PDF
Magento scalability from the trenches (Meet Magento Sweden 2016)
Divante
 
PDF
10 Million hits a day with WordPress using a $15 VPS
Paolo Tonin
 
PPTX
Boost your website by running PHP on Nginx
Harald Zeitlhofer
 
PDF
Optimizing Magento for Peak Performance
MageCloud
 
Phpworld.2015 scaling magento
Mathew Beane
 
Host and Boast: Best Practices for Magento Hosting | Imagine 2013 Technolog…
Atwix
 
Magento's Imagine eCommerce Conference 2011 - Hosting Magento: Performance an...
MagentoImagine
 
Magento Imagine eCommerce Conference February 2011: Optimizing Magento For Pe...
varien
 
Oleh Kobchenko - Configure Magento 2 to get maximum performance
Meet Magento Italy
 
Magento for-performance- v01
Rajneesh Maurya
 
Optimizing Magento Performance with Zend Server
varien
 
Nexcess Magento Imagine 2014 Performance Breakout
Nexcess.net LLC
 
Midwest PHP - Scaling Magento
Mathew Beane
 
Magento Performance Optimization 101
Angus Li
 
Magento performance & optimisation best practices
Philippe Humeau
 
Eugene Shaksuvarov - Tuning Magento 2 for Maximum Performance
Meet Magento Italy
 
Improving Performance on Magento 1*
David Z. Lerner
 
Magento e commerce performance optimization
Ranosys Technologies Pte Ltd
 
Magento performances 2015 best practices
NBS System
 
High-Performance Magento in the Cloud
AOE
 
Magento scalability from the trenches (Meet Magento Sweden 2016)
Divante
 
10 Million hits a day with WordPress using a $15 VPS
Paolo Tonin
 
Boost your website by running PHP on Nginx
Harald Zeitlhofer
 
Optimizing Magento for Peak Performance
MageCloud
 
Ad

More from Mathew Beane (8)

PPTX
MidwestPHP - Getting Started with Magento 2
Mathew Beane
 
PPTX
Z-Ray: A customizable development tool belt (Zendcon 2016)
Mathew Beane
 
PPTX
ELK Ruminating on Logs (Zendcon 2016)
Mathew Beane
 
PPTX
Madison PHP - Getting Started with Magento 2
Mathew Beane
 
PPTX
Elk ruminating on logs
Mathew Beane
 
PPTX
php[world] Magento101
Mathew Beane
 
PPTX
Zendcon zray
Mathew Beane
 
PDF
Sean McDonald Resume
Mathew Beane
 
MidwestPHP - Getting Started with Magento 2
Mathew Beane
 
Z-Ray: A customizable development tool belt (Zendcon 2016)
Mathew Beane
 
ELK Ruminating on Logs (Zendcon 2016)
Mathew Beane
 
Madison PHP - Getting Started with Magento 2
Mathew Beane
 
Elk ruminating on logs
Mathew Beane
 
php[world] Magento101
Mathew Beane
 
Zendcon zray
Mathew Beane
 
Sean McDonald Resume
Mathew Beane
 
Ad

Recently uploaded (20)

PPTX
一比一原版(SUNY-Albany毕业证)纽约州立大学奥尔巴尼分校毕业证如何办理
Taqyea
 
PPT
introductio to computers by arthur janry
RamananMuthukrishnan
 
PDF
AI_MOD_1.pdf artificial intelligence notes
shreyarrce
 
DOCX
Custom vs. Off-the-Shelf Banking Software
KristenCarter35
 
PPTX
sajflsajfljsdfljslfjslfsdfas;fdsfksadfjlsdflkjslgfs;lfjlsajfl;sajfasfd.pptx
theknightme
 
PPTX
Presentation3gsgsgsgsdfgadgsfgfgsfgagsfgsfgzfdgsdgs.pptx
SUB03
 
PDF
The-Hidden-Dangers-of-Skipping-Penetration-Testing.pdf.pdf
naksh4thra
 
PPTX
L1A Season 1 Guide made by A hegy Eng Grammar fixed
toszolder91
 
PPTX
Softuni - Psychology of entrepreneurship
Kalin Karakehayov
 
PPTX
Optimization_Techniques_ML_Presentation.pptx
farispalayi
 
PDF
The Internet - By the numbers, presented at npNOG 11
APNIC
 
PPT
Agilent Optoelectronic Solutions for Mobile Application
andreashenniger2
 
PPTX
PM200.pptxghjgfhjghjghjghjghjghjghjghjghjghj
breadpaan921
 
PDF
Build Fast, Scale Faster: Milvus vs. Zilliz Cloud for Production-Ready AI
Zilliz
 
PPTX
Orchestrating things in Angular application
Peter Abraham
 
PPTX
法国巴黎第二大学本科毕业证{Paris 2学费发票Paris 2成绩单}办理方法
Taqyea
 
PPT
introduction to networking with basics coverage
RamananMuthukrishnan
 
PPTX
internet básico presentacion es una red global
70965857
 
PDF
𝐁𝐔𝐊𝐓𝐈 𝐊𝐄𝐌𝐄𝐍𝐀𝐍𝐆𝐀𝐍 𝐊𝐈𝐏𝐄𝐑𝟒𝐃 𝐇𝐀𝐑𝐈 𝐈𝐍𝐈 𝟐𝟎𝟐𝟓
hokimamad0
 
PDF
Cleaning up your RPKI invalids, presented at PacNOG 35
APNIC
 
一比一原版(SUNY-Albany毕业证)纽约州立大学奥尔巴尼分校毕业证如何办理
Taqyea
 
introductio to computers by arthur janry
RamananMuthukrishnan
 
AI_MOD_1.pdf artificial intelligence notes
shreyarrce
 
Custom vs. Off-the-Shelf Banking Software
KristenCarter35
 
sajflsajfljsdfljslfjslfsdfas;fdsfksadfjlsdflkjslgfs;lfjlsajfl;sajfasfd.pptx
theknightme
 
Presentation3gsgsgsgsdfgadgsfgfgsfgagsfgsfgzfdgsdgs.pptx
SUB03
 
The-Hidden-Dangers-of-Skipping-Penetration-Testing.pdf.pdf
naksh4thra
 
L1A Season 1 Guide made by A hegy Eng Grammar fixed
toszolder91
 
Softuni - Psychology of entrepreneurship
Kalin Karakehayov
 
Optimization_Techniques_ML_Presentation.pptx
farispalayi
 
The Internet - By the numbers, presented at npNOG 11
APNIC
 
Agilent Optoelectronic Solutions for Mobile Application
andreashenniger2
 
PM200.pptxghjgfhjghjghjghjghjghjghjghjghjghj
breadpaan921
 
Build Fast, Scale Faster: Milvus vs. Zilliz Cloud for Production-Ready AI
Zilliz
 
Orchestrating things in Angular application
Peter Abraham
 
法国巴黎第二大学本科毕业证{Paris 2学费发票Paris 2成绩单}办理方法
Taqyea
 
introduction to networking with basics coverage
RamananMuthukrishnan
 
internet básico presentacion es una red global
70965857
 
𝐁𝐔𝐊𝐓𝐈 𝐊𝐄𝐌𝐄𝐍𝐀𝐍𝐆𝐀𝐍 𝐊𝐈𝐏𝐄𝐑𝟒𝐃 𝐇𝐀𝐑𝐈 𝐈𝐍𝐈 𝟐𝟎𝟐𝟓
hokimamad0
 
Cleaning up your RPKI invalids, presented at PacNOG 35
APNIC
 

Zendcon scaling magento

  • 2. 2 Mathew Beane @aepod Director of Systems Engineering - Robofirm Zend Z-Team Volunteer – Magento Division Family member – 3 Kids and a Wife Magento Certified Developer
  • 3. 3 Joshua Warren: Magento 2 - An Introduction to a Modern PHP-Based System Catch this talk Wednesday at 4pm in The Joint • Open-source e-commerce platform • PHP based application • Core utilizes Zend Framework 1 • Very flexible, built to modify • Extremely scalable, supports huge stores • Market leader and still growing • Magento 2 is right around the corner • It’s is in merchant beta now • http://magento.com/developers/magento2 Magento: What is it? Mathew Beane: Magento101: Getting Started with Magento Development Catch this talk Tuesday at 11:15am in Artist 4
  • 4. 4 Magento is the market leader in E-Commerce
  • 6. 6 Shopping Is Moving Online Be prepared to scale your clients Magento websites because growth is the norm.
  • 8. 8 https://github.com/aepod/magento-digitalocean-sandbox • Uses Puppet to build out cluster • To setup the initial test checkout readme in project • Initial release standard (n)webnodes + single db config • Monolith server has Web, PHP, DB, Redis all-in-one • We will come back to this later Cluster Configuration Overview
  • 9. 9 • Introduction • Magento Application: Preparing and Optimizing • Magento Cluster Architecture: Examine Typical Layouts • Magento Digital Ocean Sandbox: Demonstrating Clustering • Magento Performance Toolkit: Measuring Performance Benchmarks • Magento Digital Ocean Sandbox: Guided experiments • Conclusion: Open Q & A Todays Plan
  • 11. 11 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
  • 12. 12 • Optimize sysctl.conf: Increase limits for the servers based on Memory and CPU. • Modify Nginx settings:  Setup PHP FPM with TCP properly  Optimize Worker Connections  Tune up caching and other settings • Optimize PHP FPM  Process Manager Optimization  APC / Zend Opcode Cache  Tune up other settings • Optimize Redis • Optimize MySQL Optimizing Overview Thijs Feryn: Making Magento Go Fast Catch this talk Tuesday at 2:45PM in Artist 2
  • 13. 13 • Sysctl is an interface for configuring the Linux Kernel. • The following examples are some of the settings you may change in the sysctl.conf. • Remember to backup your configurations. • After changing you run: sysctl –p to update your kernel settings. • Always test after changing values, quantify your results against previous tests. See also: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Performance_Tuning_Guide/system- tuning.html Optimizing Hints – Linux Kernel
  • 14. 14 # Controls the maximum shared segment size, in bytes (64gb) kernel.shmmax = 68719476736 # Controls the maximum number of shared memory segments, in pages (16gb) kernel.shmall = 4294967296 #controls the number of process identifers that can be created kernel.pid_max = 2097152 # Number of open files handles fs.file-max = 2097152 # Defaults to 65536 – Deals with malloc maps. vm.max_map_count = 262144 • You can find a lot more tuning information by looking up the parameters and learning more about them. Optimizing Hints – Linux Kernel
  • 15. 15 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.rmem_default = 1048576 net.core.wmem_default = 1048576 net.core.netdev_max_backlog = 65536 net.core.somaxconn = 65536 net.ipv4.tcp_rmem = 4096 1048576 16777216 net.ipv4.tcp_wmem = 4096 1048576 16777216 net.ipv4.tcp_max_syn_backlog = 65536 net.core.somaxconn is the least trivial, and typically set very low causing the following: “apr_socket_recv: Connection reset by peer” and “connect() … failed“. Optimizing Hints – Linux Kernel
  • 16. 16 location ~ .php$ { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME /var/www/html/$fastcgi_script_name; fastcgi_param MAGE_IS_DEVELOPER_MODE 0; fastcgi_param PHP_VALUE error_log=/var/log/php-errors.log; include fastcgi_params; fastcgi_cache off; fastcgi_buffer_size 128k; fastcgi_buffers 256 16k; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; fastcgi_read_timeout 900s; } Optimizing Hints – Nginx
  • 17. 17 Battle Hardened Magento Nginx Config: https://gist.github.com/gwillem/cd5ae6845fa33aa0d481 • worker considerations:  Worker Processes = Total # CPU Cores  Worker Connections: ½ of ulimit –n  Max Connections = worker processes * worker connections  Worker_rlimit_nofile safe value = 2 * Max Connections  keepalive_timeout mitigates overage by trading for latency Optimizing Hints – Nginx
  • 18. 18 # Media: images, icons, video, audio, HTC location ~* .(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ { expires 1M; access_log off; add_header Cache-Control "public"; } •Set long expires dates on media assets. Optimizing Hints – Nginx
  • 19. 19 curl -X GET -I www.idealphp.com:80/skin/frontend/rwd/default/images/logo.gif HTTP/1.1 200 OK Server: nginx/1.0.15 Date: Tue, 14 Apr 2015 17:56:32 GMT Content-Type: image/gif Content-Length: 2320 Last-Modified: Tue, 14 Apr 2015 00:30:08 GMT X-Robots-Tag: noindex, nofollow, nosnippet, noarchive Accept-Ranges: bytes Cache-Control: max-age=31536000 Cache-control: public • Test your headers using curl or other tools Optimizing Hints – Nginx
  • 20. 20 • PHP 5.4 - Minimum • PHP 5.5 – Great, better performance if you have compatibility • PHP 5.6/5.7 - Even Better, although not supported • PHP 7 / HHVM - Not until Magento 2 – Unless your Daniel Slooth • Use TCP instead of sockets • Test with Zend OpCache instead of APC • Turn off APC Stat/Zend Revalidate  apc.stat = 0  opcache.revalidate_freq=0 Optimizing Hints – PHP-FPM
  • 21. 21 • Tune Process Manager  pm: ondemand  pm.max_children: about 6-8 per CPU *more later  pm.max_requests: 10000 *php threads should die for garbage collection  listen.backlog: set to 64000 (typically set to -1 or 128) Listen backlog is limited by net.ipv4.tcp_max_syn_backlog *only do after sytsctl.conf optimization Optimizing Hints – PHP-FPM
  • 22. 22 pm.max_children will limit the amount of memory php uses overall  Calculate the memory a thread is using while under load: ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%sn", sum/NR/1024,"M") }'  Use the total amount of memory you want php to consume: pm.max_children = Total RAM dedicated to the web server / Max child process size Based on this, you can set your max_children based on memory instead of the CPU metric used earlier. This will allow for more memory consumption in production environment. Be sure to test with higher values, more is not always better. Further Reading: http://myshell.co.uk/index.php/adjusting-child-processes-for-php-fpm-nginx/ Optimizing Hints – PHP-FPM
  • 23. 23 • Redis is the preferred cache for Magento  sysctl ulimit values affect the redis maxclients  Use three separate Redis instances with a single DB in each, on different ports  tcp-backlog can be increased, adjust sysctl somaxconn and tcp_max_syn_backlog  maxmemory can be set very low typically. • Using Sentinel and twemproxy you can horizontally scale  Clustering redis to eliminate all single threaded bottlenecks and single points of failure  Requires orchestration, and Magento application changes • Don’t forget to update Cm_Redis to newest version (Magento lags behind) • At Rackspace or AWS: Use Object Rocket https://objectrocket.com/ for Redis SaaS Optimizing Hints – Redis
  • 24. 24 • Turn off MySQL Query Caching – This is single threaded and not of use with Magento (* Before MySQL 5.7) • Innodb settings can be tweaked to consume more memory Selected Sample Configuration Settings for a 32GB Dual Hex-Core Server myisam_sort_buffer_size = 64M key_buffer_size = 256M join_buffer_size = 4M read_buffer_size = 4M read_rnd_buffer_size = 4M sort_buffer_size = 8M table_open_cache = 8192 thread_cache_size = 512 tmp_table_size = 384M max_heap_table_size = 384M max_allowed_packet = 1024M query_cache_type=0 query_cache_size=0 Optimizing Hints – MySQL innodb_thread_concurrency = 24 innodb_buffer_pool_size = 16G innodb_log_file_size = 384M innodb_log_buffer_size=24M innodb_additional_mem_pool_size = 16M innodb_io_capacity = 800 innodb_concurrency_tickets = 900 innodb_flush_neighbor_pages=cont innodb_lock_wait_timeout=75 innodb_flush_method=O_DIRECT
  • 25. 25 • Clustering MySQL for Magento Requires all tables to be converted to InnoDB • catalog_full_text_search requires MyISAM, unless your running MySQL 5.7 • When clustering always use identical hardware for each server • Master/Master is not supported by Magento • You can specify read servers and even load balance them • It is unusual for Magento to require a true MySQL Cluster, as it is typically not the bottleneck Optimizing Hints – MySQL
  • 26. 26 • Varnish is great, however your application must be suitable  Adding Varnish to a slow site is like putting a bandage on a neck wound  Varnish complicates logging and request tracking throughout the application • CDNs can have a great impact on end-user performance  Reduction in requests is also beneficial to the web server  Typically one of the cheapest/best moves to make • Always test any changes you make Optimizing Hints – Other
  • 27. 27 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
  • 28. 28 What You Don’t Want To Scale A very good un-cached response, good ready to scale. A typical bad response, scaling this will only cause you heartache.
  • 29. 29 • DOUBLE CHECK YOUR PATCHES:  https://www.magereport.com/ • Triple check your patches:  Patch Grid: https://goo.gl/X6iu8O by @knowj Magento – Apply Security Patches
  • 30. 30 Keep a clean core. If you inherit a project, one of the first things you should do is clean up core. This means you have not changed any default files including templates, layouts and code. Magento – “Clean Core” What is a clean core: • No changes to any file provided by Magento • No changes to app/code/core • No changes to any other core location • … even the index.php • Exception: PATCHES! Ben Marks, Magento Community Evangelist @benmarks (Protector of the Core)
  • 31. 31 • Find and eliminate your core changes with: https://github.com/AOEpeople/mpmd Magento Project Mess Detector by Fabrizio Branca Magento Project Mess Detector Fabrizio Branca: How to Build a Pure Evil Magento Module Catch this talk Tuesday at 4pm in Artist 3
  • 32. 32 Fabrizio Branca’s work on cleaning up Magento projects: http://fbrnc.net/blog/2014/12/magento-update In this article he goes into depth on the process of separating the core files, and using Modman and composer to package everything. Tegan Snyder's work on building modman configurations for extensions out of existing source: https://github.com/tegansnyder/meff This will allow you to generate modman files for your extensions, which may be handy in the process of cleaning up. Refactoring Bad Magento Projects – Further Study
  • 33. 33 • Use Z-Ray to profile and tune your Magento Application • Z-Ray allows you to see every request, and many details about the request. • It’s easy to install, free to try and really amazing. • It can be used to debug Mobile browsers through the Zend Server backend • Z-Ray No longer requires Zend Server: http://www.zend.com/en/products/z-ray/z-ray-preview Optimizing Your Application with Zend Z-Ray Mathew Beane: Z-Ray & Magento: A Customizable Development Tool Belt Catch this talk Wednesday at 11am in Artist 4
  • 34. 34 • Demo: http://serverdemo.zend.com/magento/ • Blog Article: http://aepod.com/debug-magento-using-zend-zray/ Optimizing Your Application with Zend Z-Ray
  • 35. 35 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
  • 36. 36 • Should apply equally to the hardware architecture as to the software application. • Every project should have a continuous development cycle in place, and ensure that it is well documented and easy to follow for all of the people involved. Development Cycle
  • 37. 37 • Design: Plan our your sprint, stick to Agile. • Develop: Toward a release on a schedule  Test: Build PHPUnit and Selenium tests into your process  Code Review: Pull requests and code review EVERYTHING.  User Acceptance Tests: Make sure the client is happy first. • Deploy: Package, test and deploy.  Deployments: Automate, and make the production servers hands off. Development Cycle - Simplified
  • 38. 38  Goal based system that encourages teamwork.  Daily communications with all involved in project.  Flexible and focused on delivery.  Encourages Teamwork and Accountability. Development Cycle – Agile / Scrum
  • 39. 39 • Keep a clean core  Never edit core files. Ensure that you can see any core changes in production. • Release / Feature Branching  Choose a branching methodology and build around it. • Use Pull Request  Code review increases the quality of your code. Make pull requests part of your workflow. Development Cycle – Branching
  • 40. 40 http://nvie.com/posts/a-successful-git-branching-model/ Successful Git Branching Model • Use Tagging as you release to master • Types of branches: • Master: Tagged versions, what's on production. • Develop: What will be merged into master with the next release. • Release: Preparing for release. Allows for last minute fixes, testing and prepping. • Feature: Usually checked out by developers to work on features. • Hotfix: Fixes aimed for unplanned production releases.
  • 41. 41 Build testing into all of your development cycles. Development Cycle – Testing Developers are working on testing with Magento 1 BDD: https://speakerdeck.com/alistairstead/bdd-with-magento BeHat: https://github.com/MageTest/BehatMage PHPSpec: https://github.com/MageTest/MageSpec Magento 2 has a built in Test Automation Framework ( PHPUnit,Selenium ) https://wiki.magento.com/display/MAGE2DOC/Magento+Automated+Testing+Guide https://github.com/magento/taf https://alankent.wordpress.com/2014/06/28/magento-2-test-automation/
  • 42. 42 • Use a deployment tool, and LOCK DOWN production. No one should ever have to touch an application file manually for any reason. • If you can extend this practice to your OS and server stack you now have fully engaged DevOps. Do Not Edit on Production
  • 43. 43 Development Cycle – Testing (Example of Fail) After introducing SOLR indexing, testing showed serious issues as it indexed.
  • 44. 44 • You must be able to package your application in order to deploy it to a cluster.  Deployment infrastructure may determine what packaging you will need to use, for instance one of the following: • Zend Server offers Application Packaging and Deployment • A simple “golden copy” (tgz or flat) that is pushed to web nodes. • RPM or other proprietary packaging tool.  Use Composer and/or modman to help maintain your application packages.  Composer can even be used to install Magento Core  Make sure your build process is repeatable and testable. Development Cycle – Application Packaging
  • 45. 45 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
  • 46. 46 • When deploying an application to a cluster of application servers, a deployment tool is a requirement. • There are many choice, here are a couple typical deployment tools.  Capistrano: Written in Ruby but well accepted and great to work with  Jenkins: Another example of Deployment Automation  Bamboo: Part of the Atlassian stack also includes testing and other features.  Roll Your Own: This is more common, using bash scripts and other tools you can build a project deployment tool fairly easily. Deployment Tools – Typical Software
  • 47. 47 • Integrated with code versioning • Supports Multi-Server • Supports the Application  Handles Maintenance Mode automatically  Runs Installers  Clears Caches • Low Downtime • Rollback Procedure (a/b switch) • Trigger events such as shell scripts, or CDN clearing Nice to have – May not be required • Integrated testing • Packaging • Integration to GUI Interfaces Deployment Tools – Some Requirements
  • 48. 48 I highly suggest researching Fabrizio Branca’s work on the subject: http://www.slideshare.net/aoepeople/rock-solid-magento Fabrizio has a ton of articles and slideshares on the subject. Also check out Joshua Warren’s slides, he has several really informative presentations on the subject. http://www.slideshare.net/joshuaswarren/ Deployment Tools – Further Study Fabrizio Branca: Rock-solid Magento Development and Deployment Workflows Catch this talk Wednesday at 2:45PM in Artist 5
  • 49. 49 • Automate server deployments:  Puppet  Chef  Ansible  Salt • Centralize configurations • Automated updates and server onlining process • Take out the human factor! Deployment Tools - Server Deployments
  • 50. 50 Before You Start Clustering Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place
  • 51. 51 Load Testing and Metrics "Don't rate potential over performance." - Jim Fassel Blaze Meter Using Blazemeter you can easily build repeatable tests, with very nice graphs. (based on JMeter) Gatling http://gatling.io/ On par with Blazemeter. JMeter Very effective, without having to purchase a SaaS Siege Can be used minimally to simulate some types of load.
  • 52. 52 https://github.com/magento/magento-performance-toolkit The Magento Performance Toolkit is a set of automated tools that enables you to quickly and consistently measure the application performance of Magento using Apache JMeter. The toolkit includes(sic requires), a PHP script for generating a test database and a JMX script for the performance measurement. From: Magento Performance Toolkit Users Guide More Reading: http://aepod.com/using-the-magento-performance-toolkit/ Magento Performance Toolkit
  • 53. 53 The Magento Performance Toolkit can be crafted to pull data from production sites, although its complicated and you may want to cover your site using your own JMeter tests. Gathering the steps for a client site typically takes a day or more depending on the complexity of the site design, and the checkout. Developing and debugging this can be a real time sink depending on how far down the rabbit hole you want to go. JMeter Tests for “Normal Sites”
  • 54. 54 siege –t 5M –b –c 5 http://www.idealphp.com/ This will start up 5 threads and hammer the URL for 5 minutes. Siege is very useful, but limited. It’s easy to put a lot of traffic on a box with it, but the traffic will be limited to GET requests. You can pull a list of URL’s from the Magento sitemap and run through random requests, this can even be used to do some pseudo-warming of the cache. Using Siege
  • 55. 55 • Analyze software in Real-time • Collect Data  Application Data  Server Data  Browser Data  Magento Data * • Synthetics provides NrSQL • Alerts * Magento data provided via Magento Extension: http://www.magentocommerce.com/magento-connect/new-relic-reporting-29133.html New Relic - Introduction
  • 56. 56 New Relic – Production
  • 57. 57 New Relic - Testing
  • 58. 58 1. Prepare a downtime:  Forward site to a non-magento maintenance page  Exclude Testing IP – So tests can still proceed  Disable Cron 2. Copy Production Database 3. Point Magento at the new copied DB 4. Modify Magento Admin Settings in copied DB  Use Test Payment Gateway  Disable Transaction Emails 5. Test against Copied DB 6. After Testing is complete:  Point site at original DB  Put site back up on live  Enable Cron Using this method, you can test against production on production. You will need to bring the site down during the time period, but if this is done at a low point in traffic it causes little to no stress to the customers. Load Testing Production
  • 59. 59 Your ready to scale Magento Optimized Server Settings Magento application is optimized and clean Development pipeline in place Deployment infrastructure is solid Rigorous testing protocols are in place • Next Up: Discussion about typical cluster architecture and components
  • 61. 61 • Load Balanced – Front End Webservers • Admin / Backend Server Separated • Single MySQL Database Master, with Slaves Magento Cluster Architecture – Typical Layout
  • 62. 62 • Web Servers  Apache / Nginx  PHP 5.4 + (depends on Magento Version)  Multiple Webservers – Easy to cluster  Varnish Can be used as a Reverse Proxy • Nexcess Turpentine is the standard Magento Extension • Magento2 has Varnish support out of the box Magento Cluster Architecture - Components
  • 63. 63 • Database  MySQL / Percona  Master / Slave • Slaves are typically used for reporting or Hotswap for failure • Can be set to read servers for application, but it requires some work  Solr / Elastic Search • Typically run on the Database or Backend Server • Can power Search or other Magento business logic  Magento 2 introduces Partitioned Databases Magento Cluster Architecture - Components
  • 64. 64 • Other Components  File Server • NFS – This is by far the most common • NAS – Can be used on high end clients  Redis / Memcache  Deployment Tools  Monitoring  ELK – Log aggregation across servers  Puppet / Chef Magento Cluster Architecture - Components
  • 65. 65 Breaking Points, where you need to adjust first. 1. PHP – More web nodes, easy to do and very performant. 2. Filesystem – Use a CDN to offset static asset traffic. 3. Redis – Cluster Redis using sentinel/nutcracker. 4. Database – Look at read server clustering At some point adding Varnish should be considered as well. Varnish requires a clean application with some thought as to which blocks can be loaded dynamically via AJAX. Magento Cluster Architecture - Bottlenecks Thijs Feryn: Making Magento Go Fast Catch this talk Tuesday at 2:45PM in Artist 2
  • 66. 66 Scaling Implementation – A Scoring System Concern Score (1-5) What is the simplicity of the solution? How difficult is it to support? Is it extensible, how broadly does it apply? Will it break often? How easy is it to update? Is it Expensive? Concern Yes / No Does it require specialized services? (os level services, not typically installed) Does it require Magento application changes? Does it create a single point of failure?
  • 67. 67 • Simple to Load Balance, most sites start here • Challenges include the following:  Session Management: should be a no brainer if your using redis for sessions  Shared file System: Use NFS for media, keep all code local to web servers  Fencing: easily solved with either software or hardware  Log Collection: ELK, Rsyslog or Splunk • Keep the system builds simple, repeatable and automate if possible Scaling – Web Server Concerns
  • 68. 68 • How to automate:  Create an complete image to work from - include puppet so it can pull from a puppet master  The puppet master spins up your webserver stack  You have your deployment process in place, so plant the application and spin up the node • Be prepared to lose nodes, the more you have the more likely failure is • When a node runs amok, you must be prepared to kill it dead Scaling – Web Server Concerns (Automation)
  • 69. 69 Best practice for sharing files: Use NFS • Most other options are not viable. (glusterfs, iscsi, and so on) • NFS will not be the bottleneck if your application is configured correctly • Share only media/ to avoid bottlenecks • Client requirements may drive cluster toward a NAS Network Attached Storage (NAS) • Typically only on very large sites • Expensive but very reliable and robust • Entry level has very high bar (typically 20+ drives) Scaling - File systems
  • 70. 70 • Share specific folders via symbolic links  Example: [docroot]/media -> /var/magento/shared/media • Mount /var/magento/shared/[directory] via NFS, typically from backend server  Maintaining these when deploying is easy, just remake symbolic links or copy them • Can also be used for local content  Example: [docroot]/var -> /var/media/local/var  Not mounted via NFS, writes local var files for this server only Scaling – File Systems and Symbolic Links
  • 71. 71 Scaling – Redis Clustering “Redis clustering using sentinel is easy to set up. Adding twemproxy allows for a highly scalable Redis cluster and you get auto fail over and a ton of other benefits with this configuration. This arrangement can also remove your Redis single point of failure.” http://aepod.com/clustering-magento-redis-caching-with-sentinel-keepalived-twemproxy-and-twemproxy-agent/ • Easy to setup… well not really • Very Complex, requires orchestration • Sentinel Monitors Redis clusters • twemproxy handles sharding • twemproxy agent monitors sentinel • Very robust when setup, nothing is single threaded, everything is HA and the speed…. • Can be transparent to Magento despite the complexity
  • 72. 72 • Requires several layers of applications  Load Balancer: Load balances redis traffic to the twemproxy servers  Twemproxy: proxy server, typically setup as a cluster of 2 servers (can be mixed on other servers)  Nutcracker Agent: Connects twemproxy to sentinel to keep track of shards of servers  Sentinel: Monitors the redis instances and maintains availability  Redis: Dozens of redis servers, sharded, maintained and fenced by Sentinel and nutcracker  VERY Complex Redis (Sentinel / twemproxy) Thijs Feryn: Making Magento Go Fast Catch this talk Tuesday at 2:45PM in Artist 2
  • 73. 73 • Should be considered a low value target, except for High Availability concerns • Magento typically is not bottlenecked by MySQL • Using Percona XtraDB you can add read slaves • Magento only supports a single Master/Write server, with multiple read servers • Setting up load balanced READ db servers is not overly difficult, but doesn’t offer much if any performance benefits • Cluster DB will require more advanced fencing • Typical production sites do not use load balanced or clustered MySQL Scaling - MySQL
  • 74. 74 • Percona XtraDB to cluster MySQL • Percona XtraBackup for duplicating and rebuilding nodes • Percona Toolkit to help debug any issues your running into • Difficult to scale Write Servers • Scale out your read servers as needed, but MySQL reads are rarely the bottleneck • Typically Slave server is used for backup and hot swap, NOT clustering. A couple quick tips: • Not all tables in Magento are InnoDB, converting the MyISAM and Memory tables is OK • You will need to be able to kill read servers and refresh • Use your Master server as a read server in the load balancer pool, when you kill all your read servers, it can fall back to master. Database Servers (Percona)
  • 75. 75 • Load Balancer Components  Hardware Load Balancer  HAProxy Software Load Balancer • Database Components  Sentinel / Nutcracker  Percona XtraDB Tools  Clusterix or ScaleARC for High End Magento Cluster Architecture – Additional Tools
  • 76. 76 • HAProxy: a free fast and reliable solution featuring high availability, load balancing and proxy for TCP and HTTP-based applications. • HAProxy can be used for web servers, database servers, Redis and any other TCP based application. • Easy to configure using a puppet module • Can be clustered easily for high availability concerns • Handles L4 Load Balancing if you want to do fancy multi- application environments • Built in stats interface Load Balancers – Software / HAProxy SOFTWARE
  • 77. 77 Hardware Load Balancers • F5 Hardware load balancers are a standard • Brocade is another common brand • Rackspace offers a very easy to use web interface to maintain a hybrid infrastructure • Hardware load balancers offer a turn-key mature solution. • Typically Managed Service, with very hands off approach. HYBRID Hybrid Load Balancers • AWS Elastic Load Balancers • Rackspace Cloud Load Balancers • Cloudflare • Many others… Load Balancers – Hardware & Hybrid
  • 78. 78 Load Balancers – Making a choice • Budget concerns will drive this decision • Hosting Choices will affect availability, costs and toolsets • Start locally with HAProxy and build test clusters using vagrant • HAProxy can still be used, even with a hardware load balancer in place.
  • 79. 79 • Reverse Proxy: Varnish or Nginx you can and should cluster these • Puppet likes to be clustered: Mastering Puppet by Thomas Uphill is a great reference • Monitoring Tools: You’re running these single threaded, what if the monitoring server fails? • Warehouse Integrations, ERP Layer Stuff, everything may need to be multi-threaded • Search back ends such as solr or elasticsearch • Consider all of the parts of the business that surrounds the Magento Application and try to encapsulate them Scaling - Others
  • 80. 80 • Turpentine is the standard go to extension for Magento to tie to Varnish  Handles ESI  Handles Dealing with updating VCL  http://www.magentocommerce.com/magento-connect/turpentine-varnish-cache.html • If on Enterprise disable FPC, CE disable lesti_fpc • https://github.com/nexcess/magento-turpentine/wiki/Installation • Typically Varnish will run on each render box, and proxy to the local nginx. • Issues will typically show up as over caching of blocks or other bits of front end page • SSL is NOT supported by vagrant, terminate it at your load balancer Reverse Proxy (Varnish) Scaling Concerns
  • 81. 81 • NGINX is a great reverse proxy and load balancer • http://nginx.com/resources/admin-guide/reverse-proxy/ • Can be used to effectively buffer and split requests to different cluster servers Reverse Proxy (Nginx) Scaling Concerns
  • 82. 82 • Insert puzzle building analogy joke here: http://www.wikihow.com/Assemble-Jigsaw-Puzzles • Each hosting environment has its own quirks and add on top of that the business logic requirements you will almost always have a unique infrastructure for every client • Build small pieces and work them into the larger picture, you can get a lot of performance with a few minor changes. • Test everything you do, keep detailed notes on the configurations and compare against the previous tests Auto-Scaling Strategy
  • 84. 84 https://github.com/magento/magento-performance-toolkit The Magento Performance Toolkit is a set of automated tools that enables you to quickly and consistently measure the application performance of Magento using Apache JMeter. The toolkit includes(sic requires), a PHP script for generating a test database and a JMX script for the performance measurement. From: Magento Performance Toolkit Users Guide More Reading: http://aepod.com/using-the-magento-performance-toolkit/ Magento CE 1.9.1.0 Ready Version: https://github.com/aepod/magento-performance-toolkit Magento Performance Toolkit
  • 85. 85 • Importing can take a long time, and may require changes to php and MySQL settings • If it fails, it fails hard. Restart with a fresh DB • You can modify the profiles to better suite your testing requirements • Using the toolkit requires putting it in the dev/tools/performance_toolkit/ directory in the Magento install you want to use it on • The scripts requires this exact directory, due to includes and autoloading • Oftentimes will require you to remove the tmp dir inside the toolkit Magento Performance Toolkit – generate.php
  • 86. 86 Magento Performance Toolkit – Generate Profiles • You can create your own profiles by copying and modifying the XML in profiles/ • Large and X-Large take extremely long times to import and require changes to the PHP and MySQL settings and it still likes to fail hard. • After it completes import, double check the following • Front end still up and running and shows the framework categories • Admin and Frontend logs in correctly • Indexes and Caches rebuild correctly
  • 87. 87 Modify the tests, learn how they work and dig into errors you may be having running them. • Easy to use • Debugging requirement • Load/Save XML • Can crash on windows Magento Performance Toolkit – jMeter GUI
  • 88. 88 ./jmeter -n -t ~/benchmark-default.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=30 - Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_user="admin" - Jadmin_password=“adminpass"; • Host: host to check against • Base Path: for building the url • User: Roughly the # of threads (more on next slid) • Ramp Period: How long to take starting up • Report Save Path: Where it saves the summary • Loops: How many times it will try to loop through (except if it runs out of customers) • Admin User/Pass: The admin user and pass for the admin site.  This is expected to be at http://www.idealphp.com/admin/ • Any JMeter Attribute from top level can be set from command line using –Jattribute= Magento Performance Toolkit – JMeter Configuration
  • 89. 89 Magento Performance Toolkit – JMeter Test Results
  • 90. 90 The JMeter tests use 4 thread groups to accommodate the testing. • Category Product Browsing(30%): Opens the homepage, goes to a category, opens up 2 simple products and then a configurable. • View Product Add To Cart(62%): Same as Category browsing however it adds each of the items to the cart in addition to viewing them. • Guest Checkout (4%): Same as View/Add to Cart, in addition it runs through the checkout as a guest user. • Customer Checkout (4%): Same as View/Add to Cart, in addition it runs through the checkout as a logged in customer. Thread groups create threads based on the number of users and the above percentages(by default). users * group_percent / 100 • This result is rounded down, and can be 0 for checkouts, making no checkout threads. • You can set EACH group independently, and they do not need to add up to 100% or less in totality. Magento Performance Toolkit –Users Demystified
  • 91. 91 Magento Performance Toolkit – More Attributes Any JMeter Attribute from top level can be set from command line using –Jattribute= Customer and Admin Password must match what was used during the generate. Orders are tricky and used to calculate “users” which are used within the thread groups. See jmeter test setup Thread Group -> validate properties and count users
  • 92. 92 1. Setup Thread Groups 1. Extracts Categories from Top Menu 2. Searches for Simple Products, builds list 3. Searches for Config Products, builds list 4. Logs in on the Admin 5. Grabs a list of customers from the customer grid 6. Builds users count which is used by the threads. 2. Loops through Each Thread Group 1. Category Product Browsing 2. Product Browsing and add items to cart 3. Guest Checkout 4. Customer Checkout 3. Teardown 1. Just removes some stats, but you could extend this spot for some killer stats. Magento Performance Toolkit –JMeter Test Plan
  • 93. 93 Using Newrelic During JMeter tests
  • 94. 94 Using Newrelic During JMeter tests
  • 96. 96 ./jmeter -n -t ~/benchmark-default.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=10 -Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_password="5678password" -Jadmin_user=admin -Jview_product_add_to_cart_percent=0 -Jview_catalog_percent=0 -Jcustomer_checkout_percent=0 -Jguest_checkout_percent=100 (This can be done in the JMX as well) Non-Trivial Settings: View Catalog : 0% View Product Add to Cart : 0% Customer Checkout: 0% Guest Checkout: 100% This test will slam orders through as fast as it can. Using it is as simple as recording the number of orders at the start and finish of a test, and then calculating the number of orders per hour on that. Orders Per Hour = (3600 / Time to Complete) * (Ending Orders – Starting Orders) Magento Performance Toolkit – Maxed Orders
  • 97. 97 To use the Magento Performance Toolkit on a existing built out production website: 1.Follow the testing on a production site plan from earlier. 2.Ensure your top-menu matches the JMeter tests search for top_menu, you can switch the jmeter test or try to modify the template to match. 3.Modify the generate script to not create categories 4.Generate your data 5.Run your tests Other Issues: • Not a trivial issue, it will take some time to setup. • Checkout and cart parts probably will fail and require changes to tests • Not the cleanest methodology, but it does work if you make the effort Magento Performance Toolkit – In Production
  • 98. 98 Magento Performance Toolkit – Benchmarking • Find metrics to base any decisions you make, and quantify the results • 30 Threads per web head is a good spot to start (except in the VM) • Orders Per Hour • Collect test start and end orders • (end orders – start orders) * (3600 / time to complete) • Customers Per Day • (86400 / time to complete) * Online Customers
  • 99. 99 Magento Performance Toolkit – Benchmarking • Max Response Time: Should stay below 10 seconds in a good test • Average Response Time: Should be below 500ms in a good test • Error Rate: 0.00% is what your looking for, some glitches may happen • Collect New Relic Stats or other stats and use them to supplement the tests
  • 101. 101 • Vagrant  https://www.vagrantup.com/ • Digital Ocean Provider  https://github.com/smdahlen/vagrant-digitalocean • Magento Vagrant Puppet Sandbox  https://github.com/aepod/magento-digitalocean-sandboxCentos 6.6 1) Install Vagrant 2) Install Digital Ocean provider 3) Grab the project source from github git clone https://github.com/aepod/Magento-Vagrant-Puppet-Sandbox.git 4) Add your Token to Vagrantfile 5) Ensure you have a ssh key 6) Stand up some boxes. Using the Magento Vagrant Puppet Sandbox
  • 102. 102 • Newrelic has provided the following signup:  http://newrelic.com/robofirm/  Free 30 day lite account • Add your newrelic key to the Vagrantfile • Puppet will handle the rest, you will see test results in New Relic Using the Magento Vagrant Puppet Sandbox - Newrelic
  • 103. 103 • All nodes run Centos 6.5 and have firewalls disabled (NOT SECURED) • Puppet does a lot of the base configuration • Some shell scripting triggered from vagrant happens before/after puppet.  Available cluster nodes: • Puppet: All other boxes require it to stand up. • Monolith: Fully functional single webserver • DB: MySQL(Percona), Redis and Shared Filesystem(NFS) • Web: The public facing haproxy server. • Render[1-4]: Nginx, PHP-fpm. These can be added/removed and scaled. • jMeter: Used for testing, simple java enabled box. Sandbox – Stand Up
  • 104. 104 1. vagrant up puppet 2. Add puppet IP from output to Vagrantfile 3. vagrant up monolith Sandbox – vagrant up (monolith)
  • 105. 105 • To use the tests locally you need to “poison your DNS” • On Mac: /private/etc/hosts http://www.tekrevue.com/tip/edit-hosts-file-mac-os-x/ • On Windows: c:windowsSystem32driversetchosts http://www.thewindowsclub.com/hosts-file-in-windows Add the following for monolith: 192.168.200.21 web www.idealphp.com Config - Switching Hosts (monolith)
  • 106. 106 • Login on JMeter box • Switch to the jmeter directory:  cd /usr/local/apache-jmeter-2.13/bin/ ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com DON’T FORGET TO NOTE THE RESULTS Test -Monolith
  • 107. 107 1. vagrant up puppet 2. Add puppet IP from output to Vagrantfile 3. Vagrant up db 4. Vagrant up web 5. Vagrant up render1 * * render2,render3,render4 can also be brought up at anytime. Sandbox – vagrant up (cluster)
  • 108. 108 • Remove or change the entry for monolith  On Mac: /private/etc/hosts  On Windows: c:windowsSystem32driversetchosts • Change the hosts on the JMeter box, in /etc/hosts Modify the hosts file: 192.168.200.12 web www.idealphp.com Config – Switching Hosts (cluster/web)
  • 109. 109 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com • Note Results Test – Cluster (1 web)
  • 110. 110 • Vagrant up render2 • Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats Config – Add a Web Node
  • 111. 111 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com • Note Results Test – Cluster (2 web)
  • 113. 113 • Vagrant up render3 • Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats Config – Add a Web Node
  • 114. 114 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com • Note Results • You can start to experiment with more threads if you wish. Test – Cluster (3 web)
  • 115. 115 • Vagrant up render4 • Note Results in haproxy stats: http://www.idealphp.com/haproxy?stats Config – Add Another Web Node
  • 116. 116 • Repeat Tests: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com - Jramp_period=30 • Note Results • Try more threads Test – Cluster (4 web)
  • 117. 117 • We are testing on small cloud servers, the results will not be good. • Doing some compare between the setups you should see throughput going up as you add boxes. • In production you will start to find bottlenecks easily using this method • Results – What does it all mean?
  • 118. 118 ./jmeter -n -t ../tests/benchmark.jmx -Jhost=www.idealphp.com -Jbase_path=/ -Jusers=10 -Jramp_period=30 -Jreport_save_path=./ -Jloops=2000 -Jadmin_password="5678password" -Jadmin_user=admin -Jview_product_add_to_cart_percent=0 -Jview_catalog_percent=0 -Jcustomer_checkout_percent=0 -Jguest_checkout_percent=100 This test will slam orders through as fast as it can. Using it is as simple as recording the number of orders at the start and finish of a test, and then calculating the number of orders per hour on that. Orders Per Hour = (3600 / Time to Complete) * (Ending Orders – Starting Orders) Try against less web nodes and compare results Test – Cluster (Max Orders)
  • 119. 119 • Adjust /etc/php-fpm.d/www.conf change max_children from 8 to 16  Edit on puppet /etc/puppet/modules/daemons/manifest/nginx/php.pp  Line 42: Change to 16  Run on web: /root/deploy_application.sh • Retest with original 4 server test: ./jmeter -n -t ../tests/benchmark-small.jmx -Jhost=www.idealphp.com - Jramp_period=30 Config – PHP FPM Tuning
  • 120. 120 Magento Vagrant Puppet Sandbox Questions / Feedback
  • 121. 121 Thank You! • Mathew Beane <[email protected]> • Twitter: @aepod • Blog: http://aepod.com/ Rate this talk: http://joind.in/talk/view/15501 (Slides will be available)Thanks to : My Family The Magento Community Robofirm Fabrizo Branca (deployments) Thjis Feryn (sentinel) Ben Rose (puppet) Digital Ocean New Relic Zend and the PHP Community
  • 122. 122 • Logo graphics:  http://payload.cargocollective.com/1/3/107799/1402304/Puppet%20Han d.jpeg  http://images.clipartpanda.com/sandbox-clipart-black-and-white- lemmling_sandbox_black_white_line_art_scalable_vector_graphics_svg_c lip_art.svg  Digital Ocean and Magento Logo

Editor's Notes

  • #2: Greenroom
  • #3: Robofirm is a Magento Solutions provider and a Magento Partner focused on Mid-level to Enterprise clients. Based out of New York City, however the bulk of our developers are in Dallas or Minneapolis. Zteam – Magento Zray and other Zend Server stuff
  • #5: Market leader
  • #13: Test everything, just because we try it doesn’t mean it always works.
  • #14: Tuning this sort of thing is a presentation in itself, but I will try and cover some of the territory and explain a few of the common tweaks we make.
  • #16: Rmem: size of receiving socket buffers Wmem: size of sending socket buffers netdev_max_backlog: processor input queue size Somaxconn: size limit on listen queue TCP tcp_max_syn_backlog: max # of remembered backlogged tcp requests
  • #17: Optimizing fastcgi correctly, Adding buffers will help a lot with the odd socket errors. Nginx is easier to use in a clustered environment The configurations are cleaner Built in Load Balancer and Reverse Proxy, very extensible Apache Performance just as well in almost all cases
  • #18: Optimizing fastcgi correctly, Adding buffers will help a lot with the odd socket errors.
  • #19: Tune Client Caching using expires When tuning Nginx cache use curl to fetch headers:
  • #20: Tune Client Caching using expires When tuning Nginx cache use curl to fetch headers:
  • #22: All of these settings are very resource dependent
  • #32: Diff your core against known sources to be able to get the junk out. AOE extensions are some of my favorite. - Scheduler - Core Store Creation was actually an AOE module
  • #43: “And I never test.”
  • #48: CCC transplanting tree
  • #52: Needs to highlight using metrics to prove performance as you go. Applications will vary, so you need to methodically use metrics to judge performance. Fassel = NY Giants coach
  • #53: We will go into a lot more detail on this later in the presentation. Basis for almost all the performance tests we are doing.
  • #54: Examples of these are rare and typically proprietary because the time to develop them and the uniqueness of the specific Application Larger clients are willing to pay for the time to develop them, or may even have internal resources. Easy solutions for partial test coverage can easily be achieved.
  • #58: Use Newrelic to supplement your JMeter tests. This test was on 2 web servers, and a single DB instance.
  • #59: This assumes you already have working jmeter or other load testing ready.
  • #60: FIRST BREAK!
  • #65: Gluster – Distributed filesystems don’t perform well
  • #66: Breaking Points, where you need to adjust first. php-fpm on it's own nodes, proxied by nginx. Varnish / Turpentine NFS becomes bottleneck Redis Clustering pairing twemproxy (nutcracker) with redis is a no-brainer as it multiplexes the redis queries, even if you just put it on the same node as redis is running. and Database Rinse and Repeat
  • #67: Partial list, full list hasn’t been published yet. Expect a blog article about this “soon”
  • #68: The demo later focuses on this type of load balancing
  • #69: The demo later focuses on this type of load balancing
  • #75: Scalearc and write-ahead-logging as a solution for single writes
  • #85: Only for EE magento, but I have added some code and have it available on my github to run in CE
  • #87: Pro-Tip, use the jmeter gui to edit and test your changes to the xml
  • #92: I have typically been just leaving Orders as 0
  • #98: Giant Trivial Pursuit @ St Vincent College
  • #102: Look at the notes in readme or on gist for more details on standup