SlideShare a Scribd company logo
PHP at 5000 Requests / Sec 
Hootsuite’s Scaling Story 
Bill Monkman 
Lead Technical Engineer - Platform 
@bmonkman
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story
Overview - Selected Current Architecture 
Users lb1 lb2 lb3 ... Nginx Load balancers 
web1 web2 web3 ... Nginx web servers 
PHP-FPM PHP-FPM PHP-FPM PHP-FPM 
Memcached cluster 
mem1 ... 
Mysql cluster 
master slave 
MongoDB cluster 
master slave 
master slave 
shard1 
shard2 
Gearman cluster 
geard1 geard2 
worker1 ... ... 
... 
Services
Technologies - at first 
• Apache 
• PHP 
• MySQL
Then...
Problem 
It’s hard to scale MySQL horizontally
Solution - Caching 
Memcached. 
● Distributed cache, cluster of boxes with lots of RAM, trivial to scale 
● Cache as much as possible, invalidate only when necessary 
● Use cache instead of DB 
● No joins - decouple entities (collection caching) 
● Twemproxy!
“There are only two hard things in 
Computer Science: cache invalidation and 
naming things.” 
• Phil Karlton
Solution - Caching 
MvcModelBaseCaching 
MvcModelBase 
MvcModelMysql 
SocialNetwork
Solution - Caching 
SELECT * FROM member WHERE org_id=888 
set individual cache records 
member_1 {data} 
member_5 {data} 
member_9 {data} 
set collection cache 
member_org_888 [1,5,9] 
Automatic invalidation of collection cache
Solution - Caching 
It’s hard to scale MySQL horizontally 
Now: 
● No need to scale MySQL 
● Able to serve the whole site on 1 MySQL server 
● 500 MySQL SELECTs per second. 50,000 Memcached GETs. 
● 99+% hit rate
Then...
Problem 
Need a way to perform asynchronous, distributed tasks using a 
single-threaded language.
Solution - Gearman 
Gearman. 
● Distribute work to other servers to handle (workers also using 
PHP, same codebase) 
● Precursor to SOA where everything is truly distributed 
● Many other solutions, queueing systems.
Solution - Gearman 
geard1 geard2 
gearworker1 gearworker2 gearworker6
Solution - Gearman 
Need a way to perform asynchronous, distributed tasks using a 
single-threaded language. 
Now: 
● Moved key tasks to Gearman 
● Another cluster, scalable separately from web 
● Discrete tasks, callable sync or async
Then...
Problem 
Need to store data with the potential to grow too big to handle 
effectively with MySQL.
Solution - MongoDB 
MongoDB. 
● Certain data did not need to be highly relational 
● NoSQL DB, many other solutions these days 
● Mongo can be a pain, lots of moving parts 
● Had to make our own sequencer where auto-incremented ids were 
necessary
Solution - MongoDB 
Need to store data with the potential to grow too big to handle 
effectively with MySQL. 
Now: 
● Multiple clusters containing amounts of data that likely would 
have crushed MySQL 
● Billions of rows per collection, many TB of data on disk
Technologies 
• Apache 
• PHP 
• MySQL 
• Memcached 
• Gearman 
• MongoDB
Then...
Problem 
With a codebase and an engineering team increasing in size, how do 
we keep up the pace of development and maintain control of the 
system? 
(SVN, big branches, merge hell)
Solution - Dark Launching 
Dark Launching. 
● Wrap code in block with a specific name 
● That name will appear in a management page 
● Can control whether or not that block is executed by modifying it’s value 
● Boolean , random percentage, session-based, member list, organization 
list, etc.
Solution - Dark Launching 
if (In_Feature::isEnabled(‘TWITTER_ADS’)) { 
// execute new code 
} else { 
// execute old code 
}
Dark Launching - Reasons 
• Control your code 
• Limit risk -> raise confidence -> speed up pace of releases 
• “Branching in Production” 
• Learning happens in Production
Solution - Dark Launching 
With a codebase and an engineering team increasing in size, how do 
we keep up the pace of development and maintain control of the 
system? 
Now: 
● Work fast with more confidence 
● Huge amount of control over production systems 
● Typically 10+ code releases to production per day 
● Push-based distribution with Consul
Then...
Problem 
With a rapidly increasing codebase and amount of users / traffic 
how do we keep visibility into the performance of the code?
Solution - Monitoring 
Statsd / Graphite. 
Logstash / Elasticsearch / Kibana. 
Sensu 
● Statsd for metrics 
● Logstash for log events 
● Sensu for monitoring / alerting
Solution - Monitoring 
Statsd::timing('apiCall.facebookGraph', microtime(true) - $startTime);
Solution - Monitoring 
Logger::event('user liked from in-stream', In_Log::CATEGORY_UX, $logData);
Solution - Monitoring 
• Visibility into the performance and behaviour of your application 
• Iterate upon your code, measure results 
• Pairs well with dark launching 
• Also systems like New Relic
Solution - Monitoring 
With a rapidly increasing codebase and amount of users / traffic 
how do we keep visibility into the performance of the code? 
Now: 
● Able to watch performance / behaviour in real time. 
● Able to view important events both in the aggregate or very 
granular 
● Able to control the system and watch the effect of changes
Optimizations
Optimizations 
• Things expand beyond their initial scope 
• Case in point: Translations
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story
Optimizations - Push work to users 
• Within reason, push work up to users 
• Make your users into a distributed processing grid 
• e.g. Stream rendering
Optimizations - Performance / Risks 
• Performance is more important than clean code, business reqts 
(in the instances where they may be mutually exclusive) 
• Fine line between future proofing and premature optimization 
• Don’t add burdensome processes, but make it easy for your team 
to do things the right way 
• Know your weak spots, protect against abuse
PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story
Technologies 
Linux 
Nginx 
ElasticSearch Varnish 
PHP-FPM 
MySQL 
Jenkins 
Scala 
MongoDB 
Consul 
Gearman 
Redis 
Akka 
Python 
Memcached 
HAProxy 
jQuery 
ZeroMQ 
Backbone RabbitMQ 
EC2 
Zend 
Docker 
Cloudfront CDN 
Logstash 
Zookeeper 
Kibana 
Statsd/Graphite 
Packer 
Vagrant 
Nagios 
VirtualBox 
Spark/Shark 
Sensu 
Symfony 
Riak 
Composer 
Websockets 
Comet 
Hadoop 
Ansible 
Git 
Webpack Redshift
Problem 
With a huge and growing monolithic codebase and over 80 
engineers, how to keep scaling in a manageable way?
Solution - SOA 
SOA. 
● Split up the system into independent services which communicate only via APIs 
● Teams can work on their own services with encapsulated business logic and have their own 
deployment schedules. 
● We chose to use Scala/Akka for services, communicating via ZeroMQ 
● SOA transition made easier by the “no joins” philosophy 
● Tons of work
Solution - SOA 
SOM. 
● “Service Oriented Monolith” 
● When splitting up a monolithic codebase, dependencies are what kill you 
● Fulfill dependencies by writing interim services using existing PHP code 
● Maintain the contract and future scala services will be drop-in 
replacements
Solution - SOA 
With a huge and growing monolithic codebase and over 130 
engineers, how to keep scaling in a manageable way? 
Today: 
● Transitioning to Scala SOA 
● PHP will still be used as the Façade, a thin layer built on top of 
the business logic of the services it interacts with.
Conclusion
Thank You! 
Bill Monkman 
@bmonkman 
More Info: 
code.hootsuite.com

More Related Content

What's hot (20)

PPTX
Introduction to WordPress
Harshad Mane
 
PPTX
Apache Spark Architecture
Alexey Grishchenko
 
PDF
Guide to alfresco monitoring
Miguel Rodriguez
 
PPTX
Apache Hive Tutorial
Sandeep Patil
 
PDF
Mongodb - Scaling write performance
Daum DNA
 
PPSX
Hadoop
Nishant Gandhi
 
PDF
MapReduce: Simplified Data Processing on Large Clusters
Ashraf Uddin
 
PDF
Windows内核技术介绍
jeffz
 
PDF
Linux Sistem Yönetimi
Kurtuluş Karasu
 
PDF
Google Bigtable Paper Presentation
vanjakom
 
PDF
Presto Summit 2018 - 09 - Netflix Iceberg
kbajda
 
PDF
Building Embedded Linux Full Tutorial for ARM
Sherif Mousa
 
PDF
SeaweedFS introduction
chrislusf
 
PDF
Introduction to elasticsearch
hypto
 
PPTX
Wordpress ppt
Crest TechnoSoft
 
PDF
Secrets of the DSpace Submission Form
Bram Luyten
 
PDF
MariaDB 10: The Complete Tutorial
Colin Charles
 
PDF
Spark Operator—Deploy, Manage and Monitor Spark clusters on Kubernetes
Databricks
 
PPT
Grundlagen Virtualisierung
inovex GmbH
 
Introduction to WordPress
Harshad Mane
 
Apache Spark Architecture
Alexey Grishchenko
 
Guide to alfresco monitoring
Miguel Rodriguez
 
Apache Hive Tutorial
Sandeep Patil
 
Mongodb - Scaling write performance
Daum DNA
 
MapReduce: Simplified Data Processing on Large Clusters
Ashraf Uddin
 
Windows内核技术介绍
jeffz
 
Linux Sistem Yönetimi
Kurtuluş Karasu
 
Google Bigtable Paper Presentation
vanjakom
 
Presto Summit 2018 - 09 - Netflix Iceberg
kbajda
 
Building Embedded Linux Full Tutorial for ARM
Sherif Mousa
 
SeaweedFS introduction
chrislusf
 
Introduction to elasticsearch
hypto
 
Wordpress ppt
Crest TechnoSoft
 
Secrets of the DSpace Submission Form
Bram Luyten
 
MariaDB 10: The Complete Tutorial
Colin Charles
 
Spark Operator—Deploy, Manage and Monitor Spark clusters on Kubernetes
Databricks
 
Grundlagen Virtualisierung
inovex GmbH
 

Similar to PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story (20)

PDF
Fixing twitter
Roger Xia
 
PDF
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
smallerror
 
PDF
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
xlight
 
PDF
Fixing_Twitter
liujianrong
 
PDF
Chirp 2010: Scaling Twitter
John Adams
 
PDF
John adams talk cloudy
John Adams
 
PDF
Qcon
adityaagarwal
 
PPTX
SOA with PHP and Symfony
MichalSchroeder
 
PDF
Experiences with Microservices at Tuenti
Andrés Viedma Peláez
 
PDF
Building data intensive applications
Amit Kejriwal
 
PDF
Modern software architectures - PHP UK Conference 2015
Ricard Clau
 
PPT
UnConference for Georgia Southern Computer Science March 31, 2015
Christopher Curtin
 
PDF
Tool up your lamp stack
AgileOnTheBeach
 
PDF
Tool Up Your LAMP Stack
Lorna Mitchell
 
PDF
Microservices Antipatterns
C4Media
 
PDF
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
LinkedIn
 
PDF
Service-Oriented Design and Implement with Rails3
Wen-Tien Chang
 
KEY
Fixing Twitter Velocity2009
John Adams
 
PDF
What drives Innovation? Innovations And Technological Solutions for the Distr...
Stefano Fago
 
PPTX
The Hard Problems of Continuous Deployment
Timothy Fitz
 
Fixing twitter
Roger Xia
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
smallerror
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
xlight
 
Fixing_Twitter
liujianrong
 
Chirp 2010: Scaling Twitter
John Adams
 
John adams talk cloudy
John Adams
 
SOA with PHP and Symfony
MichalSchroeder
 
Experiences with Microservices at Tuenti
Andrés Viedma Peláez
 
Building data intensive applications
Amit Kejriwal
 
Modern software architectures - PHP UK Conference 2015
Ricard Clau
 
UnConference for Georgia Southern Computer Science March 31, 2015
Christopher Curtin
 
Tool up your lamp stack
AgileOnTheBeach
 
Tool Up Your LAMP Stack
Lorna Mitchell
 
Microservices Antipatterns
C4Media
 
Ruslan Belkin And Sean Dawson on LinkedIn's Network Updates Uncovered
LinkedIn
 
Service-Oriented Design and Implement with Rails3
Wen-Tien Chang
 
Fixing Twitter Velocity2009
John Adams
 
What drives Innovation? Innovations And Technological Solutions for the Distr...
Stefano Fago
 
The Hard Problems of Continuous Deployment
Timothy Fitz
 
Ad

Recently uploaded (20)

PDF
July Patch Tuesday
Ivanti
 
PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PDF
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
PDF
Predicting the unpredictable: re-engineering recommendation algorithms for fr...
Speck&Tech
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
SFWelly Summer 25 Release Highlights July 2025
Anna Loughnan Colquhoun
 
PDF
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Windsurf Meetup Ottawa 2025-07-12 - Planning Mode at Reliza.pdf
Pavel Shukhman
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PDF
Why Orbit Edge Tech is a Top Next JS Development Company in 2025
mahendraalaska08
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PDF
Smart Air Quality Monitoring with Serrax AQM190 LITE
SERRAX TECHNOLOGIES LLP
 
PDF
Learn Computer Forensics, Second Edition
AnuraShantha7
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PPTX
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
July Patch Tuesday
Ivanti
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
Predicting the unpredictable: re-engineering recommendation algorithms for fr...
Speck&Tech
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
SFWelly Summer 25 Release Highlights July 2025
Anna Loughnan Colquhoun
 
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Windsurf Meetup Ottawa 2025-07-12 - Planning Mode at Reliza.pdf
Pavel Shukhman
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
Why Orbit Edge Tech is a Top Next JS Development Company in 2025
mahendraalaska08
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
Smart Air Quality Monitoring with Serrax AQM190 LITE
SERRAX TECHNOLOGIES LLP
 
Learn Computer Forensics, Second Edition
AnuraShantha7
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
Ad

PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story

  • 1. PHP at 5000 Requests / Sec Hootsuite’s Scaling Story Bill Monkman Lead Technical Engineer - Platform @bmonkman
  • 3. Overview - Selected Current Architecture Users lb1 lb2 lb3 ... Nginx Load balancers web1 web2 web3 ... Nginx web servers PHP-FPM PHP-FPM PHP-FPM PHP-FPM Memcached cluster mem1 ... Mysql cluster master slave MongoDB cluster master slave master slave shard1 shard2 Gearman cluster geard1 geard2 worker1 ... ... ... Services
  • 4. Technologies - at first • Apache • PHP • MySQL
  • 6. Problem It’s hard to scale MySQL horizontally
  • 7. Solution - Caching Memcached. ● Distributed cache, cluster of boxes with lots of RAM, trivial to scale ● Cache as much as possible, invalidate only when necessary ● Use cache instead of DB ● No joins - decouple entities (collection caching) ● Twemproxy!
  • 8. “There are only two hard things in Computer Science: cache invalidation and naming things.” • Phil Karlton
  • 9. Solution - Caching MvcModelBaseCaching MvcModelBase MvcModelMysql SocialNetwork
  • 10. Solution - Caching SELECT * FROM member WHERE org_id=888 set individual cache records member_1 {data} member_5 {data} member_9 {data} set collection cache member_org_888 [1,5,9] Automatic invalidation of collection cache
  • 11. Solution - Caching It’s hard to scale MySQL horizontally Now: ● No need to scale MySQL ● Able to serve the whole site on 1 MySQL server ● 500 MySQL SELECTs per second. 50,000 Memcached GETs. ● 99+% hit rate
  • 13. Problem Need a way to perform asynchronous, distributed tasks using a single-threaded language.
  • 14. Solution - Gearman Gearman. ● Distribute work to other servers to handle (workers also using PHP, same codebase) ● Precursor to SOA where everything is truly distributed ● Many other solutions, queueing systems.
  • 15. Solution - Gearman geard1 geard2 gearworker1 gearworker2 gearworker6
  • 16. Solution - Gearman Need a way to perform asynchronous, distributed tasks using a single-threaded language. Now: ● Moved key tasks to Gearman ● Another cluster, scalable separately from web ● Discrete tasks, callable sync or async
  • 18. Problem Need to store data with the potential to grow too big to handle effectively with MySQL.
  • 19. Solution - MongoDB MongoDB. ● Certain data did not need to be highly relational ● NoSQL DB, many other solutions these days ● Mongo can be a pain, lots of moving parts ● Had to make our own sequencer where auto-incremented ids were necessary
  • 20. Solution - MongoDB Need to store data with the potential to grow too big to handle effectively with MySQL. Now: ● Multiple clusters containing amounts of data that likely would have crushed MySQL ● Billions of rows per collection, many TB of data on disk
  • 21. Technologies • Apache • PHP • MySQL • Memcached • Gearman • MongoDB
  • 23. Problem With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system? (SVN, big branches, merge hell)
  • 24. Solution - Dark Launching Dark Launching. ● Wrap code in block with a specific name ● That name will appear in a management page ● Can control whether or not that block is executed by modifying it’s value ● Boolean , random percentage, session-based, member list, organization list, etc.
  • 25. Solution - Dark Launching if (In_Feature::isEnabled(‘TWITTER_ADS’)) { // execute new code } else { // execute old code }
  • 26. Dark Launching - Reasons • Control your code • Limit risk -> raise confidence -> speed up pace of releases • “Branching in Production” • Learning happens in Production
  • 27. Solution - Dark Launching With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system? Now: ● Work fast with more confidence ● Huge amount of control over production systems ● Typically 10+ code releases to production per day ● Push-based distribution with Consul
  • 29. Problem With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?
  • 30. Solution - Monitoring Statsd / Graphite. Logstash / Elasticsearch / Kibana. Sensu ● Statsd for metrics ● Logstash for log events ● Sensu for monitoring / alerting
  • 31. Solution - Monitoring Statsd::timing('apiCall.facebookGraph', microtime(true) - $startTime);
  • 32. Solution - Monitoring Logger::event('user liked from in-stream', In_Log::CATEGORY_UX, $logData);
  • 33. Solution - Monitoring • Visibility into the performance and behaviour of your application • Iterate upon your code, measure results • Pairs well with dark launching • Also systems like New Relic
  • 34. Solution - Monitoring With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code? Now: ● Able to watch performance / behaviour in real time. ● Able to view important events both in the aggregate or very granular ● Able to control the system and watch the effect of changes
  • 36. Optimizations • Things expand beyond their initial scope • Case in point: Translations
  • 39. Optimizations - Push work to users • Within reason, push work up to users • Make your users into a distributed processing grid • e.g. Stream rendering
  • 40. Optimizations - Performance / Risks • Performance is more important than clean code, business reqts (in the instances where they may be mutually exclusive) • Fine line between future proofing and premature optimization • Don’t add burdensome processes, but make it easy for your team to do things the right way • Know your weak spots, protect against abuse
  • 42. Technologies Linux Nginx ElasticSearch Varnish PHP-FPM MySQL Jenkins Scala MongoDB Consul Gearman Redis Akka Python Memcached HAProxy jQuery ZeroMQ Backbone RabbitMQ EC2 Zend Docker Cloudfront CDN Logstash Zookeeper Kibana Statsd/Graphite Packer Vagrant Nagios VirtualBox Spark/Shark Sensu Symfony Riak Composer Websockets Comet Hadoop Ansible Git Webpack Redshift
  • 43. Problem With a huge and growing monolithic codebase and over 80 engineers, how to keep scaling in a manageable way?
  • 44. Solution - SOA SOA. ● Split up the system into independent services which communicate only via APIs ● Teams can work on their own services with encapsulated business logic and have their own deployment schedules. ● We chose to use Scala/Akka for services, communicating via ZeroMQ ● SOA transition made easier by the “no joins” philosophy ● Tons of work
  • 45. Solution - SOA SOM. ● “Service Oriented Monolith” ● When splitting up a monolithic codebase, dependencies are what kill you ● Fulfill dependencies by writing interim services using existing PHP code ● Maintain the contract and future scala services will be drop-in replacements
  • 46. Solution - SOA With a huge and growing monolithic codebase and over 130 engineers, how to keep scaling in a manageable way? Today: ● Transitioning to Scala SOA ● PHP will still be used as the Façade, a thin layer built on top of the business logic of the services it interacts with.
  • 48. Thank You! Bill Monkman @bmonkman More Info: code.hootsuite.com