Tupperware: Containerized Deployment at FB

Tupperware: Containerized Deployment at FB

• 1.
  Tupperware: Containerized Deployment atFB Aravind Narayanan [email protected] DockerCon 2014
• 2.
  Scale makes everythingharder • Running single instance: easy • Running at scale in production: messy and complicated Provision machines Distribute binaries Daemonize processMonitoring Failover Geo-distribution Machine decoms
• 3.
  Time spent on ApplicationLogic Time spent on getting app to run in prod
• 4.
  Tupperware to therescue “This is my binary. Run it on X machines!” • Engineer is hands-off • Doesn’t need to worry about machines in prod • Handles failover, when machines go bad • Efficient use of infrastructure • 300,000+ processes, spread over 15,000+ services
• 5.
  Agenda 1. Architecture 2. Sandboxes 3.Ecosystem 4. Lessons learnt
• 6.
  Facebook Datacenters PRN SNC FRC ASH LLA
• 7.
  Terminology • A DChas one or more clusters • A cluster has multiple racks • A rack has multiple machines ! • A TW job is equivalent to a service • A job has multiple tasks, each an instance of the service
• 8.
  Architecture Scheduler Host1 Host2 Host3 config.tw twdeploy ServerDB Start Task BitTorrent-based binary package store QuoteService
• 9.
  Machine “failures” /hour Failover
• 10.
  Failover Scheduler Host1 Host2 Host3 ServerDB Start Task QuoteService QuoteService Heartbeat BitTorrent-based binary package store
• 11.
  Painless Hardware maintenance •Notify scheduler of impending operations • Scheduler can preemptively move tasks • Graceful migration for stateless services • Stateful services may endure maintenance
• 12.
  Expressive allocation policies ProductionHost MyBigJob QuoteService QuoteAggregator Production Host Top of Rack Switch NetworkHogJob NetworkHogJob Job M Job N Job M Job N
• 13.
  TW Agent Agent Helper TWAgent process Package Manager Resource Manager Task Manager API Scheduler heartbeat Agent Helper Task B Agent Helper Task C Task A Production Host
• 14.
  Agent Helper process Heartbeatwith Agent, prevents zombies TW Agent process Package Manager Resource Manager Task Manager API Scheduler heartbeat Agent Helper Task A
• 15.
  Logging Compress on thefly Log Files • Persistent logs • Instantaneous rotation Task A Agent Helper Log Catcher stderr stdout
• 16.
  Sandboxing Initially, used chrootsto contain processes • No isolation • Not secure
• 17.
  LinuXContainers • As techmatured, we switched • Separate process and file namespaces, set up by Helper • Mount required resources directly into container • Secure & isolated
• 18.
  Service permissions • Everycontainer runs sshd • SSH directly into the container • Regulate access Production Host TW Agent process Task Manager API Agent Helper Task A SSH X
• 19.
  Configuring the container
• 20.
  Resource limits • CPU,RAM & disk limits • Implemented with cgroups • Agent handles memory limits with cgroup notification API • Adaptive limits
• 21.
  Resource Limits inaction watchdog-service - tw.mem.rss_bytes
• 22.
  Migrate from Chrootsto Containers • No-op for most services • But new namespaces posed problems for some • Major hurdle was social, not technical
• 23.
  Service Discovery Scheduler Host1 Host2Host3 Server DB Service Registry QuoteService QuoteService Host1:12345 ALIVE Host2:12345 ALIVE QuoteService Host2:12345 ALIVE Host1:12345 DEAD Start Task QuoteService QuoteService Host2:12345 ALIVE Host3:12345 ALIVE QuoteService Start Task
• 24.
  Monitoring & Alerting
• 25.
  Alternatives to Tupperware •Why not use Docker / CoreOS? • They didn’t exist • TW integrates with other FB systems • Why not use VMs? • Performance penalty • Hypervisor makes debugging harder
• 26.
  Lessons learnt
• 27.
  Releases are scary! •Release often • Dry runs • Canaries are your friends • Manage dependencies
• 28.
  Sane defaults • Usersshouldn’t have to read entire manual • Choose what makes sense for most services
• 29.
  What went wrong? •Hard to understand why TW did something • It’s not about “what went wrong”, but “what should I do next?”
• 30.
  Tupperware • Automated deployment •Less work for engineers • Containers for security and isolation • Increased efficiency Questions? Time spent on getting app to run in prod Time spent on Application Logic