Uploaded byC4Media
Building Highly-resilient Systems at Pinterest
The document discusses Pinterest's strategies for building highly resilient systems, focusing on dynamic service discovery, real-time configuration, and caching. It highlights the importance of replication and failover, asynchronous processing, and detailed challenges in managing distributed systems. Additionally, it emphasizes the need for efficient testing and development practices to ensure system reliability.
More Related Content
Similar to Building Highly-resilient Systems at Pinterest
Building Highly-resilient Systems at Pinterest
- 2. InfoQ.com: News &Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/ pinterest-resilient-systems
- 3. Purpose of QCon -to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. Highly Resilient Systemsat Pinterest Yongsheng Wu Engineering Manager of Storage & Caching, Pinterest Email: [email protected] Pinterest: www.pinterest.com/yswu Nov 17, 2015
- 5. Our mission isto help people discover and do what they love
- 6. 50+ billion Pins categorizedby 100+ million Pinners into > 1 billion Boards
- 7. 10s of 1,000sof Instances on AWS
- 8.
- 9.
- 10. Failures Happen Allthe Time
- 11. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 12. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. Zookeeper • Highly reliabledistributed coordination • Hierarchical namespace: ZNode • Persistent Node • Ephemeral Node • Sequence Node • Watch • Node Children Changed • Node Created • Node Data Changed • Node Deleted
- 19. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1 Capacity Addition
- 20. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1 Capacity Addition
- 21. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar2 Capacity Addition
- 22. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar 2 Capacity Reduction
- 23. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 1, bar 2 Capacity Reduction
- 24. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 1 Capacity Reduction
- 25. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar 2 Zookeeper Failure
- 26. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar 2 Observers /discovery bar prod bar 1 bar 2 Zookeeper with Observers
- 27. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar 2 Observers /discovery bar prod bar 1 bar 2 Zookeeper with Observers Failure
- 28. Avoid Complete relianceon any single system, even if it is a highly reliable distributed system
- 29. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar 2 Observers /discovery bar prod bar 1 bar 2 ZUM bar bar 1, bar 2 Zookeeper with Observers Protected by Local Persistent ServerSet
- 30. Dynamic Service Discovery bar 1 Zookeeper /discovery bar prod bar1 foo foo apps bar 2 bar 2 bar 1, bar 2 Observers /discovery bar prod bar 1 bar 2 ZUM bar bar 1, bar 2 Zookeeper with Observers Protected by Local Persistent ServerSet
- 31. Challenges • Rapid plannedcapacity reduction • Gradual loss of instances
- 32. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 33.
- 34. Realtime Configuration • Decider •Experiment • Rate limiting • Failover • … …
- 35. Realtime Configuration Zookeeper /config decider setting_1: 53 foo fooapps {“setting_1”: 53, … …. } setting_2: 0 setting_3: 100 setting_4: 78 … … Config Admin Console Zookeeper Powered Realtime Configuration Management
- 36. Realtime Configuration foo foo apps v0:{“setting_1”: 53, … …. } Observers decider v0: {“setting_1”: 53, … …. } Zookeeper /config decider: v0 AWS S3 Config Admin Console /config decider: v0 ZUM Zookeeper, Observers, S3 Based Realtime Configuration Management
- 37. Realtime Configuration foo foo apps v0:{“setting_1”: 53, … …. } Observers ZUM decider v0: {“setting_1”: 53, … …. } Zookeeper /config decider: v0 AWS S3 Config Admin Console /config decider: v0 v1: {“setting_1”: 54, … …. } v1 Zookeeper, Observers, S3 Based Realtime Configuration Management
- 38. Realtime Configuration foo foo apps v0:{“setting_1”: 53, … …. } Observers ZUM decider v0: {“setting_1”: 53, … …. } Zookeeper /config decider: v1 AWS S3 Config Admin Console /config decider: v0 v1: {“setting_1”: 54, … …. } v1 Zookeeper, Observers, S3 Based Realtime Configuration Management
- 39. Realtime Configuration foo foo apps v0:{“setting_1”: 53, … …. } Observers decider v0: {“setting_1”: 53, … …. } Zookeeper /config decider: v1 AWS S3 Config Admin Console /config decider: v1 v1: {“setting_1”: 54, … …. } v1 ZUM v1: {“setting_1”: 54, … …. } Zookeeper, Observers, S3 Based Realtime Configuration Management
- 40. Realtime Configuration foo foo apps v1:{“setting_1”: 54, … …. } Observers decider v0: {“setting_1”: 53, … …. } Zookeeper /config decider: v1 AWS S3 Config Admin Console /config decider: v1 v1: {“setting_1”: 54, … …. } v1 ZUM v1: {“setting_1”: 54, … …. } Zookeeper, Observers, S3 Based Realtime Configuration Management
- 41. Realtime Configuration foo foo apps v1:{“setting_1”: 54, … …. } Observers decider v1: {“setting_1”: 54, … …. } Zookeeper /config decider: v1 AWS S3 Config Admin Console /config decider: v1 v1: {“setting_1”: 54, … …. } v1 ZUM v1: {“setting_1”: 54, … …. } Zookeeper, Observers, S3 Based Realtime Configuration Management
- 42. Realtime Configuration foo foo apps v1:{“setting_1”: 54, … …. } Observers decider v1: {“setting_1”: 54, … …. } Zookeeper /config decider: v1 AWS S3 Config Admin Console /config decider: v1 ZUM Zookeeper, Observers, S3 Based Realtime Configuration Management
- 43. Challenges • Staggered rollout •Configuration of huge size
- 44. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 45.
- 46.
- 47. Caching bar cache 1 bar cache m proc1 … … prock Foo 2 …… proc1 … … prock Foo 1 … … proc1 … … prock Foo n Too many connections!
- 48. Twemproxy bar cache 1 bar cache m … … proc1 … … prock Foo1 … … proc1 … … prock Foo 2 proc1 … … prock Foo n NutcrackerNutcracker Nutcracker
- 49. Cache Inconsistency Nutcracker … …foo:1 bar cache 1 bar cache 2 bar cache n set foo 2 foo: 2
- 50. Cache Inconsistency Nutcracker … …foo:1 bar cache 1 bar cache 2 bar cache n foo: 2
- 51. McRouter … … cache 1cache 2 cache n McRouter
- 52. McRouter … … cache 1cache 2 cache n McRouter No ring reshuffle
- 53. McRouter Pros • No inconsistencycaused by node joining/leaving the pool • No cascading failures in case of excessive load caused by hot keys Cons • Cache misses
- 54. Replicated Pools -Reads McRouter cache 1 cache 2 cache n… cache 1’ cache 2’ cache n’… ac Pool de Poolgetfoo
- 55. Replicated Pools -Reads McRouter cache 1 cache 2 cache n… cache 1’ cache 2’ cache n’… ac Pool de Poolgetfoo
- 56. Replicated Pools -Invalidation McRouter cache 1 cache 2 cache n… cache 1’ cache 2’ cache n’… ac Pool de Pooldeletefoo
- 57. Replicated Pools -Invalidation McRouter cache 1 cache 2 cache n… cache 1’ cache 2’ cache n’… ac Pool de Pooldeletefoo Log
- 58. Replicated Pools -Invalidation McRouter cache 1 cache 2 cache n… cache 1’ cache 2’ cache n’… ac Pool de Pooldeletefoo Log
- 59. Replicated Pools -Invalidation McRouter cache 1 cache 2 cache n… cache 1’ cache 2’ cache n’… ac Pool de Pooldeletefoo LogSinger kafka tailer PinLater
- 60. Challenges • Build thefeedback loop from persistent layer to caching layer • Move to multiple geographic regions
- 61. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 62.
- 63. Sharding Shard Id TypeLocal Id 64 bits
- 64. Clients DataServices 1 … … Master 1 Slave 1 …… Master m Slave m Clients Clients DataServices 2 DataServices n Read from Slave
- 65. Clients DataServices 1 … … Master 1 Slave 1 …… Master m Slave m Clients Clients Read from Slave DataServices 2 DataServices n Read from slave after master failing health check over a certain period of time.
- 66. Master 1 Slave 1 … … Master m Slave m Failover DataServicesProc ZUM ShardConfig ShardConfig Changes Realtime Configuration Management Powered Failover
- 67. Other Persistence Stores UMetaStore •Key value store based on HBase Zen • Graph store: nodes and edges • Flexible schema • Custom index • Both HBase and MySQL Future • Rocksdb
- 68. Challenges • Automated failoverwith MySQL replica set • HBase 1.x Upgrade • Move to multiple geographic regions
- 69. Replication and failoverare the key ingredients for building highly resilient storage and caching systems
- 70. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 71. Async Processing Use Cases •Acknowledge success with non-time-sensitive actions taken at later time • Schedule and execute large number of jobs Benefits • Faster response time • More resilient to dependent system failures
- 72. Pyres Limitations • Nomechanism for success acknowledgement • No visibility into status of individual job types • No support for scheduled job execution at a specific time in the future • Rate limiting and retries are hard to manage • Redis as only supported storage backend
- 73. PinLater Asynchronous Processing System Clients Pinlater Servers Infra Worker Pool Infra Worker Pool Worker Pools MasterSlave Clients Clients Pinlater Servers PinLater Servers Storage Backend Enqueue Dequeue/ACK
- 74. PinLater Job StateTransition Pending Running Succeeded Failed Done Failed (no more retries) Dequeued Failed (retries left)/Claim Timeout
- 75.
- 76.
- 77. Clients PinLater Server 1 PinLater Server 2 PinLater Server n … … Master 1 Slave 1 … … Master m Slave m Infra Worker Pool Infra Worker Pool Worker Pool 1 …… Infra Worker Pool Infra Worker Pool Worker Pool k Clients Clients Enqueue PinLater Asynchronous Processing System
- 78. Clients PinLater Server 1 PinLater Server 2 PinLater Server n … … Master 1 Slave 1 … … Master m Slave m Infra Worker Pool Infra Worker Pool Worker Pool 1 …… Infra Worker Pool Infra Worker Pool Worker Pool k Clients Clients Enqueue PinLater Asynchronous Processing System
- 79. Clients PinLater Server 1 PinLater Server 2 PinLater Server n … … Master 1 Slave 1 … … Master m Slave m Infra Worker Pool Infra Worker Pool Worker Pool 1 …… Infra Worker Pool Infra Worker Pool Worker Pool k Clients Clients Enqueue PinLater Asynchronous Processing System
- 80. Clients PinLater Server 1 PinLater Server 2 PinLater Server n … … Master 1 Slave 1 … … Master m Slave m Infra Worker Pool Infra Worker Pool Worker Pool 1 …… Infra Worker Pool Infra Worker Pool Worker Pool k Clients Clients Enqueue PinLater Asynchronous Processing System
- 81. Clients PinLater Server 1 PinLater Server 2 PinLater Server n … … Master 1 Slave 1 … … Master m Slave m Infra Worker Pool Infra Worker Pool Worker Pool 1 …… Infra Worker Pool Infra Worker Pool Worker Pool k Clients Clients Enqueue PinLater Asynchronous Processing System
- 83. Challenges • Multi-tenancy withfair failure isolation • Fault-tolerant async job enqueuer
- 84. Use async processingas much as possible to deliver faster response time and make request handling more robust
- 85. Dynamic Service Discovery RealtimeConfiguration Caching Persistent Storage Async Processing
- 86. Learnings • Avoid Completereliance on any single system, even if it is a highly reliable distributed system • Replication and failover are the key ingredients for building highly resilient storage and caching systems • Use async processing as much as possible to deliver faster response time and make request handling more robust
- 87. Failure Testing • Beexplicit with scope • Failure Modes • Sandbox testing • Manual testing • Automated simulation • Testing on production • AWS is doing it for us all the time • Simian Army
- 89. Watch the videowith slide synchronization on InfoQ.com! http://www.infoq.com/presentations/pinterest- resilient-systems