Architecting for Failure in a Containerized World
0 likes•237 views
The document discusses strategies for building robust systems using container technology, highlighting the importance of decomposition, orchestration, and managing stateful applications. Key rules include using battle-tested tools, minimizing complexity, and ensuring proper state management through various database architecture options. It emphasizes the necessity of creating frameworks for reliability and the significance of separation of concerns to enhance system robustness.
Architecting for Failure in a Containerized World
- 1. Architecting for Failure in a Containerized World Tom Faulhaber Infolace
- 5. How can container tech help us build robust systems?
- 6. Key takeaway: an architectural toolkit for building robust systems with containers
- 7. The Rules Decomposition Orchestration and Synchronization Managing Stateful Apps
- 8. Simplicity
- 9. Simple means: “Do one thing!”
- 10. The opposite of simple is complex
- 13. Example: a counter Counter Service 1 2 3 4 50 … Counter Service 1 2 3 4 50 …x Counter Service 1 2 3 4 50 1 2 3 4 50 1 2 3 4 50
- 14. Example: a counter Counter Service 1 2 3 4 50 … Counter Service 1 2 3 4 50 … Load Balancer 1 2 3 4 50 1 2 3 4 50
- 15. State + composition = complexity
- 19. App Server
- 27. Redis MySQL
- 28. Rule: High code churn →Easy restart
- 30. time a b c d
- 31. time x a b c d
- 34. time a b c d
- 35. time a b c d
- 36. time a b c d
- 38. The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Managing Stateful Apps
- 41. • Mesos: Marathon always restarts • Kubernetes: RestartPolicy=Always • Docker: Swarm always restarts
- 42. Rule: Create your own framework
- 45. Synchronized State Tools: - zookeeper - etcd - consul Patterns: - leader election - shared counters - peer awareness - work partitioning
- 47. Even battle-tested state management is a headache. (Source: http://blog.cloudera.com/blog/2014/03/zookeeper-resilience-at-pinterest/)
- 48. The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Use framework restarts Create your own framework Use synchronized state Minimize synchronized state Managing Stateful Apps
- 49. Part 3: Managing Stateful Apps
- 50. Rule (repeat!): Always use battle-tested tools! (State is the weak point)
- 51. Rule: Choose the DB architecture
- 52. Option 1: External DB Execution cluster Database cluster
- 53. Option 1: External DB Pros • Somebody else’s problem! • Can use a DB designed for clustering directly • Can use DB as a service Cons • Not really somebody else’s problem! • Higher latency/no reference locality • Can’t leverage orchestration, etc.
- 54. Option 2: Run on Raw HW HDFS Mesos Marathon App HDFS Mesos Marathon App HDFS Mesos Marathon App
- 55. Option 2: Run on Raw HW Pros • Use existing recipes • Have local data • Manage a single cluster Cons • Orchestration doesn’t help with failure • Increased management complexity
- 56. Option 3: In-memory DB Mesos Marathon App MemSQL Mesos Marathon App MemSQL Mesos Marathon App MemSQL
- 57. Option 3: In-memory DB Pros • No need for volume tracking • Fast • Have local data • Manage a single cluster Cons • Bets all machines won’t go down • Bets on orchestration framework
- 58. Option 4: Use Orchestration Mesos Marathon App Cassandra Mesos Marathon App Cassandra Mesos Marathon App Cassandra
- 59. Option 4: Use Orchestration Pros • Orchestration manages volumes • One model for all programs • Have local data • Single cluster Cons • Currently the least mature • Not well supported by vendors
- 60. Option 5: Roll Your Own Mesos Marathon App ImageMgr Mesos Master Framework Mesos Marathon App ImageMgr Mesos Marathon App ImageMgr
- 61. Option 5: Roll Your Own Pros • Very precise control • You decide whether to use containers • Have local data • Can be system aware Cons • You’re on your own! • Wedded to a single orchestration platform • Not battle tested
- 63. The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Use framework restarts Create your own framework Use synchronized state Minimize synchronized state Managing Stateful Apps Battle-tested tools Choose the DB architecture Have replication
- 64. Fin
- 65. References • Rich Hickey: “Are We There Yet?” (https://www.infoq.com/presentations/Are-We- There-Yet-Rich-Hickey) “Simple Made Easy” (https://www.infoq.com/presentations/Simple- Made-Easy-QCon-London-2012) • David Greenberg, Building Applications on Mesos, O’Reilly, 2016 • Joe Johnston, et al., Docker in Production: Lessons from the Trenches, Bleeding Edge Press, 2015
- 66. The Rules Decomposition Decompose vertically Separation of concerns Constrain state Battle-tested tools High code churn, easy restart No start-up order! Consider higher-order failure Orchestration and Synchronization Use framework restarts Create your own framework Use synchronized state Minimize synchronized state Managing Stateful Apps Battle-tested tools Choose the DB architecture Have replication