Scylla Summit 2017: Managing 10,000 Node Storage Clusters at Twitter
Download as PPTX, PDF0 likes1,418 views
The document discusses the challenges of managing large-scale stateful services at Twitter, focusing on availability, correctness, and scalability while operating on a 10,000-node storage cluster. It emphasizes the need for efficient operations, including tools that can automate and simplify the processes to minimize operator burden and increase effectiveness. The document also suggests strategies for managing data placement, transitions, and fault detection in a complex operational environment.
Scylla Summit 2017: Managing 10,000 Node Storage Clusters at Twitter
- 1. Managing 10,000 Node Storage Clusters at Twitter Tech Lead Real Time Storage, Twitter Boaz Avital @bx
- 2. Operations are a burden
- 3. Boaz Avital Tech Lead, Real Time Storage @bx
- 4. What’s hard about managing Stateful Services? Availability Correctness Scale
- 5. Manhattan
- 6. What’s hard about managing Stateful Services? Availability Correctness Scale
- 7. Requests can only be satisfied by specific nodes Knowing Where Data Lives #MyData
- 8. Requests can only be satisfied by specific nodes Knowing Where Data Lives #MyData
- 9. Requests can only be satisfied by specific nodes Knowing Where Data Lives #MyData
- 10. Requests can only be satisfied by specific nodes Knowing Where Data Lives Topology
- 11. Requests can only be satisfied by specific nodes Knowing Where Data Lives Topology
- 12. Requests can only be satisfied by specific nodes Knowing Where Data Lives $ manifest rolling-restart
- 13. What’s hard about managing Stateful Services? Availability Correctness Scale
- 14. Requests must be sent to the right place at the right time Changing Where Data Lives #MyData
- 15. Changing Where Data Lives #MyData Requests must be sent to the right place at the right time
- 16. Changing Where Data Lives #MyData Requests must be sent to the right place at the right time
- 17. Changing Where Data Lives Placing data Shards Nodes
- 18. Changing Where Data Lives Placing data
- 19. Changing Where Data Lives Consistent Hashing Placing data
- 20. Changing Where Data Lives Shards Nodes Placing data
- 21. Changing Where Data Lives A B C D A F E B E Static Topology Placing data
- 22. Changing Where Data Lives Topology transitions Snapshot and stream Direct writes to both sets Direct reads and writes to new set Normal
- 23. Changing Where Data Lives Snapshot and stream Direct writes to both sets Direct reads and writes to new set Normal Topology transitions
- 24. Changing Where Data Lives Snapshot and stream Direct writes to both sets Direct reads and writes to new set Normal Topology transitions
- 25. Changing Where Data Lives Snapshot and stream Direct writes to both sets Direct reads and writes to new set Normal Topology transitions
- 26. Changing Where Data Lives Snapshot and stream Direct writes to both sets Direct reads and writes to new set Normal Topology transitions
- 27. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Topology transitions
- 28. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Topology transitions
- 29. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Topology transitions
- 30. Changing Where Data Lives Requests must be sent to the right place at the right time $ screen $ manifest topology-transition topology ... replica_set37: - host1.twitter.com - host2.twitter.com - host3.twitter.com :w $ manifest topology-checkout > topology $ vi topology
- 31. Changing Where Data Lives Strong Consistency Topology transitions
- 32. Changing Where Data Lives Strong Consistency Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Topology transitions
- 33. Changing Where Data Lives Strong Consistency Snapshot and stream Stop writing to old set Direct reads to new set Normal Prepare to receive writes Direct writes to both sets Topology transitions
- 34. Changing Where Data Lives Strong Consistency Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Topology transitions
- 35. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Count new node responses New nodes are caught up Strong Consistency Topology transitions
- 36. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Count new node responses New nodes are caught up Strong Consistency Topology transitions
- 37. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Count new node responses New nodes are caught up Strong Consistency Topology transitions
- 38. Changing Where Data Lives Snapshot and stream Normal Prepare to receive writes Direct writes to both sets Stop writing to old set Direct reads to new set Count new node responses New nodes are caught up HDFS Read Only Topology transitions
- 39. Changing Where Data Lives Requests must be sent to the right place at the right time $ screen $ manifest topology-transition topology ... replica_set37: - host1.twitter.com - host2.twitter.com - host3.twitter.com :w $ manifest topology-checkout > topology $ vi topology
- 40. What’s hard about managing Stateful Services? Availability Correctness Scale
- 41. Humans can reason about actions Small Scale Operations
- 42. Humans can reason about actions Small Scale Operations
- 43. Humans can reason about actions Small Scale Operations
- 44. Humans can reason about actions Small Scale Operations
- 45. Humans can reason about actions Small Scale Operations
- 46. Many simultaneous actions are required Large Scale Operations
- 47. Many simultaneous actions are required Large Scale Operations
- 48. Many simultaneous actions are required Large Scale Operations
- 49. Many simultaneous actions are required Large Scale Operations
- 50. Many simultaneous actions are required Large Scale Operations
- 51. Many simultaneous actions are required Large Scale Operations
- 52. Goal: Minimize operator burden Ensure operations are always safe to execute Minimize domain knowledge to increase effectiveness 1 3 Make operations fire-and-forget with no babysitting 2 Operators shouldn’t have to think
- 53. Building an Operations Service Tooling that thinks so your operators don’t have to Zookeeper Ops Genie
- 54. Building an Operations Service Tooling that thinks so your operators don’t have to Goal-oriented architecture Safe operational concurrency
- 55. Building an Operations Service Tooling that thinks so your operators don’t have to Old view New view Add nodes Restart nodes Remove nodes
- 56. Building an Operations Service Tooling that thinks so your operators don’t have to Goal-oriented architecture Safe operational concurrency Incremental progress Continuous data rebalancing
- 57. Building an Operations Service Tooling that thinks so your operators don’t have to Add host1 Add host2
- 58. Building an Operations Service Tooling that thinks so your operators don’t have to Goal-oriented architecture Safe operational concurrency Incremental progress Continuous data rebalancing Restart management Liveness detection
- 59. Building an Operations Service Tooling that thinks so your operators don’t have to Zookeeper
- 60. Building an Operations Service Tooling that thinks so your operators don’t have to Goal-based architecture Safe operational concurrency Incremental progress Continuous data rebalancing Restart management Liveness detection Operation throttling
- 61. Rolling restarts $ genieclient rolling-restart
- 62. Handling node failures $ genieclient node-mark-dead --nodes=host1.twitter.com
- 63. Adding nodes to a cluster $ genieclient node-add --nodes=host2,host3,host4
- 64. Pausing all operations $ genieclient freeze-on
- 67. More interesting data placement algorithms Future Work Per-dataset backends based on usage patterns Shard splitting that reacts to load Generalized cluster manager
- 68. Questions @b x