Adopting Actors: An epic tail of loss and learning
Download as PPTX, PDF1 like252 views
The document discusses challenges and lessons learned in adopting actor-based systems for task management. It highlights issues such as message flow consistency, state management, and the importance of testing for resilience in distributed systems. Key takeaways include the distinction between actor modeling and object modeling, emphasizing the need for atomicity and consistency in actor states.
Adopting Actors: An epic tail of loss and learning
- 1. Adopting actors An epic tail of loss and learning
- 3. Workday
- 4. Growth 2013 2014 2015 2016
- 5. Cloud Master Launch tasks Assign to agents
- 6. Cloud Master Launch tasks Assign to agents
- 8. Service Growth in millions of tasks per month 0 5 10 15 20 Print Large Small Batch
- 9. Why Akka?
- 12. Parent Config Child Snapshots Changes Message flow: Ensure messages follow a consistent path
- 13. Parent Config Child Snapshots Changes Creation: Assume actor is recovering from failure (state machine)
- 14. Anti-patterns
- 15. God Class
- 16. Movie Star
- 17. Pool Agent State Agent Agent Agent Agent Queu e
- 18. Movie Star Too much state • Hard to reason about • Too many messages in flight • Hard to recover • Bad concurrency
- 19. Split Brain
- 20. Pool Agent State Agent Agent Agent Agent
- 21. Duplicate state Single source of truth • Synchronizing state is hard • Failure causes –State out of sync –Causes more failure
- 22. Split Brain
- 23. Pool Agent State Agent Agent Agent Agent Task
- 24. Passing responsibility Seems simple at first • Do not always know who is in control • Both actors updating the same row • Creates race conditions
- 25. Can you let it crash?
- 26. Pool Agent State Agent Agent Agent Agent Can you let it crash?
- 27. Lessons
- 28. Test for resilience • Chaos Marmoset • Unit test recovery • Destructive system test
- 30. Sovereignty One actor • One row • One shard • One table Otherwise failure hard to handle
- 31. Atomicity Actors Atomic receive method State not shared Comms async messages Not nestable Mutex Atomic scope State is shared Comms via mutable state Nestable (ACID)
- 33. Pool Agent State Agent Agent Agent Agent Atomicity
- 35. Lessons - Atomicity and Consistency - Actor modeling ≠ Object modeling - Test for Resilience not robustness - Refactor Early
Editor's Notes
- #3: Principle Engineer – Workday’s Grid Cloud Master team. – Who is workday
- #4: Finance and Human Capital Management – ERP Vendor – 100% in the cloud – all customers on a single version
- #5: Fiscal 2016 Total Revenue of $1.16 billion, up 48% year over year Over 5000 employees, over 500 employees in Dublin 2016: Best Workplaces in Ireland, Great Place to Work Institute (#2 for large companies) 2016: 10 Best Large Workplaces in Tech, Fortune (#2)
- #6: provide elastic grid – other services Reliable execution of background tasks or Jobs – pdf printing to payrole Cloudmaster - Agents - Schedule and assign to Agents
- #7: 5 pools of agents Different types of task, memory size, execution speed
- #8: 5 data centers Secure Reliable Safe Isolated – fairness Scalable - Efficient
- #9: This talk is about the lessons I learned migrating a multithreaded java server application to Akka. To support this growth we need to move to stateful services -- Why
- #10: Actor model of concurrency: Safer (no deadlocks) Easier to reason about Easier to test Better distribution Easier scalability Then Scala because of akka – key selling point
- #12: Trying to avoid two way relationship (coupling – mutability) Static State should be immutable
- #13: Trying to avoid two way relationship (coupling – mutability) Static State should be immutable
- #14: Trying to avoid two way relationship (coupling – mutability) Static State should be immutable
- #16: Everyone knows about the God class – threading and mutexes make this worse
- #17: Some are big - Marlon Brando – some are small Robert Downey Junior - me Even when small - entourage
- #18: AgentPoolActor - Responsible for – Agent actors – Queue of tasks – and their assignments Decomposed into separate classes and traits - Still one actor with an entourage
- #19: Also drives more bad decisions
- #21: AgentPoolActor and AgentStateActor External DB changes – sending notifications – message loss – recovery Caused by movie star – Thought problem was stream of events were inconsistent – fix that State Inconsistent – failure – production outage
- #22: … Beauty of split brains
- #24: AgentPoolActor takes job from the Queue Assigns it to an Agent Agent might fail and put it back Pool or Agent might own the job - Cannot reliably find the job EG Cancel Job
- #25: Who - When
- #27: PoolActor has decided to assign task to an agent Async message to StateActor – PoolActor must ensure agent not reused – before reply What if reply timesout??? Crash - Can I guarantee consistency – what happens to the job?
- #29: Chaos Marmoset base actor overrides the unhandled method Messages can cause failures or delays
- #30: Horizontal scalability by pushing all state into the database Actors are about data – Actors are Stateful – Impedance Stateless services cannot update the same data as actor
- #31: Autonomy – single responsibility If your actors write to the database
- #34: We want agent assignments to be consistent
- #35: Banking Transactions ACID? No - Suspense Account – Reconciliation – Compensating transactions Must handle failure cases