"Scaling in space and time with Temporal", Andriy Lupa.pdf
0 likes107 views
Design patterns like Event Sourcing and Event Streaming have long become standards for building real-time analytics systems. However, when the system load becomes nonlinear with fast and often unpredictable spikes, it's crucial to respond quickly in order not to lose real-time operating itself. In this talk, I’ll share my experience implementing and using a tool like Temporal.io. We'll explore the evolution of our system for maintaining real-time report generation and discuss how we use Temporal both for short-lived pipelines and long-running background tasks.
"Scaling in space and time with Temporal", Andriy Lupa.pdf
- 4. What is Temporal.io? #Short description based on documentation
- 5. What is Temporal Workflow? Add screen
- 7. What is Day.io? - Collect clock in/out events (punches) from mobile and tablet devices, integrations and web widgets. - For each employee for each shift calculate employees basic metrics for salary payments: worked time, rested time, night shifts time, overtime and many more. - Update aggregated metrics for payroll period report. - Generate files with custom formatting of stored JSON-data.
- 9. Requirements - Bursty Load of events that triggers recalculations:
- 10. Requirements - Deduplication & throttling Previously we had huge queues and all the events were processed one by one although we know there are duplicates.
- 11. Requirements - Persist results on different stages - Optimisations inside calculations pipeline - Scalability - Retry logic - Transactions
- 12. Where is load? - ~2M events per day - Synchronous calculations: - >30 metrics for given period (from - to) - >15 metrics that depends on previous day (from - now): changing a day 1 year ago triggers recalculation of entire year!
- 13. High-level setup - 5 kafka consumers - In-house Temporal cluster on top of Postgres - 2 workers for High Priority workflows - 15 workers for Low Priority
- 14. Workflow, version 1 - Single workflow run for same employee and same period (controlled by Temporal) - Policy: Terminate running if new with same id arrived
- 15. Workflow, version 1 Outputs: - faster than old setup as we started interrupting calculations if same event arrived - Temporal cluster was almost at ~100% CPU every time we had even small load - Workers ~100% CPU - Scaling didn’t help Next steps: - Reduce amount of workflows - Fine tune configurations
- 16. Workflow, version 2 - Single workflow run for same employee (controlled by Temporal) - Notify running workflow about consumed event for same employee - In-memory queue for each workflow with deduplication logic for intersection periods
- 17. Workflow, version 2 Outputs: - Faster for clients again. This time also because of deduplication - Amount of operations on Temporal dropped for ~30% - Temporal cluster was almost at ~100% CPU at peak time - Scaling started working. But x2-2.5 max, not as promised Next step: We cannot reduce amount of workflows anymore. So let’s reduce amount of activities
- 18. Workflow, version 3 - Merged all calculations, persisting and sending events activities into single Outputs: - Faster for clients again due to processing less system micro tasks inside each workflow. - Temporal cluster finally not hitting 100% CPU - Improved scaling from x2-2.5 to x5-7. But still not what we were promised Next step: We can merge last 2 activities but it’s not a game changer. Let’s decrease latency inside cluster by replacing database
- 19. Replacing Postgres with Cassandra Benchmark results for 12K workflows: Postgres based Cassandra based Total time ~45 minutes ~15 minutes Workflows starts, per sec ~8 ~40 History pods latency, P99 ~6ms, bursty ~0.5ms GetTasks requests rate, per sec ~4 ~75
- 20. Other workflows types (convert into describe slides) - Long running workflows to send alerts with delay
- 21. Other workflows types (convert into describe slides) - Background bulk operations processing workflow - Cron jobs - Orchestrating data migrations
- 22. Next steps - Merge our consecutive workflows to use power of signals and avoid duplicated business logic runs - Use Local Activities - Move more cron jobs and business transactions into Temporal - Implement Human-in-the-Loop business processes like onboarding
- 23. Pros: - Retry logic and transactions. Your sagas will look like pure functions. - Durable executions of long-running flows. - Safe Cron jobs, especially long running or iterative. - Triggers and workers can be implemented in different languages - Easy to setup in-house cluster using Kubernetes. - Great UI visualisation tools - Great slack community, documentation and open-source code Lessons learned Temporal is nice tool if you know what you need.
- 24. Lessons learned Temporal is NOT nice tool if you DON’T know what you need. Cons: - Not very performant for short-running workflows with huge amount of activities. - Postgres is enough for flat load. Use Cassandra for spikes - Bugs with Cassandra - CLI Batch operations on workflows are very slow - Expensive to keep cluster if workflows are not running all the time - High entry barrier
- 25. Sources Temporal.io Slack - https://t.mp/slack Official documentation - https://docs.temporal.io Blog - https://temporal.io/blog Samples - https://github.com/temporalio/samples-typescript (also for Go, Python, Java, .Net)
- 26. Q&A