Developer-friendly task queues: what we learned building MRQ, Sylvain Zimmer
1 like397 views
This document summarizes key considerations for choosing a task queue system. It discusses task properties like idempotency and reentrancy. It covers performance factors like latency and throughput as well as consistency models. Common task queue systems like Celery, RQ, and MRQ are evaluated based on factors like performance, complexity, community support, and future plans. The document emphasizes thinking through requirements for task visibility, worker reliability, language interoperability and choosing a queue that appropriately balances these needs.
![A typical job/task
def send_an_email(email_type, user):
html = template(email_type, user)
status = email.send(html, user["email"])
metrics.send("email_%s" % status, 1)
return status
KERNEL PANIC](https://image.slidesharecdn.com/6-170615115227/85/Developer-friendly-task-queues-what-we-learned-building-MRQ-Sylvain-Zimmer-6-320.jpg)
Developer-friendly task queues: what we learned building MRQ, Sylvain Zimmer
- 1. Sylvain Zimmer / @sylvinus PyParis 2017 DEVELOPER-FRIENDLY TASKQUEUES WHAT WE LEARNED BUILDING MRQ & WHAT YOU SHOULD ASK YOURSELF BEFORE CHOOSING ONE
- 2. /usr/bin/whoami ▸ (SpaceX nerd) ▸ Founder, dotConferences ▸ CTO Pricing Assistant ▸ Co-organizer Paris.py meetup ▸ User of Python taskqueues for 10+ years ▸ Main contributor of MRQ
- 3. 4 years ago... OMG !1!!!
- 5. Credit: Adrien Di Pasquale
- 6. A typical job/task def send_an_email(email_type, user): html = template(email_type, user) status = email.send(html, user["email"]) metrics.send("email_%s" % status, 1) return status KERNEL PANIC
- 7. Task properties Re-entrant Idempotent Nullipotent< < ▸ Safe to interrupt and then retry ▸ Safe to call multiple times ▸ Result will be the same ▸ Free of side-effects def reentrant(a): value = a + random() db.insert(value) def idempotent(key, value): db.update(key, value) def nullipotent(a): return a ** 2
- 8. Other task properties & best practices ▸ Serializable args, serializable result ▸ Args validation / documentation ▸ Least args possible ▸ Canonical path vs. registration ▸ Concurrent safety ▸ Statuses
- 9. Coroutines vs. Threads vs. Processes ▸ IO-bound tasks vs. CPU-bound tasks ▸ Threads offer few benefits for a Python worker (GIL) ▸ Coroutines/Greenlets are ideal for IO-bound tasks ▸ Processes are required for CPU-bound tasks ▸ If you have heterogenous tasks, your TQ should support both! $ mrq-worker --greenlets 25 --processes 4
- 10. Performance: latency & throughput APP BROKER WORKER RESULT STORE
- 11. OPS & TOOLING
- 13. Errors ▸ Exception handlers ▸ Timeouts ▸ Retry rules ▸ Sentry & friends ▸ gevent: test your tracebacks! ▸ Priorities ▸ Human process to manage failed tasks!
- 14. Task visibility ▸ Tasks by status, path, worker, ... ▸ Tracebacks & current stack ▸ Logs ▸ Timing info ▸ Cancel / Kill / Move tasks ▸ Progress
- 15. Memory leaks ▸ Workers = long-running processes ▸ gevent makes debugging harder ▸ Watch out for global variables or mutable class attributes! ▸ Python's ecosystem is surprisingly poor in this area ▸ guppy, objgraph can usually help
- 16. Misc tools ▸ Scheduler ▸ Command-line runner, e.g. mrq-run tasks.myTask {"a": 1} ▸ Autoscaling ▸ Profiler
- 17. CONSISTENCY
- 18. Consistency guarantees ▸ At least once vs. At most once vs. Exactly once ▸ Ordering ▸ Critical operations: ▸ Queueing ▸ Marking tasks as started ▸ Timeouts & retries
- 19. Types of brokers ▸ Specialized message queues (RabbitMQ, SNS, Kafka, ...) ▸ Performance, complexity, poor visibility ▸ In-memory data stores (Redis, ...) ▸ Performance, simplicity, harder to scale ▸ Regular databases (MongoDB, PostgreSQL, ...) ▸ Often enough for the job!
- 20. At the heart of the broker ▸ Atomic update from "queued" to "started" ▸ MRQ with MongoDB broker: find_one_and_update() ▸ MRQ with Redis broker: Pushback in a ZSET
- 21. ZSETs in Redis ▸ Sorted sets with O(log(N)) scalability ▸ set/get by key, order by key, lookups by key or value ▸ Very interesting properties for task queues: Unicity, Ordering, Atomicity of updates, Performance ▸ MRQ's "Pushback" model: ▸ Queue with key=timestamp ▸ Unqueue by fetching key range & setting new keys in the future ▸ After completion the task adjusts or removes the key
- 23. Consistency guarantees ▸ Must be thought of for the whole system, not just the broker! ▸ Brokers can be misused or misconfigured ▸ The workers can drop tasks if they want to ;-) ▸ Consistency starts at queueing time!
- 24. TIME TO CHOOSE!
- 25. Think hard about what you need ▸ Will your taskqueue be the foundation of your architecture, or is it just a side project? ▸ What performance do you need? (IO vs. CPU, latency, ...) ▸ What level of visibility and control do you need on queued & running tasks? ▸ Can workers terminate abruptly? Lots of design consequences! ▸ What language interop do you need?
- 26. And then all the usual questions... ▸ Is it supported by a lively community? ▸ License ▸ Documentation ▸ Future plans
- 27. Which one to pick? ▸ Celery: High performance, large community, very complex, major upgrades painful ▸ RQ: Extremely simple to understand, low performance ▸ MRQ: Adjust task visibility vs. performance, simple to understand, 1.0 soon ▸ Lots of other valid options! Just be sure to ask yourself the right questions ;-)
- 28. BE GRATEFUL FOR THE OSS YOU USE! REMINDER