Task queuing with redis and rq
- 1. Task queuing with Redis and rq Nadja Deininger
- 2. Who am I? Python developer Software engineer at Beautifeye @machine_lady
- 3. Why use queues? - Execute long tasks asynchronously - Parallelise work - Distribute work
- 4. Queues in Python - Celery - RabbitMQ - Surely, many more… - rq
- 5. Redis redis.io Key-value store Supports lists (and other data structures) Can be used to host queues
- 6. rq python-rq.org Needs a running Redis server - and that’s it!
- 7. Getting started import rq from redis import Redis from my_module import my_function with rq.Connection(Redis()): q = rq.Queue("my_queue") job = q.enqueue(my_function, args=("arg1", "arg2"))
- 8. Performing work rq comes with a default worker script: $ rqworker queue1 queue2 Picks up jobs from the first nonempty queue.
- 9. Customise worker behaviour Write your own worker class to e.g. - manage a database connection - avoid using os.fork and use it by specifying $ rqworker -w ‘path.to.MyWorker’
- 10. Get job results Return value of enqueued function (if any) can be accessed through job.result Results are stored under job’s Redis hash - default TTL: 500s - job.result will be None until task is done
- 11. Dealing with failure By default: put a job onto the failed queue; can be accessed through rq.FailedQueue Failed jobs don’t expire. They can be retried (usually manually).
- 12. Dealing with failure differently Supply your own exception handler to a Worker instance: worker = rq.Worker(queues, exc_handler=my_exception_handler)
- 13. Monitoring rq - command line
- 14. rq-dashboard - web monitoring tool
- 15. rq-dashboard - web monitoring tool
- 16. Lessons learned - Module imports can impact performance - Don’t let queues get too large - Functions cannot depend on context