Asynchronous Task Queues with Celery
Download as PPTX, PDF0 likes658 views
Celery is an asynchronous task queue/job queue based on distributed message passing. It allows tasks to be executed asynchronously (in the background) outside of the main request-response cycle. Some key uses of Celery include running long-running or expensive queries, interacting with external APIs, and scheduling periodic or recurring tasks as an alternative to cron jobs. Celery uses message brokers like RabbitMQ or Redis to ensure tasks are only executed once and are distributed across worker servers. Tasks can be defined as Python functions that are decorated to make them visible to the Celery system. Periodic and recurring tasks can also be defined for scheduling purposes.
Asynchronous Task Queues with Celery
- 1. Kishor Kumar Mahato Python Lead at Eagle Vision IT Python User Group Nepal Meetup #7 2nd August 2015 Celery Asynchronous Task Queues
- 3. Distributed Asynchronous Task Queue For Python It is a task queue which focus on real-time processing It also supports task scheduling
- 4. • Anything that needs to run asynchronously, eg. outside of the request-response cycle. • Background computation of ‘expensive queries’ • Interactions with external API’s (eg. Twitter) • Periodic tasks (instead of cron & scripts) • Long-running actions with results displayed via AJAX. Potential Uses
- 6. Work Flow
- 7. Celery Workers Run at least 1 celery worker server ( Redis / RabbitMQ) Can be on different machines Celery guarantees that tasks are only executed once
- 8. Task from celery.task import Task from celery.registry import tasks class FetchUserInfoTask(Task): def run(self, screen_name, **kwargs): logger = self.get_logger(**kwargs) try: user = twitter.users.show(id=screen_name) logger.debug("Successfully fetched {0}".format(screen_name)) except TwitterError: logger.error("Unable to fetch {0}: {1}".format( screen_name, TwitterError)) raise return user tasks.register(FetchUserInfoTask)
- 9. >>> from myapp.tasks import FetchUserInfoTask >>> result = FetchUserInfoTask.delay(“kishor_14”) Run It!
- 10. Periodic Tasks from celery.task import PeriodicTask from celery.registry import tasks from datetime import timedelta class FetchMentionsTask(Task): run_every = timedelta(seconds=60) def run(self, **kwargs): logger = self.get_logger(**kwargs) mentions = twitter.statuses.mentions() for m in mentions: ProcessMentionTask.delay(m) return len(mentions) tasks.register(FetchMentionsTask)
- 11. Periodic Tasks from datetime import timedelta from celery.schedules import crontab CELERYBEAT_SCHEDULE = { 'order-reminder-every-1-hours': { 'task’: 'application.views.cron_job_for_application_processing', 'schedule': crontab(minute=0, hour='*/1') # 'schedule': timedelta(seconds=30) }, } CELERY_TIMEZONE = 'UTC'
- 14. Thank You! Kishor Kumar Mahato [email protected] @kishor_14 fb:kishrorkumar