Airflow presentation
4 likes•5,368 views
Airflow is a workflow management system for authoring, scheduling and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It has features like DAGs to define tasks and their relationships, operators to describe tasks, sensors to monitor external systems, hooks to connect to external APIs and databases, and a user interface for visualizing pipelines and monitoring runs. Airflow uses a variety of executors like SequentialExecutor, CeleryExecutor and MesosExecutor to run tasks on schedulers like Celery or Kubernetes. It provides security features like authentication, authorization and impersonation to manage access.
![CLI
https://airflow.apache.org/cli.html
airflow variables [-h] [-s KEY VAL] [-g KEY] [-j] [-d VAL] [-i FILEPATH] [-e FILEPATH] [-x KEY]
airflow connections [-h] [-l] [-a] [-d] [--conn_id CONN_ID]
[--conn_uri CONN_URI] [--conn_extra CONN_EXTRA]
[--conn_type CONN_TYPE] [--conn_host CONN_HOST]
[--conn_login CONN_LOGIN] [--conn_password CONN_PASSWORD]
[--conn_schema CONN_SCHEMA] [--conn_port CONN_PORT]
airflow pause [-h] [-sd SUBDIR] dag_id
airflow test [-h] [-sd SUBDIR] [-dr] [-tp TASK_PARAMS] dag_id task_id execution_date
airflow backfill dag_id task_id -s START_DATE -e END_DATE
airflow clear DAG_ID
airflow resetdb [-h] [-y]
...](https://image.slidesharecdn.com/airflowpresentation1-180717183432/85/Airflow-presentation-47-320.jpg)
Airflow presentation
- 1. Airflow Workflow management system Ilias OKACHA
- 2. Index - Workflows Management Systems - Architecture - Building blocks - More features - User Interface - Security - CLI - Demo
- 3. WTH is a Workflow Management System ? A worflow Management system is: Is a data-centric software (framework) for : - Setting up - Performing - Monitoring of a defined sequence of processes and tasks
- 4. Popular Workflow Management Systems
- 6. Airflow architecture SequentialExecutor / LocalExecutor
- 8. Airflow architecture HA + CeleryExector
- 9. Airflow architecture ● MesosExecutor : already available in contrib package ● KubernetesExecutor ??
- 10. Building blocks
- 11. Dags : - Directed Acyclic Graph - Is a collection of all the tasks you want to run - DAGs describe how to run a workflow Building blocks
- 13. Building blocks Operators : - Describes a single task in a workflow. - Determine what actually gets done - Operators generally run independently (atomic) - The DAG make sure that operators run in the correct certain order - They may run on completely different machines
- 14. Building blocks Operators : There are 3 main types of operators: ● Operators that performs an action, or tell another system to perform an action ● Transfer operators move data from one system to another ● Sensors are a certain type of operator that will keep running until a certain criterion is met. ○ Examples include a specific file landing in HDFS or S3. ○ A partition appearing in Hive. ○ A specific time of the day.
- 15. Operators : - Operators : - BashOperator - PythonOperator - EmailOperator - HTTPOperator - MySqlOperator - SqliteOperator - PostgresOperator - MsSqlOperator - OracleOperator - JdbcOperator - DockerOperator - HiveOperator - SlackOperator Building blocks
- 16. Operators : - Transfers : - S3FileTransferOperator - PrestoToMysqlOperator - MySqlToHiveTransfer - S3ToHiveTransfer - BigQueryToCloudStorageOperator - GenericTransfer - HiveToDruidTransfer - HiveToMySqlTransfer Building blocks
- 17. Operators : - Sensors : - ExternalTaskSensor - HdfsSensor - HttpSensor - MetastorePartitionSensor - HivePartitionSensor - S3KeySensor - S3PrefixSensor - SqlSensor - TimeDeltaSensor - TimeSensor - WebHdfsSensor Building blocks
- 19. Tasks : a parameterized instance of an operator Building blocks
- 20. Building blocks Task Instance : Dag + Task + point in time - Specific run of a Task - A task assigned to a DAG - Has State associated with a specific run of the DAG - States : it could be - running - success, - failed - skipped - up for retry - …
- 21. Building blocks Workflows : ● DAG: a description of the order in which work should take place ● Operator: a class that acts as a template for carrying out some work ● Task: a parameterized instance of an operator ● Task Instance: a task that ○ Has been assigned to a DAG ○ Has a state associated with a specific run of the DAG ● By combining DAGs and Operators to create TaskInstances, you can build complex workflows.
- 22. Building blocks
- 23. More features
- 24. - Features : - Hooks - Connections - Variables - XComs - SLA - Pools - Queues - Trigger Rules - Branchings - SubDags More features
- 25. Hooks : - Interface to external platforms and databases : - Hive - S3 - MySQL - PostgreSQL - HDFS - Hive - Pig - … - Act as building block for Operators - Use Connection to retrieve authentication informations - Keep authentication infos out of pipelines. More features
- 26. Connections : Connection informations to external systems are stored in the airflow metadata Database and managed in the UI More features
- 27. More features
- 28. Exemple de Hook + connection : More features
- 29. More features Variables : - A generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow. - Variables can be listed, created, updated and deleted from the UI (Admin -> Variables), code or CLI. - While your pipeline code definition and most of your constants and variables should be defined in code and stored in source control, it can be useful to have some variables or configuration items accessible and modifiable through the UI.
- 30. More features
- 31. XCom or Cross-communication: ● Let tasks exchange messages allowing shared state. ● Defined by a key, value, and timestamp. ● Also track attributes like the task/DAG that created the XCom and when it should become visible. ● Any object that can be pickled can be used as an XCom value. XComs can be : ● Pushed (sent) : ○ Calling xcom_push() ○ If a task return a value (from its operator execute() method) or from a PythonOperator’s python_callable ● Pulled (received) : calling xcom_pull() More features
- 32. More features SLA : - Service Level Agreements, or time by which a task or DAG should have succeeded, - Can be set at a task level as a timedelta. - An alert email is sent detailing the list of tasks that missed their SLA.
- 33. More features Pools : - Some systems can get overwhelmed when too many processes hit them at the same time. - Limit the execution parallelism on arbitrary sets of tasks.
- 35. Queues : (only on CeleryExecutors) : - Every Task can be assigned a specific queue name - By default, both worker and tasks are assigned with the default_queue queue - Workers can be assigned multiple queues - Very useful feature when specialized workers are needed (GPU, Spark…) More features
- 36. More features Trigger Rules: Though the normal workflow behavior is to trigger tasks when all their directly upstream tasks have succeeded, Airflow allows for more complex dependency settings. All operators have a trigger_rule argument which defines the rule by which the generated task get triggered. The default value for trigger_rule is all_success and can be defined as “trigger this task when all directly upstream tasks have succeeded”. All other rules described here are based on direct parent tasks and are values that can be passed to any operator while creating tasks: ● all_success: (default) all parents have succeeded ● all_failed: all parents are in a failed or upstream_failed state ● all_done: all parents are done with their execution ● one_failed: fires as soon as at least one parent has failed, it does not wait for all parents to be done • one_success: fires as soon as at least one parent succeeds, it does not wait for all parents to be done • dummy: dependencies are just for show, trigger at will.
- 37. User Interface
- 38. User Interface Dags view :
- 39. User Interface Tree view :
- 43. User Interface Data Profiling : SQL Queries
- 44. User Interface Data Profiling : Charts
- 45. User Interface Data Profiling : Charts
- 46. CLI
- 47. CLI https://airflow.apache.org/cli.html airflow variables [-h] [-s KEY VAL] [-g KEY] [-j] [-d VAL] [-i FILEPATH] [-e FILEPATH] [-x KEY] airflow connections [-h] [-l] [-a] [-d] [--conn_id CONN_ID] [--conn_uri CONN_URI] [--conn_extra CONN_EXTRA] [--conn_type CONN_TYPE] [--conn_host CONN_HOST] [--conn_login CONN_LOGIN] [--conn_password CONN_PASSWORD] [--conn_schema CONN_SCHEMA] [--conn_port CONN_PORT] airflow pause [-h] [-sd SUBDIR] dag_id airflow test [-h] [-sd SUBDIR] [-dr] [-tp TASK_PARAMS] dag_id task_id execution_date airflow backfill dag_id task_id -s START_DATE -e END_DATE airflow clear DAG_ID airflow resetdb [-h] [-y] ...
- 48. Security
- 49. Security By default : all access are open Support ; ● Web authentication with : ○ Password ○ LDAP ○ Custom auth ○ Kerberos ○ OAuth ■ Github Entreprise Authentication ■ Google Authentication ● Impersonation (run as other $USER) ● Secure access via SSL
- 50. Demo
- 51. Demo 1. Facebook Ads insights data pipeline. 2. Run a pyspark script on a ephemeral dataproc cluster only when s3 data input is available 3. Useless workflow : Hook + Connection + Operators + Sensors + XCom +( SLA ): ○ List s3 files (hooks) ○ Share state with the next task (xcom) ○ Write content to s3 (hooks) ○ Resume the workflow when an S3 DONE.FLAG file is ready (sensor)
- 53. Thanks