Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache Airflow
Download as PPTX, PDF0 likes127 views
This document discusses automating Apache Cassandra operations using Apache Airflow. It recommends using Airflow to schedule and automate workflows for ETL, data hygiene, import/export, and more. It provides an overview of using Apache Spark jobs within Airflow DAGs to perform tasks like data cleaning, deduplication, and migrations for Cassandra. The document includes demos of using Airflow and Spark with Cassandra on DataStax Astra and discusses considerations for implementing this solution.
![Challenge
Business
Platform
Playbook
Framework
Approach
Technology
Management
Solutions
[Data] Services Catalog
Fully Managed Service
Subscriptions
We offer Professional Services to engineer Solutions and
offer Managed Services to clients where it makes sense, after an
Assessment](https://image.slidesharecdn.com/cassandra-221114191427-59e7f6dc/85/Data-Engineer-s-Lunch-82-Automating-Apache-Cassandra-Operations-with-Apache-Airflow-6-320.jpg)
Data Engineer's Lunch #82: Automating Apache Cassandra Operations with Apache Airflow
- 1. Automating Apache Cassandra Operations with Apache Airflow Go beyond cron jobs to manage ETL, Data Hygiene, Data Import/Export Rahul Xavier Singh Anant Corporation | Cassandra.Link Data Engineer’s Lunch 11/14/2022
- 2. Playbook Design Framework Airflow Spark Approach Airflow/Spark Cassandra ETL Spark in Airflow Bonus: Deleting Data in Cassandra at Scale in Airflow Code/Demos Cassandra Operations with Google Dataproc / Spark in Airflow (Astra) SQL Queries with Presto and Cassandra in Airflow Airflow and Spark Agenda
- 3. We help platform owners reach beyond their potential to serve a global customer base that demands Everything, Now.
- 4. We design with our Playbook, build with our Framework, and manage platforms with our Approach so our clients Think & Grow Big.
- 6. Challenge Business Platform Playbook Framework Approach Technology Management Solutions [Data] Services Catalog Fully Managed Service Subscriptions We offer Professional Services to engineer Solutions and offer Managed Services to clients where it makes sense, after an Assessment
- 7. 7 Business / Platform Dream Enterprise Consciousness : - People - Processes, - Information - Systems Connected / Synchronized. Business has been chasing this dream for a while. As technologies improve, this becomes more accessible. Image Source: Digital Business Technology Platforms, Gartner 2016
- 9. 9 Thinking about Cassandra as a Data Fabric XDCR: Cross datacenter replication is the ultimate data fabric. Resilience, performance, availability, and scale. Made widely available by Cassandra and Couchbase
- 10. 10 Generic Data Platform Operations
- 11. Distributed Realtime Components To create globally distributed and real time platforms, we need to use distributed realtime technologies to build your platform. Here are some. Which ones should you choose?
- 12. 12 How do you choose from the landscape? Lots and lots of components in the Data & AI Landscape. Which ones are the right ones for your business?
- 13. 13 So Many Different “Modern Stacks?” Lots of “reference” architectures available. They tend not to think about the speed layer since they are focusing on analytics. Many don’t mention realtime databases… but we can learn from them.
- 15. Framework Components ● Major Components ○ Persistent Queues ( RAM/BUS) ○ Queue Processing & Compute ( CPU) ○ Persistent Storage (DISK/RAM) ○ Reporting Engine (Display) ○ Orchestration Framework (Motherboard) ○ Scheduler (Operating System) ● Strategies ○ Cloud Native on Google ○ Self-Managed Open Source ○ Self-Managed Commercial Source ○ Managed Commercial Source Customers want options, so we decided to create a Framework that can scale with whatever Infrastructure and Software strategy they want to use.
- 16. 16 Playbook for Modern Open Data Platform Platform Design Evaluate Framework Cloud - Public - Private - Hybrid Data - Data:Object - Data:Stream - Data:Table - Data:Index - Processor:Batch - Processor:Stream DataOps - ETL/ELT/EtLT - Reverse ETL - Orchestration DevOps - Infrastructure as Code - Systems Automation - Application CICD Architecture (Design) - Cloud - Data - DevOps - DataOps Engineering - Configuration - Scripting - Programming Operation - Setup / Deploy - Monitoring/Alerts - Administration User Experience - No-Code/Low Code Apps/Form Builders - Automatic API Generator/Platform - Customer App/API Framework Execute Approach Discovery (Inventory) - People - Process - Information (Objects) - Systems (Apps)
- 17. 17 Framework
- 18. Data Modernization / Automation / Integration In addition to vastly scalable tools, there are also modern innovations that can help teams automate and maximize human capital by making data platform management easier.
- 20. Approach 20
- 21. Apache Airflow + Apache Spark + Spark Python/Scala/Java/R + Airflow Python DAG = DataOps for Apache Cassandra Good enough for rock and roll.
- 22. ● Scheduling and automating workflows and tasks ● Automating repeated processes ○ Common ETL tasks ○ Machine learning model training ○ Data hygiene ○ Delta migrations ● Write workflows in Python ○ Anything Python compatible works ○ Dependencies for workflow sections ○ Workflows are a DAG of tasks ● Recurring, One-time Scheduled or Adhoc ○ Cron-like syntax or frequency tags ○ “Only run again if data changed” ● Monitor tasks and collect/view logs Apache Airflow
- 23. Apache Spark ● Unified analytics engine ● High performance batch and streaming data ● Also has a DAG, scheduler, a query optimizer, and a physical execution engine. ● Offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python, R, and SQL shells. C# also available. ● Powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. ● You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, on Mesos, or on Kubernetes. Access data in basically anything.
- 24. Bonus Round
- 25. 25 Coldish ● S3 ● HDFS ● ADLS ● GFS Warm ● Hive / * ● Data Warehouse ● Data Lakehouse Big Data Options Hot ● Cassandra* ● Datastax* ● Scylla* ● Yugabyte* ● Mongo ● REDIS ● … Hot* ● Astra* ● Scylla Cloud* ● YugaByte Cloud* ● Azure CosmosDB* ● AWS Keyspaces* ● AWS Dynamo ● Google BigTable ● … * PSSST. These all use CQL!!!
- 26. 26 Cleaning Big Data: Same $h1t Different Day Data Cleaning as part of Data Engineering - Step 1: Remove duplicate or irrelevant observations - Step 2: Fix structural errors - Step 3: Filter unwanted outliers - Step 4: Handle missing data - Step 5: Validate and QA https://www.tableau.com/learn/articl es/what-is-data-cleaning Data Cleaning after the Fact - Enforce a custom data retention policy (TTL) - Enforce GDPR / Right to be Forgotten - Move application, customer, user from one system to another - Remove x “versions” or “revisions” of data - Remove test data from a stress test
- 27. 27 Cleaning Big Data: In SQL …. Data Cleaning in SQL - Find what do you want to delete. - Delete it.
- 28. 28 Cleaning Big Data: In Spark SQL …. Data Cleaning in SPARK SQL - What do you want to delete. - Delete it. WARN: Doesn’t work with all data in Spark SQL. Only if connector supports Table Delete https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-delete-from.html https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/connector/catalog/SupportsDelete.html
- 29. 29 Cleaning Big Data: Cleaning data in Spark / SQL Data Cleaning in Spark for Cassandra - What do you want to delete. - Delete it. https://stackoverflow.com/questions/28563809/delete-from-cassandra-table-in-spark
- 30. 30 Cleaning Big Data: Deduping in Spark SQL Deduping Data in Spark for Cassandra - What do you want to dedupe. - Do some deduping. - What you want to delete. - Delete it.
- 31. 31 Airflow DAG to Migrate Cassandra Data Airflow can help us take any data process, compiled, interpreted etc, coordinate the steps as a DAG (“Directed Acyclic Graph”) and then to make it even more awesome, parametrize it either via the Airflow GUI or our own table somewhere. github/scylladb/scylla-migrator
- 32. 32 Airflow DAG to Clean Cassandra Data Since we write abstracted code, we can replace the “Migrator” process with a Delete, Dedupe, Validate. Whatever. Airflow allows us to reuse conventions we set in a team for large scale operations, and most importantly… make it easy for people to run Data Operations like this without being Cassandra , Spark, Python experts.
- 33. Demo ● https://github.com/Anant/example-cassandra-etl-with-airflow-and-spark ● Astra: Set up Astra Account / Database / Keyspace / Access ● Gitpod: Set up Airflow and Spark ● Airflow: Connect Airflow and Spark ● Trigger DAG with PySpark jobs via Airflow UI ● Confirm data in Astra
- 34. Other Demos with Airflow ● https://github.com/anant?q=airflow ● Most have videos / blogs ○ See “Cassandra.Lunch” Repo ○ See anant.us/blog ● Airflow + Google Dataproc + Astra ● Airflow + DBT + Great Expectations ● Airflow + Cassandra + Presto ● Airflow + Cassandra ● Airflow + Spark ● Airflow + Amundsen + Cassandra (DSE)
- 35. 35 Considerations for Spark/Airflow Solution Considerations for Airflow - Figure out if you are going to manage it / run it. - Figure out for whom you are going to run it (Platform, Environment, Stack, App, Customer?) - Not all DAGs just work. Sometimes they need tweaking across Environments, Stacks, Apps, Customers. - The same DAG may fail over time. Need to watch execution times. - Who has access to it? Considerations for Spark - Figure out if you want to manage it / run it. - Not all Spark code is created equally. - Not all Spark languages run the same. - Compiled Jobs with input parameters can work better in the long run, less room for code drift. - Don’t let people do adhoc delete operations until and unless it’s absolutely necessary. - Who has access to it?
- 36. 36 Key Takeaways for Cassandra Data Operations - You can look, but Apache Spark is basically it. Look no further. - Learn Spark, Python / Scala is fine. Just start using Apache Spark. - Airflow, Jenkins, Luigi, Prefect, any scheduler can work, but Airflow has been proven for this. - Airflow works with more than just Apache Cassandra, Apache Spark.. There are numerous Connections and Operators. Don’t reinvent the wheel. Use Apache Spark Use a Scheduler ( Apache Airflow w/ Python )
- 37. 37 Thank you and Dream Big. Hire us - Design Workshops - Innovation Sprints - Service Catalog Anant.us - Read our Playbook - Join our Mailing List - Read up on Data Platforms - Watch our Videos - Download Examples www.anant.us | [email protected] | (855) 262-6826 3 Washington Circle, NW | #301 | Washington, DC 20037
Editor's Notes
- #8: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #10: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #11: Challenge Currently the components are broken up in to different vendors and parts. Similar to building a computer every time for every client.
- #13: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #14: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #16: Challenge Currently the components are broken up in to different vendors and parts. Similar to building a computer every time for every client.
- #17: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #26: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #27: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #28: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #29: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #30: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #31: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #32: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #33: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #36: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #37: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.
- #38: What makes a good story? Once you get good at it, presenting becomes easy. Shared stories with people we’ve bonded with (community for example). This format is not good for Metastories.