[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf

NFTBank x Snowflake Tech Seminar
Building modern data pipeline with
Snowflake + DBT + Airflow

Index
Session 1: Data Quality & Productivity
- Data Quality
- Data Quality Validation
- Data Catalog, Lineage Documentation
- DBT Introduction
Session 2: Integrate DBT with Airflow
- DBT Cloud or Airflow?
- Astronomer Cosmos
- dbt deps
Session 3: Cost Optimization
- Query Optimization
- Cost Monitoring

Chris Hoyean Song
NFTBank VP of AIOps 2021.11 ~
Riiid VP of AIOps 2020.09-2021.11
Naver AI Engineer 2017.12-2020.05
Kakao Data Engineer 2015.08-2017.12
Startup CEO 2013-2015
KAIST Innovation & Tech Management Master
KAIST Computer Science Bachelor
Commercialization of AI Tech
& Scalable AI
I am in charge of connecting artificial intelligence technology
with business.
The area of expertise is ML Pipeline, which increases the
productivity of artificial intelligence projects.
"The biggest lesson that can be read from 70 years of AI research is that
general methods that leverage computation are ultimately the most
effective, and by a large margin. "
-Richard Sutton
https://www.linkedin.com/in/chris-song-0bb03439/

onchain(blockchain)
data
off-chain data
feature vector Estimated price
models for NFTs
Processed
data
API
customers
Data Value Chain
MLOps

Data Quality & Productivity
Session 1

Data Quality: The Rule of Ten
Image source: https://www.barlog.de/en/services/engineering-cae/
The Rule of Ten for defect costs shows
how important cost savings are.
This defines that the cost of an
undiscovered defect increases tenfold per
value generation step.

1-1. Data Quality Validation
Data Quality
- Data Freshness Monitoring
- Data is not loading
- Data Unique Test
- Business logic twisted due to redundant data
- Data Count Test
- Query is executed, but the data is empty
- Data Min Max Test
- Invalid data
- Schema Validation
- Partner API response results suddenly change
- If this is not Schema validated, the pipeline failure is not easily detected

1-2. Data Catalog, Lineage Documentation
- Data Catalog
- Where is the data and what column do you have?
- Who's in charge?
- Who modified it at the end?
- Data Lineage
- What data did you make this data from?
- What table is this data being used to create?
- Isn't there a circular reference in creating data?
- I want to see the whole big picture of creating our core data.

Trial #1. Datahub
Datahub is one of the best data documentation tools today
Problem: Datahub documentation cleanup does not become a priority due to a backlog of tasks

Trial #2. Airflow DAG DVL
특징:
- Manually Implemented Airflow Module
단점:
- Too many DVL files are difficult to manage in a
unified manner

Too many Test Queries
Too many test codes,
and if it's not managed, it turns into legacy

Trial #3. PyDantic
Features:
- Suitable for schema validation
- Serialize integration
- FastAPI Default Packages
Cons:
- Slow: Spend a lot of time serializing
- If you optimize the response speed, you remove it
- Memory: twice as much memory as
collections.namedTuple
- If you optimize memory, you can remove it

Trial #4. Datadog SLO Monitoring
- Periodically sending custom metrics to Datalog
- Used primarily to monitor real-time data presentation
Features
- Suitable for real-time monitoring
- SLO, useful for SLA management

Trial #5. DBT
Features:
- Data Catalog and Data Lineage are automatically defined when SQL is defined according to DBT
conventions only
- Add one line to the yaml file to create a Data Validation Test
- You can see what tests are on each model and see if they pass
- Easily work with all major data solutions: Snowflake, BigQuery, Databricks, Postgresql
Cons:
- Airflow and integration are not seamless: need to care a lot about integration
- DBT Cloud has limitations because it cannot work with other Airflow DAGs
- Not many references yet

DBT: Data pipeline with just SQL files

$ dbt test

$ dbt docs serve

DBT 도입 소감

Integrate DBT with Airflow
Session 2

2-1. How Can we trigger DBT models?: Airflow
Cloud
or

2-2. How can we build task dependency?: Astronomer Cosmos
https://github.com/astronomer/astronomer-cosmos
Features:
- Library to run DBT in Airflow
- Implement the lineage of the DBT Model
- After running dbt run, check quality with
connected dbt test
- Reduce time to implement Airflow DAG

2-3. How can we sync dbt files to Airflow?: dbt deps

Data Warehouse Cost Optimization
Session 3

Data Warehouse Cost Optimization
Data Warehouse
Cost Optimization
Query Optimization Cost Monitoring

3-1. Query Optimization
The following 4 conditions must be met to optimize the query
1. Partitioning: Must be partitioned based on frequently used query conditions
2. File Size: File size in a partition should be as small as 50-512MB in size suitable for large queries for
better performance
3. Clustering: clustering, optimizing query performance by aligning data
4. Query Tuning: Reduce the scope of data scanning.
- Remove unnecessary scans
- Eliminating inefficient JOIN
- Remove CROSS JOIN that is not essential
- Only scan the MERGE statement you need
- Specify only the necessary scan ranges for the dbt test

Snowflake vs BigQuery: Characteristics
Snowflake BigQuery
Partitioning Micro Partition Explicit Partition
Cost Data Delay Near Real-time (custom dashboard)
1-2 Days (built-in dashboard)
2-3 Days
(GCP cost analysis)
Cost Analysis Per Warehouse Size, User, Tag Per Project, Product
(BigQuery Computing,
Storage)

Snowflake Micro Partition
- Normally, it is common to set the partitions explicitly
- Snowflake prevents users from touching it at all, and takes the task of partitioning to managed

3-2. Cost Monitoring: Realtime Snowflake Cost Monitoring
Our Senior Data Engineer, Jensen visualized the
real-time cost dashboards from with automated
tools
This allows you to immediately identify where your
costs go
You can query query usage history directly with
Snowflake queries to make cost estimates in real
time
- Warehouse Type
- Time (Minutes or Hour)
- Credit / Warehouse * Hour
- Cost / Credit

3-2. Cost Monitoring: Daily cost review
- Data engineers gather to check cost monitoring and check issues before daily stand-up starts
- If I find a problem, I fix it right away
- A disability retrospective should be conducted for significant cost impairments

3-2. Cost Monitoring: Predicting and Retrospecting Costs
- Estimate how much money will be incurred before running a large backfill,
and be confirmed to spend by the leader
- Contrast and recall estimated and actual expenditure after performing backfill
- The more records are accumulated, the more accurate the estimation becomes,
the faster you can recognize and tune inefficient queries

Thanks to Jensen
Senior Data Engineer
https://www.linkedin.com/in/jensen-yap-8020b535/

Join NFTBank! 🚀
https://nftbank.breezy.hr/
35
NFTBank Medium NFTBank 지원 NFTBank YouTube

Thank you

[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf

More Related Content

What's hot (20)

Similar to [EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf (20)

Recently uploaded (20)

[EN] Building modern data pipeline with Snowflake + DBT + Airflow.pdf