Building a Marketing Data Warehouse from Scratch - SMX Advanced 202
1 like•621 views
The document outlines a strategy for building a marketing data warehouse from scratch, detailing its importance, phases, and technical components including data ingestion, visualization, and operational analytics. It emphasizes the need for a clear data ecosystem and provides practical insights on leveraging Google Cloud and BigQuery for effective data management. Key takeaways include understanding the benefits of data warehouses, designing analytics questions, and avoiding common pitfalls in analytics practices.
Building a Marketing Data Warehouse from Scratch - SMX Advanced 202
- 1. Building a Marketing Data Warehouse from Scratch Christopher Gutknecht | @chrisgutknecht | Bergzeit
- 2. 2. Dashboards 1. Greenfield Our Plan: The Three Phases of a Data Platform 3. Operational Analytics
- 3. What You Will Take Away from this Session 1. When and why you should invest in a Marketing DWH 3. Interesting use cases by combining data sources 5. Design outcome-oriented questions for analytics projects 2. Learn the data ecosystem and the benefits of BigQuery 4. Many tactical tips for daily use
- 4. About Chris: Head of Acquisition & Optimization Digital Marketer Tech nerd Climber 1997 2008 2013 2020 Dad of 2 Big Thanks to Steffi, our Data Scientist
- 5. Bergzeit: Combining Love for Mountains & Data Online Store for Mountain Gear 122 M Revenue in Financial Year 20/21 14 Countries & 5 Languages Commerce. Content. Guided Tours
- 6. Let’s Set Clear Expectations for this Session Technical deep dive Intro to data ecosystem What this session IS What it’s NOT Machine learning Customer data focused Google Cloud focused Practical tips & mistakes Ecommerce use cases
- 7. Why Is Data Knowledge Important for You? Behavioural data -> digital success Operational analytics on the rise Requirement Privacy by design
- 8. Most Desirable Digital Marketing Skills 20/21 source: marketingcharts.com
- 9. The Components of a Modern Data Platform
- 10. Not so fast… Why All This Complexity?
- 11. Why Do I Need a Marketing Data Warehouse? Connectors
- 12. If your Frustration Grows with Reporting Tools Transformations Volume Operational Use Cases Complexity More Data Sources
- 13. Don’t Be this Guy - Know When To Scale Up
- 14. But Don’t Overengineer - Gradually Scale
- 15. Infrastructure Maintenance Long upfront planning Past: The Old On-Premise Data Veterans Manual Scaling Strict Dimensional Modeling
- 16. Today: The Cool Cloud Kids on the Block 100% cloud, no Ops Seamless scaling Instantly ready
- 17. The Components of a Modern Data Platform Data Warehouse Data Ingestion Data Catalog & Governance Activation Data Quality Job Orchestration Visualization Transformation
- 18. Analysts are Turning into Analytics Engineers Data Warehouse Data Ingestion Data Catalog & Governance Activation Data Quality Job Orchestration Visualization Transformation
- 20. Let’s Start from The Beginning: Data Sources Data Ingestion Google Ads Data Transfer GA4 Export Google Merchant Center Paid Connectors, e.g. Fivetran Custom Ingestion Scripts Google Sheets Cost Connectors, e.g. Funnel.io
- 21. 1. Navigate to Data transfers Set up a Big Query Ads Transfer in One Minute 2. Configure the Transfer details
- 22. 2. Storage & Transfer Get Your GA4 Data into BQ in Two Minutes 1. BQ Linking in Admin UI
- 23. Easily Connect Your Sheets as BigQuery Tables
- 24. Data Loader Vendors: Easy Setup, Instant Results
- 25. Suggestions for Interesting Data Sources Domain Data Source Available in Data Loaders? SEO Google Search Console SEO Pagespeed Insights & Lighthouse SEO Google Bot Logfiles Ecom Inventory Data & Attributes Ecom Trusted Shops Reviews Ecom Awin Open Orders Social Instagram Social Facebook .... ...
- 26. 2. Storage & Transfer The Best Custom Way to Ingest Data into BQ Data Source 3. BigQuery 1. Cloud Function Data Transfer handles ingest job Observability via alerts Data Fetch with Python and Pandas
- 27. How Do We Batch-Ingest New Data? Change Data Capture Snapshots Copies Full History Daily State very easy For <300k rows rather easy duplicate rows storage efficient complex architecture
- 29. We’ve Got Data: Show Me Shiny Reports! Data Warehouse Data Ingestion Visualization
- 30. How do Price Discounts affect Sales? Sources:
- 31. Price Discounts Data Model Sources: date sku detail_views product order value ga sessions date sku price sale_price diff_price_to_sale diff_price_to_sale_grouped products
- 32. How do Season Categories Affect Sales?
- 33. Season Category Data Model Sources: date sku detail_views product order value ga sessions date sku season_category products
- 34. Category Revenue Share without Ratings
- 35. Category Rev & Ratings Data Model Sources: date sku top_category product order value ga sessions date sku rating_count rating_count_grouped products ratings
- 36. Which Category Has More Selection Orders?
- 37. Selection Orders Data Model Sources: date transaction_id sub_category is_multi_same_category is_multi_same_color is_multi_same_size ga sessions
- 38. Wait… Don’t You Need to Know SQL for This?
- 39. You Need to Learn SQL: Take a BigQuery Course Self-paced 45€ / User / Month Takes 4-8 weeks Certificate
- 40. WAIT! Before you build 100s of Dashboards... Avoid code repetition Style Guidelines Test Coverage Apply DEV Best Practices SQL Version Control
- 41. Warning Signs You’re Doing Analytics Wrong
- 42. Don’t Create Datasets in the US, use EU Always keep in EU Can’t join with EU datasets
- 43. Avoid Saved Queries - They Don’t Scale
- 44. This Report is Broken. Who’s the Owner? No Idea, Help Yourself!
- 45. Don’t Use Custom Queries as Data Sources
- 46. Each Analyst uses a different SQL Code & Naming
- 47. Instead, Apply “Data Product” Thinking Data Product Owner Scrum Process Data SLAs Treat Data as a Product
- 48. #1 Pick And Implement A SQL Styleguide https://github.com/mattm/sql-style-guide
- 49. #2 Define A Dataset & Table Naming Convention Fields Domains Datasets Tables product seo seo_google_search_console query_by_page_daily ga_product_order_value page
- 50. #3 Use dbt for all SQL Transformations
- 51. Build Your Data Product like a German House
- 52. Phase 3: Operational Analytics in Production
- 53. You Need Clean Data for Operational Analytics
- 54. What is Operational Analytics for Bergzeit? ML Products Profit Bidding Rule-based Products Data Uploads Attribution Model Updating Affiliate Sales
- 55. Case: Upload Your Shopping Feed Every 10 Min 2. Cloud Function with 15 lines of code 3. Schedule Cronjob 1. Get GCS Bucket Name Code samples: https://gist.github.com/ChrisGutknecht/fde93092e21039299ab76715596eac01
- 56. Case: Profit Bidding & Report More Details: https://www.slideshare.net/ChristopherGutknecht/gross-profit-bidding-for-ecommerce-smx-virtual-2021
- 57. Operational Data Errors Can be Really Costly
- 58. We Need To Prepare for Data Pipeline Errors 2. Execution 1. Source Data 3. Target Table Test Coverage Test Coverage Retry Policies
- 59. How Can We Solve Our Missing Data Problem?
- 60. Solution: Define a Table Freshness Alert with dbt
- 61. Define All Data Quality Tests in dbt Non Null Values Restricted Values Uniqueness Data Freshness
- 62. Simple: Cloud Scheduler for Retry Execution
- 63. Advanced: Use Airflow For Data Task Graphs Retries on Every Task Alerting & Monitoring Data Tasks in DAGs Snaspshots & Backfills
- 64. Let’s Finish With a Strategic View From the Peak
- 65. How Can We Generate Value? Focus on Actions 1. Define Actions 3. Factors 2. Success Metrics What Will You Do Differently If You Have the Data? What Would Success in Metrics Look Like? Which Factors Influence Success? 4. Tests How Can We Test Actions on These Factors?
- 66. Who Should Be Your Data Hire? 1. Focus: SQL & Warehouse 3. Focus: ML Models 2. Focus: Data Pipelines
- 67. Your Takeaways from this Session 1. When and why you should invest in a Marketing DWH 3. How to explore use cases by combining data sources 5. Design outcome-oriented questions for analytics projects 2. Learn the data ecosystem and the benefits of BigQuery 4. Many tactical tips for daily use
- 68. Thanks for Your Time. Looking Forward To Questions! Chris Gutknecht | Teamlead A&O | Hiring a PPC!
- 69. 2. Dashboards 1. Greenfield ANNEX: The Three Phases of a Data Platform 3. Operational Analytics
- 70. Data Warehouse vs Data Lake Structured Data Table Schemas Transactions Sharded Files Unstructured Data Lakehouse
- 71. Why Focus On Google Cloud & Big Query? Market Leader in Data Analytics* Free Google Data Connectors Seamless low-tech scaling source:: Forrester Research 2021
- 72. The Best Cloud Data Warehouse? It Depends Source: https://medium.com/pocket-gems/a-comparative-analysis-between-bigquery-redshift-and-snowflake-8d194fdf5693 Google Data Sources BigQuery = Google Cloud
- 73. How often do we Ingest Data? Real-Time Stream Processing Batch Processing or
- 74. Connected Sheets = ‘NoCode’ Analysis on BQ
- 75. Dom Woodman’s Search Console Downloader https://www.pipedout.com/resources/tools/download-search-console/
- 76. Data Modeling: Star Schema & 3rd Normal Form Third Normal Form
- 77. Data Modeling Choices: Denormalized Third Normal Form Denormalized
- 78. 2. Dashboards 1. Greenfield ANNEX: The Three Phases of a Data Platform 3. Operational Analytics
- 79. Or Pick the Official “Data Analytics” Certificate 8 Courses 4-6 Months (longer) Intro to R Certificate
- 80. Export Your Your GPC Costs to BigQuery
- 81. Monitor Your BigQuery Costs in a Dashboard Monitor Expensive Queries: https://www.pascallandau.com/bigquery-snippets/monitor-query-costs/
- 82. 2. Dashboards 1. Greenfield ANNEX: The Three Phases of a Data Platform 3. Operational Analytics
- 83. How To Sync Segments: CDP vs Reverse ETL? Customer Data Platform Reverse ETL (DWH = Central)