Distilling Insights @ Appsflyer (Data Architecture)
3 likes•1,901 views
The document discusses a company's data architecture and strategy for transforming raw data into useful insights. It outlines the various technologies used at each stage, from ingesting raw data using Kafka to storing aggregated data in a columnar database and performing analytics with Spark. It also touches on evaluating different SQL query engines and machine learning with Spark ML to generate predictive insights for dashboards.
Distilling Insights @ Appsflyer (Data Architecture)
- 1. Distilling insights @ Arnon Rotem-‐Gal-‐Oz Chief Data Officer
- 4. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts
- 5. Data’s hierarchy of needs* *With apologies to Maslow Acted upon presented Distilled Usable Accessible Exist
- 6. Exist
- 7. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts
- 8. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts
- 9. Working off of RAW data
- 10. “Malting” Just slap SQL on everything Accessible
- 11. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts
- 13. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts
- 15. Kafka Columnar Database (Redshift- evaluating Vertica) IMDG (Ignite - evaluating Geode) Secor Spark Aggregations SparkSQL (evaluating Drill, Presto) SQL SQL Raw (sequence files) DW (parquet files) DM (Aggregations) Application dashboard Self-serve BI (TBD) Spark ETL Spark Spark ML Latest Events Scoring exploration Agg. logic Internal tools installs clicksinapplaunches Accounts
- 17. Sidetrack: On use of Spark
- 18. Hadoop & Mesos
- 20. Land data in a queue
- 21. All data is time-‐series
- 22. Enrich with foreign keys before persisting
- 23. Analyze and balance jobs
- 24. Not everything is big data