OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pete Hunt - Elementl.pdf
0 likes50 views
The document discusses insights from frontend engineering that can benefit data engineering, highlighting three main topics: the shift in platforms, the rise of serious engineering, and the lessons data can learn from frontend practices. It emphasizes the complexity challenges faced by both domains and the importance of modules, abstraction, and improved tooling, as well as the transition from imperative to declarative paradigms. Ultimately, data and frontend are on parallel paths towards becoming more structured and efficient engineering disciplines.
OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pete Hunt - Elementl.pdf
- 1. What Data Engineering can learn from Frontend Engineering Pete Hunt @floydophone or @dagster on Twitter
- 2. I’m at Elementl, the company behind Dagster. Previously, I was a founding member of React.js, the most popular JS framework.
- 3. Three topics: 1. The platform shift 2. The rise of serious engineering 3. What data can learn from frontend
- 4. IT’S LIKE POETRY. THEY RHYME George Lucas on the synergies between sci-fi filmmaking and software development (1999)
- 5. 1. The platform shift
- 6. Changes to a layer of the stack ripple upwards and downwards. Your infrastructure You & your tools Your users
- 8. Fast runtimes → Ambitious apps → Complex software
- 9. Two problems with the JS stack 10 years ago: 1. Codebases became large and hard to manage. No tools for large-scale code reuse. 2. Business logic became extremely complex. Existing tools like jQuery were imperative and hard to work with at scale.
- 10. Data went through a similar transition.
- 12. Fast runtimes → Ambitious apps → Complex software
- 13. Scalable engines → Ambitious pipelines → Complex queries
- 14. Two problems with the JS stack 10 years ago: 1. Codebases became large and hard to manage. No tools for large-scale code reuse. 2. Business logic became extremely complex. Existing tools like jQuery were imperative and hard to work with at scale.
- 15. Two problems with the new data stack: 1. Large queries are hard to manage. SQL has no tools for large-scale code reuse. 2. Connecting your warehouse or lake to the outside world is complex. This requires an orchestrator, and most are imperative and hard to reason about.
- 16. Frontend and data have become…
- 18. Chaos → Source Control (and CI)
- 19. Copy/paste → Code Reuse
- 20. Lines of Code : Your Project :: Fossil Fuels : Modern Life
- 21. Two weapons in the war on complexity: modules and abstraction.
- 22. Modules ● Keep your code organized ● Let you reuse code in multiple places
- 23. Frontend: CommonJS (2009) Analytics/data: dbt models (2016)
- 24. To get modules, we need a compiler: webpack or dbt.
- 25. Compilers can add complexity ● You need a compile step ● The compile step can be slow ● You need to debug compiled code
- 26. Abstraction ● Reuse code for different purposes. ● Limit how much of the system you need to hold in your head.
- 27. Frontend: React components (2013) Analytics: dbt macros (2016) Orchestration: Airflow operators (2014)
- 28. Text editors → IDEs
- 29. Yes, emacs is better than vi.
- 30. Life without an IDE sucks.
- 31. ❌ Where is this thing defined? ❌ Where are my docs? ❌ How do I run my linter?
- 32. ✅ Jump-to-definition ✅ Inline docs and autocomplete ✅ Continuous linting, autoformatting etc.
- 34. 3. What can data learn from frontend?
- 35. Frontend and data are both on a journey to become Real Engineering™.
- 36. Imperative → Declarative via reconciliation. 1.
- 37. Express your app as a desired output state. The framework figures out how to transition the world to that state.
- 42. ✅ Correctness checks ✅ Documentation ✅ Autocomplete ✅ Jump-to-definition
- 43. 🤔 Adopt mypy? 🤔 Data contracts? 🤔 SDAs? 🤔 SQL → TypeSQL?
- 46. ❌ Poor IDE / tooling support ❌ Verbose ❌ Large templates are rats’ nests
- 47. ✅ Full language ecosystem ✅ Expressivity ✅ Large-scale code reuse ✅ Large hiring pool
- 48. 🤔 Less YAML? Please…? 🤔 dbt/SQL → TypeSQL?
- 50. 🚩🚩🚩 “Hey is anyone using staging3 right now?” 🚩🚩🚩
- 56. ✅ No fighting over staging ✅ Easy collaboration ✅ Fast iteration
- 60. To summarize:
- 61. ● Data and frontend are both progressing towards Real Engineering™ and are following similar trajectories. ● Imperative → Declarative code via reconciliation ● Dynamic → Static typing ● Templating languages → Programming languages ● Staging → Branching ● Plan on consolidation in the future
- 63. ● Imperative → Declarative: ○ Software-Defined Assets ● Dynamic → Static typing ○ Pandera ○ Strongly typed APIs ● Templating languages → Programming languages ○ Declaratively express assets as Python code ○ No DSLs, YAML, or templating languages. Real Python code. ● Staging → Branching ○ Dagster Cloud branch deployments ● Plan on consolidation in the future ○ dbt-backed assets ○ Airbyte-backed assets ○ Many more supported
- 64. @floydophone or @dagster on Twitter