James Dales

It’s interesting how technology & landscape changes so much in the industry. I remember in early 2022, going around major data conferences and speaking about “Open Table Formats” and “Lakehouse architecture”. At that time, people knew very little about open storage formats. In fact, most of my time was spent on educating engineers/architects on the benefit of open storage. How this idea of keeping data in an open & independent tier and *not locking* into proprietary vendor warehouses could provide control to customers. Today, all the major data conferences and hallway chats revolves around open table formats. In fact, the number of people understanding and adopting these architectures have also increased. But there’s still work to be done! Now that we understand what an open lakehouse brings, it’s time to think about how we can operationalize it. How do we tackle some of the problems around storage optimization, query performance, cost & governance etc, as we move to the next phase. PS: The photo is from Databricks (Data+AI 2023). This was the only and very first talk on Apache Iceberg. I presented how to build an open & minimalistic lakehouse using everything open source like Apache Spark (compute), Nessie (catalog). The future always excites me! Link of the talk in comments! #dataengineering #softwareengineering

71

7 Comments

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

700

65 Comments

New data infrastructure lore just dropped: The Pendantic Layer. Joe Reis has a great article introducing this idea [https://lnkd.in/gaXw_TPr], but this quote sums it up perfectly. "The Pedantic Layer slows us down by focusing attention on taxonomy and semantics over outcomes. It makes practitioners feel like they need to pass some purity test of definitions before they can get value. It feeds a culture where thought leaders posture about whether CSVs are structured “enough,” while the practitioners who ignore the debate are quietly building models that drive millions in revenue." Coming from startups where you are obsessed with execution over perfection, the "pedantic layer" has been one of my largest gripes with the wider data industry. In celebration of yet another made-up layer (YAML), leave a comment about your favorite data concept, and I'll tell you why you are wrong for obscure nuances that don't apply to most use cases.

44

11 Comments

Don't write a 1000-line SQL Query. Build a data transformation pipeline with modular components, version control, and column-level lineage. Cramming every edge case into one SELECT statement is exhausting. Modularity with maintainability is where it's at. Ready to get started? / Break monoliths into small, named models / Reuse logic functions and macros / Add tests for keys, nulls, ranges, and referential integrity / Put everything in version control / Document inputs, outputs, owners, and assumptions / Make sure column-level lineage is visible / Boring predictability > latest tech tricks Want to level up? Follow - → Zach Wilson at DataExpert.io → Tobias (Toby) Mao at Tobiko → Benjamin Rogojan at Seattle Data Guy → Xinran Waibel at Data Engineer Things → Carly Taylor, MSc at Rebel Data Science → Madison Schott at Learn Analytics Engineering Happy Engineering! #dataengineering #datatransformation #sqlmesh Tobiko

84

12 Comments

Disaster Recovery for PostgreSQL databases is built on proven practice, refined through real production experience. In this webinar, Semab Tariq and Warda Bibi will share how to design PostgreSQL DR strategies that deliver reliability and resilience at scale. Drawing from critical deployments, they will cover: ✅ Trusted approaches to safeguard data ✅ Methods to align RPO and RTO with business priorities ✅ Lessons learned from production environments where uptime is essential This session is designed for teams that want their PostgreSQL environments to be fully prepared and resilient. Reserve your spot here: https://lnkd.in/d5_VC_Ah

32

#Delivering End-to-End Data Engineering with Azure #2025 ----------------------------------------------------------------------------------- . . . . . Recently, I worked on an Azure Data Engineering project where the goal was to build a scalable ETL pipeline for handling large volumes of e-commerce data. Here’s a quick breakdown of how I approached it as a Senior Data Engineer: ✅ Data Ingestion – Used Azure Data Factory to orchestrate ingestion from multiple sources (APIs, SQL DB, and Blob Storage). ✅ Transformation – Leveraged Azure Databricks (PySpark & Delta Lake) for cleaning, joining, and transforming raw data into analytics-ready datasets. ✅ Data Storage – Designed a Lakehouse architecture: - Raw data → Azure Data Lake Gen2 - Curated & transformed → Delta tables in Databricks ✅ Analytics Layer – Exposed the data for the Power BI team via Azure Synapse, enabling real-time dashboards. ✅ CI/CD – Automated deployments with Azure DevOps pipelines to ensure consistency across environments. 📌 Key Learning: Having a well-defined orchestration strategy + Delta Lake reduced processing costs and improved data reliability. 👉 Question for the community: What’s your preferred tool for data transformation in Azure – Databricks, Synapse, or SQL-based ELT? #Azure #DataEngineering #Databricks #AzureDataFactory #PowerBI #SeniorDataEngineer

25

2 Comments