Ole Olesen-Bagneux’s Post

Understanding the Bronze Layer in the Medallion Architecture -------------------------------------------------------------------- In a data engineering workflow, the Bronze Layer acts as the foundation of the Medallion Architecture. It’s the first stage where raw data from various sources is ingested and stored in its original form without any transformation or cleaning. The main purpose of this layer is to capture data exactly as it arrives, ensuring that no information is lost during the process. This makes it easier to reprocess or trace back the data if any issue occurs in later stages. Key points about the Bronze Layer: * It handles raw, unprocessed data from multiple sources such as APIs, databases, or streaming services. * Data is often stored in parquet, CSV, or JSON formats for flexibility. * It provides data lineage and traceability, allowing teams to verify the source and quality of data. * The data from this layer is later refined and processed into the Silver Layer for cleansing and structuring. The Bronze Layer ensures that the data foundation is reliable, auditable, and ready for downstream transformations. It’s a crucial step in building scalable and trustworthy data pipelines. #DataEngineering #MedallionArchitecture #AzureDatabricks #AzureDataFactory #ETL #DataPipelines #BigData #Python #SQL #CareerInData

Grab your Free Copy of O’Reilly e-book for Data Enthusiasts 🔥 Meet Buf, a modern take on event streaming that’s fast becoming a drop-in replacement for Apache Kafka, and it’s 8x cheaper to operate. But cost isn’t the only reason engineers are paying attention. It’s how Buf rethinks data streaming at its core. Here’s what makes Buf stand out: ⚙️ Schema-first approach: Every message has a contract, eliminating consumer–producer mismatches. 🧩 Semantic validation built-in: Data isn’t just transmitted, it’s understood. 💡 Drop-in Kafka replacement: Works with familiar patterns but removes operational overhead. 🚀 High performance, lower cost: Optimized for real-world, large-scale workloads, 8x cheaper to operate than Kafka. 🔍 Clarity over chaos: Brings predictability to data pipelines that were once brittle and error-prone. And coming to the most interesting part: Buf is sponsoring a free O’Reilly book - “Data Engineering Design Patterns” by Bartosz Konieczny, it’s packed with lessons every data engineer should know: - How to design reliable ingestion pipelines Ensuring data quality, idempotency, and error handling - Building observability right into your data workflows - Reusable patterns that work across any tech stack - Open-source examples and real-world use cases you can apply right away It’s inspiring to see a company that’s not just innovating the tech, but also investing in the community’s knowledge. 👏 Huge thanks to the Buf team for their contribution to the data engineering ecosystem and also for collaborating on this post. 👉 Grab your free O’Reilly copy of “Data Engineering Design Patterns” e-book here: https://fandf.co/4nTJWy3 Please consider Reposting and commenting to help this reach every Data engineering enthusiast.

Is your RAG system over-engineered? 🤔 While specialized vector databases get attention, experienced engineers value proven infrastructure. This session makes the case for Postgres as a robust, scalable foundation for enterprise AI. At QCon AI New York 2025 (December 16-17), Gwen (Chen) Shapira, Co-Founder & CPO @Nile, previously Engineering Leader @Confluent demonstrates using Postgres with pgvector to build production-grade RAG systems. You'll learn: 🔹 Data modeling for RAG systems 🔹 Performance tuning for large datasets 🔹 Integration with existing enterprise data ecosystems Streamline your AI architecture, reduce operational overhead, and ground AI in business facts using infrastructure you already trust. Early bird ends October 14! Explore the session and register: https://bit.ly/3IvKlrD #QConAI #Postgres #RAG #VectorDatabase #EnterpriseAI

When George Fraser from Fivetran said dbt was like “Rails for data” at Coalesce this week, it made me smile because that was exactly how I felt when I started with dbt. Ruby on Rails taught me a lot about; - capturing requirements from users - developing in an agile way - infrastructure is code - automated testing - automated deployment - ci/cd and code reviews - convention over configuration - simplicity and having fewer moving parts that can break - and how small teams can have a big impact An enterprise application that we delivered at Amgen back in 2012 is still in use. I was shadow IT doing everything different from the enterprise. No one used Postgres there at the time. No one knew what Ruby was and no one had 90% test coverage. We had just two developers and one product manager, me. Later on when I was part of the team shaping our new Data platform at Amgen I brought a lot of those lessons into what we built, but it wasn’t until I found dbt that I was finally able to do what I initially envisioned for what a good data architecture could be. dbt felt like Rails, but it didn’t go far enough. It still doesn’t go far enough too many decisions are left for users that have no experience and too many options to choose from one thing that rails does well is telling you exactly how to do something even if you don’t know why you’re doing it eventually, you appreciate why that decision was made. Rails encapsulates the many years of experience of the rails community into conventions that help you scale. This is how we think at Datacoves. How can users know what they don’t know when they’re starting out? How do they know that one decision today is gonna have a lasting impact in a year or two? As a community, we can still make things better and help each other mature the state of the art and analytics. At Datacoves we’re not consultants we’re just opinionated because we have learned what’s important at scale and even when working with small teams, we helped them accelerate their processes by not having to learn so many new things.

🚀 New Open Repository: Delta Drift Governance Architecture As enterprises move from isolated models to production pipelines, data & model drift quietly becomes the silent disruptor of reliability. During my time leading Data Engineering & Science teams at Mashreq, I saw how even well-structured ML systems lose trust without drift observability. Built on Databricks Community Edition, this POC tracks data & model drift using PSI — with Delta-safe writes, heatmaps, and JSON telemetry for dashboards. 📘 Read the story: https://lnkd.in/gA6XtcrS 💻 Explore the repo: https://lnkd.in/g9GBbcwr #Databricks #DeltaLake #ResponsibleAI #ModelGovernance #AIObservability #MLOps #DataEngineering #OpenSource

Ever had a “green” pipeline in Prefect… and an empty dashboard the next morning? That’s what happens when observability is an afterthought. In the rush to deliver, many teams build one big monolithic flow ‒ simple to start, painful to scale. But when it fails, debugging turns into log archeology. One table breaks, the whole job fails, and you’re left guessing what went wrong. There’s a smarter way to build ‒ granular, focused flows. Breaking pipelines into smaller, independent deployments gives you visibility where it matters. 🔗Read more on breaking down deployments for improved efficiency: https://lnkd.in/dhWmHmch How do you keep observability front and center in your data pipelines? #DataEngineering #DataOps #Observability #ETL #DataReliability #DataPlatforms

⚙️ Kappa Architecture — The Simplified Future of Data Processing For years, data teams struggled with the complexity of the Lambda Architecture — two codebases, two data pipelines, double the maintenance. Then came a revolutionary idea 💡 — “What if we could handle both real-time and batch data in a single system?” That’s where Kappa Architecture enters the picture. 📊 According to Fundamentals of Data Engineering, Kappa Architecture eliminates the batch layer entirely. Instead, it treats streaming data as the source of truth and reprocesses everything through that single pipeline. The benefits? 🔹 Simplified codebase — one pipeline to rule them all 🔹 Real-time insights with no duplication of effort 🔹 Scalable, event-driven design for modern data platforms Tech giants like LinkedIn, Netflix, and Uber have embraced this philosophy — using stream processing frameworks like Apache Kafka, Flink, and Spark Streaming to power instant, reliable data flows. 🚀 Kappa isn’t just a framework. It’s a mindset shift — from “collect now, analyze later” to “analyze as it happens.” As real-time data becomes the norm, Kappa Architecture isn’t just the future — it’s the present. 👉 What’s your take? Do you think Kappa Architecture will completely replace Lambda, or will both coexist for years to come? #DataEngineering #KappaArchitecture #StreamingData #BigData #DataArchitecture #RealTimeAnalytics #ApacheKafka #DataOps #MachineLearning #CloudComputing

🚀 𝑰𝒔 𝑴𝒆𝒅𝒂𝒍𝒍𝒊𝒐𝒏 𝑨𝒓𝒄𝒉𝒊𝒕𝒆𝒄𝒕𝒖𝒓𝒆 𝒕𝒉𝒆 𝑭𝒖𝒕𝒖𝒓𝒆 𝒐𝒇 𝑺𝒄𝒂𝒍𝒂𝒃𝒍𝒆 𝑫𝒂𝒕𝒂 𝑷𝒍𝒂𝒕𝒇𝒐𝒓𝒎𝒔? As data volumes explode and analytics demands grow, the Medallion Architecture (Bronze → Silver → Gold) is becoming a go-to pattern for organizing data lakes and lakehouses. But like every architecture, it comes with trade-offs. 🔷 ✅ 𝑷𝒓𝒐𝒔 𝒐𝒇 𝑴𝒆𝒅𝒂𝒍𝒍𝒊𝒐𝒏 𝑨𝒓𝒄𝒉𝒊𝒕𝒆𝒄𝒕𝒖𝒓𝒆: 🥇 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 – Each layer improves cleanliness, trust, and usability 🔄 𝗠𝗼𝗱𝘂𝗹𝗮𝗿𝗶𝘁𝘆 – Easier debugging, versioning, and scaling 🧩 𝗦𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝗖𝗼𝗻𝗰𝗲𝗿𝗻𝘀 – Raw, processed, and business-ready data are cleanly separated 🔐 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗥𝗲𝗮𝗱𝘆 – Fine-grained access control and auditing ⚙️ 𝗧𝗼𝗼𝗹 𝗔𝗴𝗻𝗼𝘀𝘁𝗶𝗰 – Works well with Databricks, Delta Lake, Spark, etc. 🔶 ⚠️ 𝑪𝒐𝒏𝒔 𝒕𝒐 𝑾𝒂𝒕𝒄𝒉 𝑶𝒖𝒕 𝑭𝒐𝒓: 📦 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝗢𝘃𝗲𝗿𝗵𝗲𝗮𝗱 – Multiple data copies = higher storage costs ⏱️ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 – Layer-by-layer processing may slow real-time use cases 🛠️ 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗖𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆 – More moving parts to monitor and maintain 💰 𝗧𝗼𝗼𝗹𝗶𝗻𝗴 𝗗𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝗰𝘆 – Often tied to specific platforms like Databricks 🎯 𝑩𝒐𝒕𝒕𝒐𝒎 𝑳𝒊𝒏𝒆: Medallion Architecture is not a silver bullet, but when applied thoughtfully, it adds incredible structure, scalability, and trust to your data platform. Start small. Evolve layer by layer. Let your data maturity guide your design. 💬 Curious to hear from others: How are you implementing or adapting Medallion Architecture in your stack? 💡 Follow AAKRATI MISHRA for more such Data Engineering topics #DataEngineering #MedallionArchitecture #Lakehouse #Databricks #DataScalability #ETL #ModernDataStack #DataQuality #Analytics #ADF #MedallionArchitecture

The Rise of Metadata-Driven Data Engineering Let’s be honest — our pipelines are getting out of control. Thousands of tables. Hundreds of DAGs. Dozens of tools. And when something breaks, the first question everyone asks is: 👉 “Where did this data even come from?” That’s where metadata-driven data engineering changes the game. Instead of managing pipelines through code alone, we’re now managing them through context — information about the data itself: lineage, quality, ownership, schema evolution, business meaning. Here’s what this shift looks like in practice 👇 ⚙️ Yesterday: ETL scripts scattered across repos No visibility into dependencies Manual documentation nobody updates 💡 Today: Tools like OpenMetadata, DataHub, and Atlan automatically track lineage Pipelines adapt based on metadata (e.g., schema changes trigger transformations) Data governance becomes operational, not afterthought And the best part? When metadata becomes a first-class citizen, data engineers can finally focus on designing systems, not chasing data drift. This is the future: ➡️ Pipelines that describe themselves. ➡️ Quality rules that evolve automatically. ➡️ Discovery that’s instant, not tribal. Metadata is no longer documentation. It’s automation. What do you think — Are we moving toward a world where metadata drives pipelines instead of pipelines generating metadata? 👇 Drop your thoughts — I’d love to hear how your team handles this shift. #DataEngineering #Metadata #DataGovernance #ETL #DataOps #ModernDataStack

I’ve been using the Gemini CLI and Model Context Protocol (MCP) lately — and honestly, it feels like the missing layer in modern data engineering. What used to be fragile glue code between models, APIs, and warehouses is now structured, permissioned automation. Gemini gives me a command-line interface to reason with models. MCP defines how those models connect — securely and contextually — to tools like BigQuery, dbt, or GitHub. Together, they turn AI from a sidekick into an orchestrator — one that understands your data topology, documents transformations, and even proposes pipeline configs safely. It’s not about what the model knows, but what it’s allowed to know. That’s the real unlock for bringing AI into the operational core of data systems. Read more in this post #DataEngineering #Gemini #MCP