Trends Shaping Data Workflows

Data Integration Revolution: ETL, ELT, Reverse ETL, and the AI Paradigm Shift In recents years, we've witnessed a seismic shift in how we handle data integration. Let's break down this evolution and explore where AI is taking us: 1. ETL: The Reliable Workhorse      Extract, Transform, Load - the backbone of data integration for decades. Why it's still relevant: • Critical for complex transformations and data cleansing • Essential for compliance (GDPR, CCPA) - scrubbing sensitive data pre-warehouse • Often the go-to for legacy system integration 2. ELT: The Cloud-Era Innovator Extract, Load, Transform - born from the cloud revolution. Key advantages: • Preserves data granularity - transform only what you need, when you need it • Leverages cheap cloud storage and powerful cloud compute • Enables agile analytics - transform data on-the-fly for various use cases Personal experience: Migrating a financial services data pipeline from ETL to ELT cut processing time by 60% and opened up new analytics possibilities. 3. Reverse ETL: The Insights Activator The missing link in many data strategies. Why it's game-changing: • Operationalizes data insights - pushes warehouse data to front-line tools • Enables data democracy - right data, right place, right time • Closes the analytics loop - from raw data to actionable intelligence Use case: E-commerce company using Reverse ETL to sync customer segments from their data warehouse directly to their marketing platforms, supercharging personalization. 4. AI: The Force Multiplier AI isn't just enhancing these processes; it's redefining them: • Automated data discovery and mapping • Intelligent data quality management and anomaly detection • Self-optimizing data pipelines • Predictive maintenance and capacity planning Emerging trend: AI-driven data fabric architectures that dynamically integrate and manage data across complex environments. The Pragmatic Approach: In reality, most organizations need a mix of these approaches. The key is knowing when to use each: • ETL for sensitive data and complex transformations • ELT for large-scale, cloud-based analytics • Reverse ETL for activating insights in operational systems AI should be seen as an enabler across all these processes, not a replacement. Looking Ahead: The future of data integration lies in seamless, AI-driven orchestration of these techniques, creating a unified data fabric that adapts to business needs in real-time. How are you balancing these approaches in your data stack? What challenges are you facing in adopting AI-driven data integration?

This year, the State of Data and AI Engineering report has been marked by consolidation, innovation and strategic shifts across the data infrastructure landscape. I identified 5 key trends that are defining a data engineering ecosystem that is increasingly AI-driven, performance-focused and strategically realigned. Here's a sneak peek at what the report covers: - The Diminishing MLOps Landscape: As the standalone MLOps space is rapidly consolidating, capabilities are being absorbed into broader platforms, signaling a shift toward unified, end-to-end AI systems. - LLM Accuracy, Monitoring & Performance is Blooming: Following 2024's shift toward LLM accuracy monitoring, ensuring the reliability of generative AI models has moved from "nice-to-have" to business-critical. - AWS Glue and Catalog Vendor Lock-in: While Snowflake just announced read/write support for federated Iceberg REST catalogs, finally loosening its catalog grip, AWS Glue is already offering full read/write federation, and is therefore the neutral catalog of choice for teams avoiding vendor lock-in. - Storage Providers Are Prioritizing Performance: in line with the growing demand for low-latency storage, we see a broader trend in which cloud providers are racing to meet the storage needs of AI and real-time analytics workloads. - BigQuery's Ascent in the Data Warehouse Wars: with 5x the number of customers of both Snowflake and Databricks combined, BigQuery is solidifying its role as a cornerstone of Google Cloud’s data and AI stack. These trends highlight how data engineering is evolving at an unprecedented pace to meet the demands of a rapidly changing technological landscape. Want to dive deeper into these critical insights and understand their implications for your data strategy? Read the full report here: https://lnkd.in/dPCYrgg6 #DataEngineering #AI #DataStrategy #TechTrends #DataInfrastructure #GenerativeAI #DataQuality #MLOps

Our expectations of AI are about to go up a level. The next wave of AI workplace products will enter the market, and they’re going to make use cases that wowed us a year ago look…kinda old. The thing that’s notable about the next wave of applications is that they’re taking the power of an LLM and connecting it with live enterprise data. Together, this connection creates the conditions for new use cases that will transform the way we approach so many day-to-day tasks. As an example, think of the task of writing a sales email. Over the last year or so, you might have used an LLM to help you create a first draft of a standard sales email. Helpful? Yes. But it’s just the beginning. Now imagine writing a sales email with an AI application that has access to live data in your CRM, email tool, and other apps. It knows the company, role, industry, and country of the person you want to email. It knows what has worked well in prior emails to similar people. It even knows what competitor product they may be using. And it generates an email draft that’s tailored to that specific person using all of that information. That’s just one example of course. Imagine daily workflows across every role at a company, all potentially benefiting from applications that combine LLMs with enterprise data. An interesting second-order effect of this shift is that it’ll lead to many companies having a reckoning with the quality and cleanliness of their data. Because as the potential of AI in the workplace increases, the importance of having rich, accurate, organized data will increase too. We’re approaching the “end of the beginning” of the chapter that began with the launch of ChatGPT in November 2022. And things are about to get really interesting! What AI workplace use cases do you think will improve the most in 2024? #ai #data #futureofwork

Are we witnessing the end of forms in business applications? For decades, forms have been the backbone of data collection and interaction in organizations—structured, reliable, but rigid. Now, AI is rewriting the rules. Imagine a workplace where entering data isn’t about filling fields but having a conversation. Picture an AI that can understand text, recognize visuals, and even interpret sound to meet your needs seamlessly. Instead of navigating endless forms, employees can: ➡️ Ask an AI chatbot for instant insights ➡️ Scan shelves with a smartphone to check inventory in real time ➡️ Use natural language to retrieve and interact with data This evolution is about more than efficiency—it’s about creating fluid, intuitive experiences that empower employees to focus on what really matters: decision-making, innovation, and customer impact. But with every opportunity comes challenges. As we move beyond forms, organizations must rethink data governance, security, and compliance to ensure this shift is as sustainable as it is transformative. Are you ready to embrace a workplace without forms? Let’s discuss how conversational AI, vision capabilities, and human-centered design are reshaping the future of business applications. Check out my latest article to dive deeper into this transformation:

The future of analytics is a metrics-first operating system. Let’s discuss three macro trends driving this inevitable evolution. Three Macro Trends: 1) Sophisticated and Standardized Data Modeling Data modeling is now widely accepted and implemented by data teams of all sizes. These models are increasingly capturing the nuances of varied business models. - From the early days of Kimball to today, powered by advanced data modeling and management tools, practitioners are coalescing around concepts like time grains, entities, dimensions, attributes and metrics modeled on top of a data platform. - Compared to even 7-8 years ago, we’ve made significant strides in tailoring these concepts for various business types—consumer, enterprise, and marketplace—across different usage and monetization models. - We’re now proficient in standardizing metrics and calculations for specific domains, such as sales funnels, lifetime value calculations for marketing, cohort tracking for finance, and usage and retention models for product teams. The architecture of data production is more robust than ever as data and analytics engineers refine their practices. Now, let’s look at the consumption side. 2) Repeatable Analytics Workflows Analytics workflows are becoming repeatable, and are centered around metrics: - Periodic business reviews and board meetings demand consistent metrics root-cause analysis, including variance analysis against budgets or plans. - Business initiatives, launches, and experiments require expedient analysis to extract actionable insights and drive further iterations. Experimentation is becoming a core workflow within organizations. - Organizations need to align on strategy, formulate hypotheses, and set metric targets to monitor progress effectively. 3) Limitations of Scaling Data Teams The cold reality is that data teams are never going to be big enough. This has become even more apparent as investment levels have waned over the past three years. Combining these insights: 1) The increasing standardization of data models across business models 2) The secularization and rise of repeatable workflows centered around metrics. 3) The need to maximize data team leverage It is clear that a metrics-first, low to no code operating system is the future. Such a system will provide immense leverage for data teams, while empowering executives and operators. This shift towards a metrics-first operating system represents the next evolution in analytics, driving both operational efficiency and strategic agility.

There's a major evolution coming in data management that I argue will reshape our entire industry-- it's Shift Left Data. Over the past few years, I've watched data contracts move from being a fringe idea in LinkedIn posts to becoming a real driver of organizational change at global enterprises (check out our Shift Left Data Conference recordings). Specifically, I'm noticing that the lines that differentiate the workflows of different teams are being rewritten and bringing stakeholders from various disciplines together in a way we haven't seen before. This is especially true among software and data teams. Yes... AI is a huge catalyst for these shifts (more attention, budget, and scrutiny), but I argue we have been moving towards this direction even before ChatGPT gained global traction. In particular, DevOps and DevSecOps teams have already gone through their "shift left" moment and found success. I firmly believe it's now the data industry's turn. I'm going to be writing more heavily on this on LinkedIn in the coming weeks but here are a few resources from myself and others in the industry that I think are a great start: 1. Shift Left Data Manifesto (https://lnkd.in/gU36qr54) 2. Glassdoor: Data Quality at Petabyte Scale: Building Trust in the Data Lifecycle (https://lnkd.in/gbEApwzD) 3. Shifting Left with Data DevOp (https://lnkd.in/g5G57f9T) 4. Wayfair’s Multi-year Data Mesh Journey (https://lnkd.in/g2YpAdXW) 5. Creating source-aligned data products in Adevinta Spain (https://lnkd.in/gjdE5Dgf) Good luck!

🚀 Composable Data Platforms: Building the Future, Piece by Piece 🚀 The world of data is evolving—fast. The old monolithic systems are making way for composable data platforms: modular, scalable, and designed for innovation. This isn't just a trend for founders, builders, and visionaries. It’s an opportunity. ✨ Why Composable? It’s not just about tools—it’s about freedom: 🔗 Freedom to combine the best storage, transformation, and querying technologies. ⚡ Freedom to scale each component independently, optimizing for your needs. 🎯 Freedom to break free from vendor lock-in and design systems as agile as your vision. At the heart of this revolution are game-changing technologies: 🪶 Apache Arrow: A universal data format transforming in-memory analytics. With Arrow Flight, data moves faster across distributed systems, powering real-time decision-making. 🏗 Apache Iceberg: The table format redefines how we manage data lakes. Iceberg gives you ACID compliance, schema evolution, and even time travel—turning raw data into a goldmine. 🔄 Apache DataFusion: A Rust-based query execution framework, lightning-fast and endlessly flexible. For those building custom pipelines, this is your secret weapon. 🦆 DuckDB: Think "SQLite for analytics." Simple, lightweight, and incredibly powerful for interactive queries—perfect for data apps, notebooks, and on-the-go insights. 🔥 Spark, Trino, and others: Query engines are stepping up, enabling federation, advanced optimizations, and seamless lakehouse integration. But here’s the real magic: connection. The composable future is about systems that work together effortlessly: 💡 Use Arrow with DuckDB for in-memory analytics at lightning speed. 💡 Combine Iceberg with Trino for scalable, real-time querying. 💡 Integrate DataFusion into workflows that prioritize flexibility and precision. 📊 The Vision for Tomorrow: 1️⃣ Data lakehouses blending the speed of warehouses with the flexibility of lakes. 2️⃣ The rise of Rust in data engineering—performance and safety, redefined. 3️⃣ Cost-aware, pay-as-you-go architectures fueling lean startups and enterprise efficiency. 4️⃣ Open standards like Arrow and Iceberg, creating a world where interoperability is the norm. 🔮 For Founders and Builders: This is more than a technical shift—it’s a call to action. The composable revolution is a blank canvas, waiting for bold ideas. 💭 What industries can you transform by redefining how data flows and connects? 💭 What tools or platforms will YOU create to power the next generation of analytics? 💭 How will you take advantage of modular, composable systems to outpace legacy competitors? ✨ The future is here, and it’s composable. What are you building? What challenges inspire you to innovate? Share your vision below! 👇 #ComposableDataPlatforms #Innovation #StartupFounders #ApacheArrow #ApacheIceberg #DuckDB #DataEngineering #Startups #Founders #AI #ML #LLM #GenAI #Data

AI-powered answers are transforming enterprise productivity by 30% when integrated with business data systems. The future is here. Today we analyzed the SAP - Perplexity partnership announced at SAP Sapphire 2025, and the implications are profound for the entire HR technology ecosystem. We love what Aravind Srinivas is building. What we observed: → Joule (SAP's AI assistant) now delivers visual, structured answers by combining internal business data with Perplexity's real-time external intelligence ↳ HR leaders can instantly forecast talent needs based on market shifts, competitor moves, and internal workforce data → The integration unifies structured (HRIS, ATS) and unstructured data (documents, web) into a single seamless experience ↳ This eliminates the "system-hopping" that currently consumes 40% of an HR professional's productivity → Custom AI agents now automate complex talent workflows across the employee lifecycle ↳ Think: personalized onboarding that adapts in real-time to each new hire's behavior and engagement signals The most striking insight? Organizations implementing this integrated approach are seeing HR teams shift from 70% administrative work to 70% strategic talent advisory. ✨ Navigating the New Work Frontier ✨ As we shape tomorrow's workplace, consider these pivotal moves: → Establish your "HR Knowledge Graph" connecting disparate people data sources before siloes become barriers → Create cross-functional AI governance teams with explicit charters for responsible automation → Develop new HR capability models emphasizing human+AI collaboration skills → Audit existing workflows to identify high-value integration points for intelligent automation The future belongs to HR leaders who can orchestrate this symphony of human expertise and machine intelligence. This is precisely why we're building People Atom to empower visionary HR leaders with the private network and tools needed to navigate this transformation together. Curious how other forward-thinking HR executives are implementing these approaches? Join our waiting list and become part of the conversation shaping the next era of work. Love exploring the frontiers of HR transformation with you, Joe PS: Building PeopleAtom—the private network for HR leaders transforming people and technology. Because the most powerful insights happen when brilliant minds connect.

Back to my roots :) A fast growing, cool data startup Coalesce.io asked me for my predictions for the data space in 2025. Here are my top 4: 1.) AI DECISIONING - We’ll see data, marketing, and digital product teams adopting AI Decisioning platforms on top of their data warehouses to drive 1:1 personalization with their customers across all channels. Instead of building manual rules, audiences, and journeys, AI will look at each customer and decide the best actions to drive your company’s goals, and continuously learn and get smarter. 2.) DATA ACTIVATION - Data warehouses are going to continue to grow as the center of data gravity and more business teams will want to get data out of these warehouses into the tools they use every day. 3.) WAREHOUSE 3.0 - Open format tables like Iceberg will continue to grow in adoption by companies and the warehouses. Separately, batch and streaming workflows are going to continue to converge as data warehouses support low-latency use-cases. 4.) DATA PRODUCTS - We’re entering the era of data products, where companies don’t just build one off reports or analyses but really think about what artifacts (eg “marketing user data” or “customer service ticket insights”) they should be exposing to other data teams and the business for ongoing consumption. You should check out the full report here: https://lnkd.in/ew9C2Yxs I’d love your reactions— do these trends seem right for the next year? What else am I missing?