And it really happened, dbt Labs and Fivetran agreed to merge. This is definitely one of those "I was here when it happened" moments for the community. For years Fivetran handled data movement and dbt Labs handled transformation, now they are the same company. Dbt wants Fusion to become the new standard for data transformation. There are clear benefits: state-aware orchestration could cut compute costs by more than half, the dbt team reportedly reduced their Snowflake bill from around $1M a year to about $250K. The Rust-powered Fusion engine brings faster linting, quick SQL parsing, and overall a better developer experience. It was also announced that partners will be able to run the Fusion Engine for free. That’s a smart move and aligns with Fivetran’s numbers, they’re still only at around 5% market share. There’s a lot of transformation market left to win, and dbt clearly plans to push hard to make Fusion the open standard everyone builds on. This merger could for sure define what modern data infrastructure looks like in the AI era. If they don’t, it’ll be a case study in over-integration. Either way, this one is interesting.

Harness is revolutionizing database migrations with its new AI-Powered Database Migration Authoring feature! Developers can now describe schema changes in natural language and instantly get a production-ready migration. This innovative tool, part of Harness Database DevOps, leverages the Software Delivery Knowledge Graph and MCP Server to analyze schemas, generate backward-compatible and validated migrations, and even create rollback options. Say goodbye to manual SQL scripts and spreadsheet tracking. This is a game-changer for addressing the 'AI Velocity Paradox' by bringing AI-powered automation to database delivery. #Harness #DatabaseMigration #AIDevOps #SoftwareDevelopment #TechInnovation https://blog.nitinr.live/

We’re getting ready to launch our data pipeline platform varCHAR - Dev Edition and are currently looking to raise seed funding in the range of $100K to $2M USD. The investment will mainly go toward scaling infrastructure and building out our core engineering team. What makes varCHAR different • 80–90% more cost-efficient than major ETL / data engineering platforms • AI-driven pipeline optimization, debugging and data quality insights, with natural language assistance • Unified platform for databases, APIs/microservices and file pipelines — all with drag-and-drop visual designers - Seamless Db to Kafka to APIs to Cloud Integrations • Developer-friendly UI, built for real production use, with CLI Integration • Powered by Apache Spark, Apache Camel and Kafka — proven enterprise-grade stack • Built-in Kafka ecosystem, from creating Topics to Pub-Sub-DLQ-Consumer-Lag also has Kafka Connect Integration with 15+ connectors support • Native ML integration + SQL & Kafka SQL editors, fully connected to user data sources • Platform has been systematically engineered for the past 12 months — not a weekend hack • Customizable — we offer dedicated support to tailor workflows to a customer’s environment Developer Plans - Free, Basic $29, Pro $129, Max $299 USD. Drop a message or email for Pitch Deck If this excites you, contact@datamolecule.ai Share it with any Investor/VC contacts you may have.

Every data leader knows the feeling - scale goes up, so do your compute bills. But what if your pipelines could think before they ran? What if you could cut unnecessary compute completely? That’s what state-aware orchestration (SAO) brings to dbt Fusion: a way to make pipelines self-aware of what’s actually changed, rebuilding only what’s needed. The outcome: 💡 30x Faster runs 💰 ~30%+ cost reduction 🧠 Less cognitive load on your team dbt Labs has already seen 64% savings by using this internally. Scale doesn’t have to be expensive. It just has to be intelligent.

Announced at #dbtCoalesce: state-aware orchestration is now in Preview for dbt Fusion projects. As data estates grow, so do costs and complexity. Across the dbt ecosystem, teams built over 33 billion models last year. Scale brings real compute costs and cognitive load—but it doesn’t have to. State-aware orchestration changes how pipelines run by only rebuilding what actually needs to run. dbt tracks code and data state, automatically reusing models that haven’t changed. The result: faster runs, lower costs, and less manual scheduling. Early results show the impact: • 10% immediate cost reduction just by turning it on • 15%+ additional savings through fine-tuned freshness configurations • 4% testing efficiency gains by skipping tests when they do not need to re-run Together, that’s a potential 29%+ reduction in annual compute costs with no project rewrites required. At dbt Labs, using Fusion with state-aware orchestration led to a 64% cost reduction in our own analytics project. State-aware orchestration is available in Preview for Enterprise customers running on Fusion. Learn more: https://lnkd.in/g-4ffjpJ

EL <3 T I couldn’t be prouder to see dbt Labs join forces with Fivetran to set the standard for open data infrastructure for the analytics and AI ecosystem. Every company is becoming a data company, and nearly a decade ago, Tristan saw the opportunity to evolve the old art of data transformation into a new discipline: analytics engineering. Today, dbt is the data transformation layer of record for more than 80,000 companies, bringing the principles of software engineering (version control, testing, documentation, and collaboration) into the world of data infrastructure. Now, the data market is again shifting rapidly, as the de facto standard shifts from closed proprietary to open formats, and as the AI-ification of the enterprise elevates the importance of and demands on data teams. This merger with Fivetran is the natural next chapter in dbt's journey. Together, they’re defining the architecture for open data infrastructure, running across any compute engine or AI workload; unifying ingestion, transformation, metadata, and activation; and grounded in open standards like SQL and Iceberg. Enterprises will unlock the value of AI through open data infrastructure, and Fivetran + dbt Labs will be a cornerstone of this new paradigm. The combined company will be a juggernaut, with nearly $600M in ARR, and there’s lots of great work to do together. dbt was one of my first investments at Sequoia Capital together with Matt Miller. I’m honored to represent Sequoia on the combined Board. Congratulations to Tristan Handy, George Fraser, Taylor Brown, Connor McArthur, Drew Banin, Sarah Riley, Brandon Sweeney and both teams.

In most large systems, background processes start life as tactical fixes - a quick script to sync data, clean records, or trigger downstream updates. Over time, they multiply into hundreds of independent jobs, each with its own repo, scheduler, and dependency graph. We’ve been experimenting with collapsing that complexity into a rules engine - a single platform that evaluates conditional logic across data streams and schedules. At its core: - React UI for managing jobs and rules, backed by a Node.js service using json-rules-engine. - Rules are expressed as declarative JSON - each job defines a trigger (schedule or event), a scope (dataset or API feed), and an action (e.g. API update, notification, DB write). - The backend abstracts data ingestion via connectors - REST, ODS feeds, message queues - and persists rules, execution logs, and metadata. Architecturally, it’s fully stateless. Every execution cycle ingests data, evaluates rules, and emits events. Scaling horizontally is just a matter of spawning more evaluators off a queue. The idea isn’t new - but applying it to enterprise operations at scale turns “scripting chaos” into declarative infrastructure. Once logic is externalised, you can version it, test it, and see it - something traditional background jobs rarely offer.

A few months ago, we shared how IBM Data Replication Kafka KCOPs bring 💪 flexibility and ⚡ performance to your Kafka data replication. 👉  https://lnkd.in/gFafPuEn Today’s blog builds on that story — focusing on how to test your KCOPs efficiently using the 🧩 KCOP Test Harness. It’s a powerful time-saver many teams overlook. ✅ Test KCOP logic locally ✅ Debug without a full CDC setup ✅ Iterate faster and with more confidence If you’re working with KCOPs, this quick read will make your workflow smoother 👇 https://lnkd.in/g3vkX3b2 Majed Abed Moosa Shawn Robertson Yong Li

We’re getting ready to launch our data pipeline platform varCHAR - Dev Edition and are currently looking to raise funding in the range of $100K to $2M USD. The investment will mainly go toward scaling infrastructure and building out our core engineering team. What makes varCHAR different • 80–90% more cost-efficient than major ETL / data engineering platforms • AI-driven pipeline optimization, debugging and data quality insights, with natural language assistance • Unified platform for databases, APIs/microservices and file pipelines — all with drag-and-drop visual designers - Seamless Db to Kafka to APIs to Cloud Integrations. Monitoring, Scheduling, Pipeline Sequence Orchestration with mix of all types (Db, APIs, File) along with Data Quality gates. • Developer-friendly UI, built for real production use, with CLI Integration • Powered by Apache Spark, Apache Camel and Kafka — proven enterprise-grade stack • Built-in Kafka ecosystem, from creating Topics to Pub-Sub-DLQ-Consumer-Lag also has Kafka Connect Integration with 15+ connectors support • Native ML integration + SQL & Kafka SQL editors, fully connected to user data sources • Platform has been systematically engineered for the past 12 months — not a weekend hack • Customizable — we offer dedicated support to tailor workflows to a customer’s environment Developer Plans - Free, Basic $29, Pro $129, Max $299 USD. Drop a message for Pitch Deck If this excites you #funding #startup #vc #angelinvestors #preseed #earlystagestartup #investors #data #ai #etl #dataengineering

❓Are your data pipelines truly unit-tested or just data-validated? Most teams write validation checks on tables, but skip unit tests that protect pipeline logic itself. Here’s how strong Data & AI teams approach it: ✅ Unit test your code, not just data. Airflow tasks, AWS Lambdas, dbt macros, and Dataflow transforms. ⚙️ Mock external dependencies like S3, Pub/Sub, or APIs to isolate logic. 🔁 Automate with PyTest + CI/CD for every commit or DAG change. 🧩 Keep data validation separate (nulls, schema drift, late events) using frameworks like Great Expectations or dbt tests. 📊 Extend later into integration and regression tests once unit coverage is solid. Building this foundation ensures your pipelines fail fast and clearly, not silently in production. #DataEngineering #AIEngineering #UnitTesting #DataQuality #ETL #DataPipelines #MLOps #SoftwareQuality #BestPractices