Isolating resource-intensive operations in Tinybird

We isolated resource-intensive populate operations in Tinybird so they never impact production queries. Here's how ↓ 🚨 Problem Before, populate operations (data transformations and backfills) competed with production queries for resources. Heavy populates could impact live dashboards and user-facing applications. Big no-no for our customers. ⚠️ Resource Contention Some customers had complex aggregations consuming gigabytes of memory or intensive transformations saturating CPU cores. These operations have highly variable resource needs, so it's hard to predict when they might affect read operations. 💰 Cost Challenge Our traditional setup meant over-provisioning infrastructure for occasional heavy operations, so we were paying for capacity that our customers only needed sporadically. ✅ Solution Compute-compute separation provisions ephemeral, dedicated ClickHouse instances just for heavy populate operations. This offers complete isolation from production workloads, so a read never fails because of a heavy write job. ⚙️ How It Works Our orchestration layer automatically provisions right-sized instances when a populate starts. It runs the transformation with dedicated resources, then decommissions when complete. 😃 Benefits - No resource contention with production queries - Right-sized, ephemeral provisioning reduces costs - It eliminates the risk of OOM errors or query timeouts 📊 Result We can now handle larger datasets, more complex transformations, and frequent backfills without architectural constraints. We provision only the necessary infrastructure, and nothing more. Read more in the blog post. Link in the comments.