Tinybird

We isolated resource-intensive populate operations in Tinybird so they never impact production queries. Here's how ↓ 🚨 Problem Before, populate operations (data transformations and backfills) competed with production queries for resources. Heavy populates could impact live dashboards and user-facing applications. Big no-no for our customers. ⚠️ Resource Contention Some customers had complex aggregations consuming gigabytes of memory or intensive transformations saturating CPU cores. These operations have highly variable resource needs, so it's hard to predict when they might affect read operations. 💰 Cost Challenge Our traditional setup meant over-provisioning infrastructure for occasional heavy operations, so we were paying for capacity that our customers only needed sporadically. ✅ Solution Compute-compute separation provisions ephemeral, dedicated ClickHouse instances just for heavy populate operations. This offers complete isolation from production workloads, so a read never fails because of a heavy write job. ⚙️ How It Works Our orchestration layer automatically provisions right-sized instances when a populate starts. It runs the transformation with dedicated resources, then decommissions when complete. 😃 Benefits - No resource contention with production queries - Right-sized, ephemeral provisioning reduces costs - It eliminates the risk of OOM errors or query timeouts 📊 Result We can now handle larger datasets, more complex transformations, and frequent backfills without architectural constraints. We provision only the necessary infrastructure, and nothing more. Read more in the blog post. Link in the comments.

Here's how we scaled our ClickHouse streaming ingestion service when compute-compute separation wasn't enough ↓ 🚨 Problem Even dedicated ClickHouse writer instances with compute-compute separation hit limits with extremely high-throughput ingestion. Single-writer becomes bottleneck at petabyte scale, which was creating problems for our streaming ingestion service. 😫 The Scale Challenge We had customers ingesting massive data volumes during traffic spikes. Best practice is to isolate writes to a dedicated replica with compute-compute separation, but what happens when even that isn't enough? ✅ Solution Multi-writer mode distributes ingestion across multiple ClickHouse instances within the same cluster. We manually route traffic by data source or workspace to different writers, which is simple and efficient. 🔧 Technical Approach We preferred static routing over dynamic for reliability and simplicity. We configure routing rules at the data source/workspace level for explicit control. Predictable, testable, safe. ⚙️ Varnish + Custom VMOD We built a custom Varnish extension with a `backend_by_index()` function. It routes to "Nth healthy instance" instead of a specific replica. Automatic failover if a writer goes down. 📈 Real Impact We had a customer hitting memory limits daily during peak ingestion. Multi-writer mode moved one workspace to a different replica and eliminated the bottleneck. 💡 Key Insight We prioritized simplicity and operational safety over complex load balancing. Static routing with intelligent failover beats unpredictable dynamic routing. Read the full implementation details in the blog post. Link in comments.

Here's a step-by-step guide on building an analytics agent with Agno and Tinybird. This agent can: - Explore the data in your Tinybird workspace - Generate and run SQL queries - Fetch your API endpoints (with parameters) - Send you a notification via Slack or email - Instantiate sub-agents with its findings as a prompt Check it out ↓ https://tbrd.co/agno-agent

The engineers at Framer hit Redshift's limits building real-time, user-facing analytics. ClickHouse was the right tech, but they had a choice: - Self-host - Use ClickHouse Cloud - Try Tinybird Speed wins. They chose the bird. One engineer → less than 3 months to production → 60X faster analytics handling 1M+ requests/day Bonus: They finally shipped the new revenue-generating features they had put off for so long. Read how it all went down. Link in the comments ↓

We cut CPU utilization for streaming ingestion into ClickHouse by 25% or more. Here's how ↓ 🚨 Problem JSON to RowBinary conversion was in the critical path of ingestion performance. Our Python implementation was efficient but hard to optimize further and difficult to maintain. ☀️ The Opportunity We're ingesting petabytes of JSON daily, and it all needs to be converted to ClickHouse RowBinary. Every performance improvement in this path directly impacts platform capacity. 📋 Technical Requirements - Support JSONPaths for complex JSON → table mapping - Handle quarantine for invalid data - Maintain backward compatibility (no existing customer impact) ✅ Solution We rewrote our JSON to RowBinary conversion in C++ using ClickHouse internals + simdjson parser. We delegate encoding to the database where it belongs and the use fastest JSON parser available (plus a few other tricks) ⚠️ Key Challenges - simdjson's on-demand API is fast but unforgiving (ephemeral values, single iteration) - We had to write a custom JSONPath and date/time parsers - We still needed a legacy conversion mode to handle existing quirks 🧪 Complex Testing We used a three-stage validation: - Internal data comparison (old vs new conversion) - Controlled production tests with monitoring - AI-generated random test cases for edge cases 📊 Results We reduced CPU utilization by at least 25% and better leveraged ClickHouse internals. We ended up with a faster solution that's easier to maintain, plus a more efficient platform capacity for the same cost. Read the full technical post for more. Link in the comments.

How we cut AWS costs 20% overall (90% on CI!) ↓ 🚨 The Problem: AWS Cluster Autoscaler becomes unmanageable at scale. We 5-6 instance types × 3 AZs × x86 + Graviton × On-Demand + Spot × 5 AWS regions = hundreds of NodeGroups requiring manual management and updates. Cluster changes took days of operational work. ✅ The Solution: We use Karpenter. Karpenter replaces the NodeGroup complexity with intelligent, on-demand provisioning. You set high-level policies (instance families, avoid old generations, prefer spot instances) and Karpenter handles the rest. 🏗️ Our Architecture: We have two logical NodePools: Critical/Stateful (On-Demand with Savings Plans coverage) and Stateless/Spot (maximum cost savings). Both support x86 and ARM architectures for optimal flexibility. KEDA integration uses Tinybird metrics for horizontal scaling based on actual workload signals. 💡 Key Insights: We don't restrict Karpenter to small instance type sets (more options = better optimization). We use Disruption Budgets for critical time periods. We Set 24-48h node expiration for automated security updates. 📊 Results: 20% AWS cost reduction overall, 90% savings on CI workloads. What used to take days of manual work now happens automatically with zero operational overhead. Read the complete technical setup in our blog post. Link is in the comments.

We ditched Prometheus for autoscaling, and we don't miss it. More ↓ 🚨 The Problem: Traditional CPU/memory autoscaling fails for real-time analytics workloads. By the time CPU spikes, your queues are already backed up. Prometheus adds scraping delays when you need scaling most. ⚡ Our Challenge: Our Kafka service ingests terabytes daily, and we can have 10x traffic spikes during customer launches or other notable events. CPU-based scaling was too slow, memory-based too vague. We needed real signals, not lagging resource metrics. ✅ The Solution: We use KEDA and swapped out Prometheus for a single Tinybird API. Instead of the traditional multi-hop delay (Prometheus scraping → aggregation → federation → KEDA query), we go direct: KEDA polls live metrics computed fresh from streaming data, directly in Tinybird. Dog food never tasted so good. 🔧 Technical Implementation: We built a Tinybird pipe that exposes Prometheus-format endpoints with metrics like Kafka lag. KEDA's pulls these directly with zero scraping lag. Metrics are computed on-demand when KEDA requests them - always fresh data driving scaling decisions 💡 The Key Insight: Traditional approaches have multi-hop delays at every step. Our approach eliminates the middle layers entirely. Metrics are computed from the same streaming data we already trust for analytics. 📊 Results: Faster scaling, simpler infrastructure, and autoscaling based on real workload signals. No separate monitoring stack to run and maintain. This is what happens when you use your own platform to solve your own scaling challenges. 💪 Read the full technical implementation in our blog post. You'll find it in the comments.

Speed is a feature. Today, we go behind the scenes with 6 infrastructure and performance improvements our team shipped to make working with data faster and more reliable. Tinybird engineers solve the hard problems so you can go fast without worrying about scaling. Here's what we shipped ↓ 📈 KEDA + Tinybird Autoscaling We ditched Prometheus for autoscaling, and we don't miss it. Instead of Prometheus, we use a real-time metrics from a Tinybird API + KEDA for faster, simpler autoscaling. Now we avoid all the service hops and scraping delays, with no extra monitoring stack to manage. Read more -> tbrd.co/keda 🪚 EKS + Karpenter Optimization We cut AWS costs by 20% overall (90% on CI!) with intelligent node provisioning using Karpenter. We eliminated hundreds of NodeGroups that we used to manage manually. Read more -> tbrd.co/karpenter ☯️ Compute-Compute Separation We isolated resource-intensive populate operations so they never impact your production queries, regardless of data volume. Read more -> tbrd.co/csquared ⫚ Multi-Writer Ingestion We built a distributed, multi-writer ingestion service. Even with compute-compute separation, single writers can hit capacity limits at petabyte scale. Now, we smartly and safely distribute the load across multiple writers. Read more -> tbrd.co/multiwriter 🚣 25% less CPU usage for ClickHouse streaming ingestion We rewrote our JSON to ClickHouse RowBinary conversion in C++ using ClickHouse internals plus some of our own magic for a big performance improvement in the critical ingestion path. We learned a ton. Read more -> tbrd.co/rowbinary 🧠 Making automatic ClickHouse schema migrations safer and smaller Tinybird handles all of the details of schema migrations in ClickHouse. Our prior approach was a bit brute force. It worked, but it was inefficient. We optimized the deployment algorithm to avoid migrating terabytes of unnecessary data. What used to take days now takes minutes. Read more -> tbrd.co/smartdeploy --- Each improvement emerged from real production challenges at scale. The infrastructure complexity we solve is the complexity you don't have to build. Speed wins.

Speed is a feature. Today, we go behind the scenes with 6 infrastructure and performance improvements our team shipped to make working with data faster and more reliable. Tinybird engineers solve the hard problems so you can go fast without worrying about scaling. Here's what we shipped ↓ 📈 KEDA + Tinybird Autoscaling We ditched Prometheus for autoscaling, and we don't miss it. Instead of Prometheus, we use a real-time metrics from a Tinybird API + KEDA for faster, simpler autoscaling. Now we avoid all the service hops and scraping delays, with no extra monitoring stack to manage. Read more -> tbrd.co/keda 🪚 EKS + Karpenter Optimization We cut AWS costs by 20% overall (90% on CI!) with intelligent node provisioning using Karpenter. We eliminated hundreds of NodeGroups that we used to manage manually. Read more -> tbrd.co/karpenter ☯️ Compute-Compute Separation We isolated resource-intensive populate operations so they never impact your production queries, regardless of data volume. Read more -> tbrd.co/csquared ⫚ Multi-Writer Ingestion We built a distributed, multi-writer ingestion service. Even with compute-compute separation, single writers can hit capacity limits at petabyte scale. Now, we smartly and safely distribute the load across multiple writers. Read more -> tbrd.co/multiwriter 🚣 25% less CPU usage for ClickHouse streaming ingestion We rewrote our JSON to ClickHouse RowBinary conversion in C++ using ClickHouse internals plus some of our own magic for a big performance improvement in the critical ingestion path. We learned a ton. Read more -> tbrd.co/rowbinary 🧠 Making automatic ClickHouse schema migrations safer and smaller Tinybird handles all of the details of schema migrations in ClickHouse. Our prior approach was a bit brute force. It worked, but it was inefficient. We optimized the deployment algorithm to avoid migrating terabytes of unnecessary data. What used to take days now takes minutes. Read more -> tbrd.co/smartdeploy --- Each improvement emerged from real production challenges at scale. The infrastructure complexity we solve is the complexity you don't have to build. Speed wins.

We just launched Sinks in Forward, and as with most things Tinybird, the simple UX hides the complex engineering problems our team had to solve. Some examples ↓ 1. Cross-Cloud Billing Optimization One of the most sophisticated features of sinks: We automatically detect actual cloud storage regions and override user configuration to avoid inter-cloud transfer and minimize data transfer costs. 2. Dynamic File Templating with SQL Introspection The templating system validates file path templates against actual query schemas. 3. Plan-Based Intelligent Rate Limiting We make cron recommendations based on your plan limits. 4. Format-Specific Performance Optimizations Different export formats (CSV, NDJSON, Parquet) trigger specific ClickHouse-level optimizations.