Beyond Tiered Storage: Serverless Kafka with No Local Disks
0 likes1,502 views
Warpstream presents a cloud-native, serverless Kafka solution that eliminates local disks to significantly reduce costs and complexity, particularly for high-volume workloads. By optimizing for cloud unit economics, Warpstream can achieve an estimated 85% cost reduction compared to self-hosted Kafka deployments while maintaining real-time performance. The approach utilizes S3 for data storage, separates data from metadata, and supports advanced features like compaction without the inefficiencies associated with tiered storage models.
Beyond Tiered Storage: Serverless Kafka with No Local Disks
- 1. WarpStream Beyond Tiered Storage: Serverless Kafka with No Local Disks
- 2. ● Cloud disks are expensive. ● Long retention workloads TCO can be 80% disk cost even at low throughput ● EBS vs instance storage doesn’t matter ○ double/triple replication expensive compared to S3 2 EBS (GP2) $0.1/GiB pre-replication EBS (GP2) $0.3/GiB post-replication S3 $0.02/GiB post-replication Why Tiered Storage? Retention
- 3. Tiered storage helps, but not enough
- 4. ● Stateful brokers with attached storage make operations complex, difficult, and inelastic ● Requires consensus, topic-partition leaders, custom operations for scaling in/out and doing node replacements ● Balancing in general always a problem 4 Why Drop the Disks? Operations
- 5. ● 80%+ of TCO for high throughput Apache Kafka clusters can be networking fees ● $0.053 / compressed GiB transferred in 100% ideal conditions with fetch-from-follower enabled 5 Why Drop the Disks? Networking
- 6. Zero disks would be better
- 7. WarpStream’s Cloud-Native design 7 greatly reduces the cost and complexity of Apache Kafka
- 8. How It Works
- 9. ● Entire storage engine redesigned around minimizing PUT / GET operations 9 ● S3 PUT: $0.000005 ● S3 GET: $0.0000004 Optimize for Cloud Unit Economics
- 10. ● Entire storage engine redesigned around minimizing PUT / GET operations ● Networking is free, and storage is cheap. 10 Optimize for Cloud Unit Economics ● S3 PUT: $0.000005 ● S3 GET: $0.0000004 ● S3 Storage: $0.023/GB-mo ● S3 Cross-AZ Networking: Free
- 11. Step 1: Eliminate per-TopicPartition Files 11 �� �� ��
- 12. Step 1: Eliminate per-TopicPartition Files 12
- 13. Step 1: Eliminate per-TopicPartition Files 13
- 14. Step 2: Separate Data from Metadata 14 No per-TopicPartition leaders!
- 15. 15 Step 3: Introduce Data Locality for Live Reads
- 16. Step 4: Introduce Data Locality for Historical Reads 16
- 17. 17 Step 4: Introduce Data Locality for Historical Reads
- 18. 18 Hard Mode: Compacted Topics
- 19. 19 Hard Mode: Compacted Topics ● Tiered Storage in open source Apache Kafka does not support compacted topics ● WarpStream already does compaction internally ● … how hard could it possibly be?
- 20. 20 Hard Mode: Compacted Topics
- 21. 21 Hard Mode: Compacted Topics
- 22. 22 Hard Mode: Compacted Topics
- 23. 23 Hard Mode: Compacted Topics
- 24. 24 Hard Mode: Compacted Topics
- 25. 25 Hard Mode: Compacted Topics
- 26. WarpStream costs ~85% less than self-hosted Kafka for high volume workloads 26 Deployment Model Workload Profile Hardware Network Object Storage Total Costs WarpStream Avg. ingress: 1 GiB/sec Avg. egress: 3 GiB/sec Retention: 1 day Replication Factor = 3 3 Availability Zones $223k/year $<2k/year $61k/year $286k/year Self Hosted Apache Kafka $223k/year $1.68M/year $0 $1.9M/year
- 27. WarpStream Is Still Real Time 27 ● P99 producer latency of ~400ms ● Producer to Consumer – End to End Latency <1.5s
- 28. WarpStream Supports S3 Express One Zone 28 ● P99 producer latency as low as 150ms ● Uses a majority quorum of 3 buckets to provide regional high-availability ● Data is moved to S3 Standard asynchronously to reduce storage costs
- 29. 29