How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Rosanova, Confluent) Kafka Summit 2020
0 likes875 views
Confluent Platform 6.0 and Project Metamorphosis complete the event streaming platform by providing elastic scalability, infinite storage, global access, and transforming Kafka. Key features include self-balancing clusters and dynamic scaling on Confluent Cloud, tiered storage and infinite retention on the platform, and cluster linking to simplify hybrid and multi-cloud deployments. These new capabilities help remove limitations on scale, storage, and deployment that traditionally challenged Kafka applications.
![Confluent Cloud
Milliseconds Minutes
Basic, Standard [0-100Mbps]
Do Nothing
Elastic Scaling w/ fully-managed Confluent Cloud
*Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already.
Dedicated [Mbps - Gbps]
1 Click—Select CKU from drop
down in cluster management UI and
click Apply Changes
Other Kafka Services
Days - Weeks
Determine how much capacity is needed
Procure capacity*
Configure new brokers
a. Disks b. OS c. Network d. Kafka (application)
Identify partitions on specific brokers to
rebalance & topics they are part of
For each Topic: migrate partitions
a. Increase ISR +1 b. Wait for new replica to sync
c. Failover master d. Reduce ISR -1 e. Delete old replica](https://image.slidesharecdn.com/s061addisonhuddydanrosanovahowconfluentcompletestheeventstreamingplatform-200910020228/85/How-Confluent-Completes-the-Event-Streaming-Platform-Addison-Huddy-Dan-Rosanova-Confluent-Kafka-Summit-2020-6-320.jpg)
How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Rosanova, Confluent) Kafka Summit 2020
- 1. How Confluent Completes the Event Streaming Platform With Confluent Platform 6.0 and Project Metamorphosis @DanRosanova @AddisonHuddy
- 2. As our roadmap may change in the future, the features referred to herein may change, may not be delivered on time or may not be delivered at all. This information is not a commitment to deliver any functionality and customers should make their purchasing decisions based upon features that are currently available.
- 3. Hadoop ... Device Logs ... App ...MicroserviceMainframes Data Warehouse Splunk ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Real-time Inventory Real-time Fraud Detection Real-time Customer 360 Machine Learning Models Real-time Data Transformation ... Contextual Event-Driven Applications Universal Event Pipeline CONFLUENT
- 4. Elastic Confluent Cloud ● Self-serve clusters ● GBps+ scale Confluent Platform 6.0 ● Self-Balancing Clusters Cost-effective Confluent Cloud ● 99.5% Basic SLA ● New connectors Confluent Platform 6.0 ● Admin REST API Infinite Confluent Cloud ● Infinite retention ● Schema registry Confluent Platform 6.0 ● Tiered Storage ● ksqlDB Transforming Kafka with Confluent Platform 6.0 and Project Metamorphosis Global Confluent Cloud ● Cluster linking ● New cloud regions Confluent Platform 6.0 ● Cluster Linking (preview)
- 6. Confluent Cloud Milliseconds Minutes Basic, Standard [0-100Mbps] Do Nothing Elastic Scaling w/ fully-managed Confluent Cloud *Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already. Dedicated [Mbps - Gbps] 1 Click—Select CKU from drop down in cluster management UI and click Apply Changes Other Kafka Services Days - Weeks Determine how much capacity is needed Procure capacity* Configure new brokers a. Disks b. OS c. Network d. Kafka (application) Identify partitions on specific brokers to rebalance & topics they are part of For each Topic: migrate partitions a. Increase ISR +1 b. Wait for new replica to sync c. Failover master d. Reduce ISR -1 e. Delete old replica
- 7. Dynamic Scaling with Confluent Platform T3, P0 T1, P0 T2, P1 T3, P0 T1, P1 T2, P0 T3, P1 Broker 1 Broker 2 New: Broker 3 T1, P0 T2, P1T1, P1 T2, P0T3, P1 Broker 1 Broker 2 Broker 3 Scaling with OSS Kafka T3, P0T1, P0 T2, P1 Broker 1 Broker 2 New: Broker 3 Object Store Broker 1 Broker 2 Broker 3 Object Store T3, P1T1, P1 T2, P0 T3, P0T1, P0 T2, P1T3, P1 T1, P1 T2, P0 Scaling with Tiered Storage + Self- balancing Clusters on Kubernetes Rebalance in Hours... Rebalance in Seconds...
- 9. 9 Historically, the way to scale storage in Kafka is to scale brokers. Broker storage can be expensive and is tightly bound to compute: Brokers
- 10. Confluent Cloud Infinite Retention Scale up storage without adding brokers by removing storage limits on topics to enable infinite data retention in Kafka Cluster Broker 1 Broker 2 Broker 2 Topic 1 Topic 2 Topic 3 Topic 2 Topic 3 Topic 2 Topic 3 Topic 1Topic 1
- 11. Confluent Platform Tiered Storage Offload old events to cheaper, 3rd party object storage to reduce costs and enable infinite data retention in Kafka 11 Tiered Storage enables Kafka to recognize two layers of storage: Brokers Object Storage Offload old data to object store
- 12. Global Access
- 13. 13 Cluster Linking (preview) Cluster Linking simplifies hybrid-cloud and multi-cloud deployments for Kafka Connecting independent clusters or migrating clusters presents major challenges: 1. Requires deploying a separate Connect cluster, adding architectural complexity and operational burden 1. Offsets are NOT preserved, so consumers are at risk of skipping or rereading messages 1 2
- 14. 14 Cluster Linking (preview) Cluster Linking simplifies hybrid-cloud and multi-cloud deployments for Kafka Cluster Linking requires no additional infrastructure and preserves offsets:
- 16. Confluent Developer developer.confluent.io Learn Kafka. Start building with Apache Kafka at Confluent Developer.