Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
17 likes45,488 views
The document discusses architecture patterns for deploying Apache Kafka in multi-cluster, hybrid, edge, and global environments. It emphasizes the necessity of multiple clusters for scenarios involving high availability and disaster recovery, detailing various deployment models and their requirements. Key takeaways include the importance of understanding trade-offs for achieving zero data loss and the operational aspects of managing multi-cluster environments.
![12
www.kai-waehner.de | @KaiWaehner
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)](https://image.slidesharecdn.com/2020devnexusglobalkafkakaiwaehner-200126162700/85/Architecture-patterns-for-distributed-hybrid-edge-and-global-Apache-Kafka-deployments-12-320.jpg)
Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
- 1. 1 Kai Waehner | Technology Evangelist, Confluent [email protected] | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
- 2. 2Abstract Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka. Key takeaways: • In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments. • Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs. • Learn about features and limitations of Kafka for multi cluster deployments • Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception. www.kai-waehner.de | @KaiWaehner
- 3. 3 Agenda 1) Definition ‘Kafka Cluster’ 2) One Kafka Cluster 3) Multiple Kafka Clusters 4) Multi-Region / Global Kafka Cluster 5) Infrastructure Options www.kai-waehner.de | @KaiWaehner
- 4. 4 Agenda 1) Definition ‘Kafka Cluster’ 2) One Kafka Cluster 3) Multiple Kafka Clusters 4) Multi-Region / Global Kafka Cluster 5) Infrastructure Options www.kai-waehner.de | @KaiWaehner
- 6. 6 The Beginning of a New Era www.kai-waehner.de | @KaiWaehner https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The first use case: Log Analytics. This is why Kafka was created!
- 7. 7 Event Streaming Platform – The Commit Log www.kai-waehner.de | @KaiWaehner Time P C1 C2 C3
- 8. 8 Event Streaming Platform – A Distributed System www.kai-waehner.de | @KaiWaehner Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 9. 9Apache Kafka (kafka.apache.org) includes Kafka Connect and Kafka Streams www.kai-waehner.de | @KaiWaehner Kafka Streams Your app sinksource KafkaConnect KafkaConnect Kafka Cluster
- 10. 10A Streaming Platform is the Underpinning of an Event-driven Architecture www.kai-waehner.de | @KaiWaehner Microservices DBs SaaS apps Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experiences Streams of real time events Stream processing apps Connectors Connectors Stream processing apps
- 11. 11 Apache Kafka at Scale at Tech Giants www.kai-waehner.de | @KaiWaehner > 7 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
- 12. 12 www.kai-waehner.de | @KaiWaehner Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
- 13. 13 A Kafka Cluster www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect
- 14. 14 A Kafka Cluster www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect “Server Side” of a Kafka Cluster 1 - * Kafka Broker 1 - * ZooKeeper 0 - * Schema Registry 0 - * Kafka Connect 0 - * REST Proxy Security, Ops, Monitoring, … “Client Side” of a Kafka Cluster Kafka Clients (Java, C, C++, Python, Go, JavaScript, …) Kafka Stream Processing Apps (Kafka Streams, ksqlDB) External Producers / Consumers (Oracle, Hadoop, Flink, …)
- 15. 15 Why Multiple Kafka Clusters? www.kai-waehner.de | @KaiWaehner * Not a representative survey J ** Many DCs does NOT necessarily mean more than one Kafka Cluster
- 16. 16 Disaster Recovery – RPO and RTO www.kai-waehner.de | @KaiWaehner RPO = Recovery Point Objective RTO = Recovery Time Objective
- 17. 17 Agenda 1) Definition ‘Kafka Cluster’ 2) One Kafka Cluster 3) Multiple Kafka Clusters 4) Multi-Region / Global Kafka Cluster 5) Infrastructure Options www.kai-waehner.de | @KaiWaehner
- 18. 18 A Kafka Cluster www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect
- 19. 19 A Kafka Cluster for High Availability www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect PROD / Pre-PROD / TEST 3 - * Kafka Broker 3 / 5 / 7 ZooKeeper 2 Schema Registry 2 - * Kafka Connect 2 - * REST Proxy Security, Ops, Monitoring, … DEV / Functional TEST 1 Kafka Broker 1 ZooKeeper 0 - 1 Schema Registry 0 - 1 Kafka Connect 0 - 1 REST Proxy Security, Ops, Monitoring, …
- 20. 20 A Stretched Kafka Cluster over 3DC www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry DC1 DC2 DC3
- 21. 21 A Stretched Kafka Cluster over 3DC www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry DC1 DC2 DC3 High availability (Survives DC outage) Zero data loss and zero downtime Automatic client fail-over Works well in cloud (3 AZs in 1 region) Requires “good” latency (à DCs ”close” to each other) Requires three DCs (Quorum / split brain) Complex to configure and operate
- 22. 22 A Stretched Kafka Cluster over 2DC www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry DC1 DC2 Kafka Broker
- 23. 23 A Stretched Kafka Cluster over 2DC www.kai-waehner.de | @KaiWaehner High availability (Survives DC outage) Zero data loss or zero downtime Automatic client fail-over Stopgap solution for on premise (if only 2 DCs available) à 2.5 DC deployment as workaround Requires “good” latency (à DCs ”close” to each other) Quorum in 2 DCs not possible Complex to configure and operate Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry DC1 DC2 Kafka Broker
- 24. 24 A Single Kafka Cluster www.kai-waehner.de | @KaiWaehner Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Simple setup Works Often used “at the Edge” No high availability
- 25. 25 Agenda 1) Definition ‘Kafka Cluster’ 2) One Kafka Cluster 3) Multiple Kafka Clusters 4) Multi-Region / Global Kafka Cluster 5) Infrastructure Options www.kai-waehner.de | @KaiWaehner
- 26. 26 Independent Kafka Clusters www.kai-waehner.de | @KaiWaehner Total Independence Owned by the project teams, central ops or SaaS Different sizing, security, infrastructure Related projects should run on the same Kafka cluster Independent projects can run on the same Kafka cluster • similar SLAs and requirements • e.g. NOT Instant payment vs. log analytics vs. file transfer vs. video streaming • ACLs / RBAC for fine-grained authentication and authorization • throughput typically no issue (Confluent Cloud processes 1 Gigabyte / sec and more in one cluster) • reduce overhead (operations, hardware, …) Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect
- 27. 27 Hybrid Integration of 2 Kafka Clusters www.kai-waehner.de | @KaiWaehner Hybrid integration On premise and cloud or multi-cloud scenarios (due to technical, business or legal reasons) Uni- or bi-directional Know the best practices or get help Know your SLAs and timelines Choose the right (battle-tested?) tool Works Relatively easy to setup (some tools are complex / not up-to-date / not mature / not documented well) Example: Replicate data from production to analytics cluster Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect MirrorMaker 1 / 2 Confluent Replicator uReplicator (Uber) Mirus (Salesforce) Brooklin (LinkedIn) Custom Replication DC1 DC2 Streaming Replication
- 28. 28 Migration of Kafka Clusters www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect MirrorMaker 1 / 2 Confluent Replicator uReplicator (Uber) Mirus (Salesforce) Brooklin (LinkedIn) Custom Replication DC1 DC2 Streaming Replication Common migration scenarios On premise à Cloud Cloud A à Cloud B Vendor 1 à Vendor 2 Self-Managed à SaaS Migration steps: 1) Create new Kafka cluster 2) Producer / Consumer re-configuration 3) Shutdown of old Cluster Know the best practices or get help Know your SLAs and timelines Choose the right (battle-tested?) tool
- 29. 29 Disaster Recovery with 2 Kafka Clusters www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect MirrorMaker 1 / 2 Confluent Replicator uReplicator (Uber) Mirus (Salesforce) Brooklin (LinkedIn) Custom Replication DC1 DC2 Streaming Replication Know the trade-offs! If Kafka Cluster 1 is down, Kafka Cluster 2 is still live and running Timestamp preservation Offset translation Manual client-failover / custom client code Data loss in case of DC outage (asynchronous replication)
- 30. 30 Disaster Recovery @ JPMorgan www.kai-waehner.de | @KaiWaehner https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
- 31. 31 Aggregation of Kafka Clusters www.kai-waehner.de | @KaiWaehner Local smaller Kafka Clusters in each site for critical real time applications (high SLAs) Central bigger Kafka Cluster for analytics use cases (often less critical SLAs) Works Relatively easy to setup (some tools are complex / not up-to-date / not mature / not documented well) Some tools do not support same topic name in each DC Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Kafka Connect Kafka Connect Kafka Broker Kafka Broker Kafka Broker
- 32. 32 Aggregation Cluster @ Royal Caribbean www.kai-waehner.de | @KaiWaehner https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
- 33. 33 Aggregation of Edge Kafka Clusters www.kai-waehner.de | @KaiWaehner Small Kafka clusters in each site for data collection (often low SLAs, sometimes single Kafka broker) Kafka at the edge sometimes OEM / hardware appliance Central big Kafka cluster for critical use cases and edge integration (high SLAs) Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Kafka Broker Schema Registry OPC-UA MQTT PLC4X KSQL Grafana Postgres Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Kafka Connect Kafka Connect Kafka Broker Kafka Broker Kafka Broker
- 34. 34 Real Time Streaming ML at the Edge @ Severstal www.kai-waehner.de | @KaiWaehner https://www.confluent.io/customers/severstal/
- 35. 35 Cross-Company Kafka Integration (Special Case of Hybrid Integration) www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect MirrorMaker 1 / 2 Confluent Replicator uReplicator (Uber) Mirus (Salesforce) Brooklin (LinkedIn) Custom Replication Company A Company B Streaming Replication Streaming integration between companies API Management (REST et al) not appropriate for streaming data Infosec and politics are your biggest enemy
- 36. 36 Agenda 1) Definition ‘Kafka Cluster’ 2) One Kafka Cluster 3) Multiple Kafka Clusters 4) Multi-Region / Global Kafka Cluster 5) Infrastructure Options www.kai-waehner.de | @KaiWaehner
- 37. 37 A Stretched Kafka Cluster over 3 Regions? www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry US-East US-Central US-West No! Fail! Error! Sorry!
- 38. 38Replication Between Kafka Clusters over Multiple Regions or Continents www.kai-waehner.de | @KaiWaehner Streaming replication works (MirrorMaker 2, Confluent Replicator) Same challenges as in one region (data loss, custom code for fail-over, offset translation, etc.) Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Schema Registry Schema Registry Producer Consumer Kafka Connect Kafka Connect China USA Europe
- 39. 39 A Single Kafka Cluster over 3 Regions with Multi-Region Replication www.kai-waehner.de | @KaiWaehner Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry US-East US-Central US-West
- 40. 40A Single Multi Region Kafka Cluster (MRC) www.kai-waehner.de | @KaiWaehner High availability (Survives region outage) Zero data loss and zero downtime Automatic client fail-over over regions Works well in cloud and on premise No external tools (like MirrorMaker) needed Not part of Open Source Kafka à Build vs. Buy Zookeeper Zookeeper Zookeeper Kafka Broker Kafka Broker Kafka Broker Kafka Connect Schema Registry Producer Consumer Producer Consumer Kafka Connect Schema Registry US-East US-Central US-West How does this work? Region-awareness Synchronous or asynchronous replication per Topic Follower-fetching Regional topic locality Replication rules … (Confluent Platform)
- 41. 41A Single Multi Region Kafka Cluster (MRC) www.kai-waehner.de | @KaiWaehner Broker 1 Broker 2 Broker 3 ZK1 Broker 4 Broker 5 Broker 6 Broker 1 Broker 2 ZK2 Client D Client F Client G Failover site ZK3 Broker 3 Broker 4 Broker 5 Broker 6 Client A Client B us-central-1 Client A Client B automated client failover Observer replicas us-west-1 us-east-1 Site failure! “tie-breaker” datacenter Single Kafka Cluster (Confluent Platform) $ bin/kafka-topics.sh --bootstrap-servers localhost:2181 --create --topic trades-west --partitions 3 --config replication-factor={us-west: 2} --config min.insync.replicas=2 --config async.replication-factor={us-east: 2} --config max.async.time.behind.min=5 --config replay.truncated.messages=true
- 42. 42 Vision: One Global Kafka Cluster www.kai-waehner.de | @KaiWaehner Topic‘pos_payments’
- 43. 43 ZooKeeper Removal (KIP-500) www.kai-waehner.de | @KaiWaehner https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
- 44. 44 Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments… www.kai-waehner.de | @KaiWaehner à There is no best solution. It depends!
- 45. 45 Agenda 1) Definition ‘Kafka Cluster’ 2) One Kafka Cluster 3) Multiple Kafka Clusters 4) Multi-Region / Global Kafka Cluster 5) Infrastructure Options www.kai-waehner.de | @KaiWaehner
- 46. 46 Infrastructure Options www.kai-waehner.de | @KaiWaehner Infrastructure is your choice! à Bare metal vs. VM vs. container vs. cloud… Software is your choice! à Open source vs. commercial vs. SaaS Ops and management are your choice! à Self-Managed vs. PaaS vs. fully-managed Integration is your choice! à Kafka-native vs. other tools / services Find the right solution for your business case and for your SLAs…
- 47. 4747 CONFLUENT PLATFORM EFFICIENT OPERATIONS AT SCALE PRODUCTION-STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY Multi-language Development Rich Pre-built Ecosystem SQL-based Stream Processing GUI-driven Mgmt & Monitoring Flexible DevOps Automation Dynamic Performance & Elasticity Enterprise-grade Security Data Compatibility Global Availability APACHE KAFKA Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-LED EXPERTISE PartnersTraining Professional Services Enterprise Support DEVELOPER OPERATOR ARCHITECT Hybrid Infrastructure
- 48. 48 Kafka as a Service – Fully Managed? Infrastructure management (commodity) Scaling ● Upgrades (latest stable version of Kafka) ● Patching ● Maintenance ● Sizing (retention, latency, throughput, storage, etc.) ● Data balancing for optimal performance ● Performance tuning for real-time and latency requirements ● Fixing Kafka bugs ● Uptime monitoring and proactive remediation of issues ● Recovery support from data corruption ● Scaling the cluster as needed ● Data balancing the cluster as nodes are added ● Support for any Kafka issue with less than X minutes response time Infra-as-a-Service Harness full power of Kafka Kafka-specific management Platform-as-a-Service Evolve as you need Future-proof Mission-critical reliability Most Kafka-as-a-Service offerings are partially-managed Kafka as a Service should be a serverless experience with consumption-based pricing!
- 49. 4949 I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model 49 Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi-Region) Build Contextual Event-Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...
- 50. 50End-to-End Integration with 24/7 Uptime and Zero Data Loss? www.kai-waehner.de | @KaiWaehner …. more components, clusters, technologies means more risks, conflicts, incompatibilities, operations burden! ETL MQ Storage Streaming Messaging: Kafka Core Storage: Kafka Core Caching: Kafka Core Real-Time, Batch: Kafka Clients Integration: Kafka Connect Stream Processing: Kafka Streams / KSQL Request-Response: REST Proxy Replication between Kafka Clusters (Edge, On Premises, Hybrid, Cloud) Multi-Region Kafka Cluster ”Eat your own dog food” vs. http://www.kai-waehner.de/blog/2019/03/07/apache-kafka-middleware-mq-etl-esb-comparison/
- 51. 51 Questions? Let’s connect... Kai Waehner Technology Evangelist [email protected] @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn