Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subramanian and Jennifer Snipes | Kafka Summit London 2022
0 likes1,055 views
The document discusses disaster recovery options for running Apache Kafka on Kubernetes, emphasizing resilient architectures such as active-passive and active-active setups. Key topics include recovery time objectives (RTO), recovery point objectives (RPO), and best practices for configuring and managing Kafka clusters in a multi-region environment. It also covers the use of Kubernetes operators, chaos testing, and considerations for ensuring high availability and performance.
![Networking between Kubernetes Clusters
14
stubDomains:
{ "west.svc.cluster.local":
["34.83.255.165"],
"central.svc.cluster.local":
["34.69.152.240"] }](https://image.slidesharecdn.com/bs10jennifersnipesconfluent-220503173050/85/Disaster-Recovery-Options-Running-Apache-Kafka-in-Kubernetes-with-Rema-Subramanian-and-Jennifer-Snipes-Kafka-Summit-London-2022-14-320.jpg)
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subramanian and Jennifer Snipes | Kafka Summit London 2022
- 1. Disaster Recovery Options running Apache Kafka On Kubernetes Rema Subramanian Customer Success Technical Architect Jennifer Snipes Staff Customer Success Technical Architect
- 2. Contents 2 1. Resilient Kafka Architectures 2. Kafka & Kubernetes 3. Getting Started 4. Kubernetes Operator 5. Stretch Cluster on Kubernetes 6. Putting it to the Test 7. Wrapping it Up 8. Demo / Q & A
- 4. Know your RTO & RPO RPO RPO (Recovery Point Objective) is about how much data you can afford to lose before it impacts business operations. For example, for a banking system, 1 hour of data loss can be catastrophic as they operate live transactions. 4 RTO RTO (Recovery Time Objective) is the timeframe within which an application and systems must be restored after an outage.
- 5. Resilient Kafka Architectures 5 Active DC-1 / Passive DC-2 ● Two independent clusters in different Data Centers / Regions ● Producers only in one Data Center ● Consumers in both Data Centers ● Multi-cloud / Multi-region ● One way Replication ● Asynchronous Replication ● RPO >0, RTO >0
- 6. Resilient Kafka Architectures 6 Active DC-1 / Active DC-2 ● Two independent clusters in different Data Centers / Regions ● Producers in both Data Centers ● Consumers in both Data Centers ● Multi-region / Multi-cloud ● Bi-Directional Replication with Provenance Headers ● Asynchronous Replication ● RPO >0, RTO >0
- 7. Resilient Kafka Architectures 7 Stretch Cluster ● Single Cluster stretched across different Data Centers ● Producers write transparently across Data Centers ● Consumers in all Data Centers ● RPO = 0, RTO near 0 ● Synchronous Replication native to cluster ● Asynchronous Replication with Observers* ● Replica & Observer placement defines Active-Active vs Active-Passive* ● Auto Observer Promotion* ● Multi-region*
- 9. Kafka Deployment Arenas 9 Traditional vs K8s • Broker • Hostnames /IPs • Placement across DCs • Communication across DCs • Failure • Broker and ZK co-location • Multi-tenancy
- 10. Stretch Cluster on Kubernetes 10
- 11. Built-in Disaster Recovery on Kubernetes 11 Kubernetes Operator Kafka Stretch Cluster Chaos Testing / Monitoring
- 12. Getting Started
- 13. Building the K8s Cluster 13 GCP VPC Native Cluster ● Alias IP address range for nodes, pods and services ● Requires non- overlapping CIDR ranges GKE Cluster ● Separately Managed Node Pool ● Node machine type ● Configurable number of nodes distributed across AZs ● Distinct namespace per cluster Networking ● VPC Native cluster installs routing ● Firewall rules ○ allow tcp between k8s clusters ○ allow access to 2181, 2888, 3888, 9092, 7778, 3000 StorageClass ● provisioner: kubernetes.io/gce-pd ● allowVolumeExpansion : true ● type: pd-ssd ● fstype: ext4 ● reclaimPolicy: Retain ● volumeBindingMode: WaitForFirstConsumer 2 3 1 4
- 14. Networking between Kubernetes Clusters 14 stubDomains: { "west.svc.cluster.local": ["34.83.255.165"], "central.svc.cluster.local": ["34.69.152.240"] }
- 16. Operator 16 CRDs ● Define various application components ● Medium to tie to kafka server.properties Controller ● CRD’s behavior ● Reconciliation loop Services ● Headless Service ● Expose individual pods as external services ● Bootstrap LB service to get metadata StatefulSets ● PVC claim 2 3 1 4
- 17. Pod Accessibility 17 Identifying the kafka pods • Unique broker IDs • Internally, each pod resolves kafka-{n}.kafka.east.svc.cluster.local • Externally, broker prefixes to map to pod ordinals {region}-b{n}.{domain}
- 18. Kafka CRD Services Galore 18
- 19. Stretch Cluster on Kubernetes
- 20. 20 East Cluster ● Single Kubernetes Cluster (us-east1-cluster1-gke) ● 3+ Brokers ● Single Zookeeper Central Cluster ● Single Kubernetes Cluster (us-central1-cluster1-gke) ● Single Zookeeper West Cluster ● Single Kubernetes Cluster (us-west1-cluster1-gke) ● 3+ Brokers ● Single Zookeeper Multi Region (Stretch) Cluster
- 23. 23 • Broker Rack Awareness • Synchronous Replicas • Asynchronous Observers* • Observer Promotion Policy* Topic Replica Placement*
- 24. Putting it to the Test
- 25. Testing 25 ● Deleted a pod Pod Kill ● Node VM down ● Auto-scaler off & Node VM down Node Fault ● Introduced pod failures with chaos-mesh Pod Failure ● Edited kube-dns stub-domain Network Fault ● 2 ZK nodes down - disrupt quorum ZK Quorum Failure ● Producers don’t stop ● Controller broker and topic leader broker move to west ● ZK west is accessible for write Region Down
- 28. Region Down 28
- 29. Network fault 29
- 31. Wrapping it Up
- 32. Best Practices - Node to Pod Ratio 32 Pod Pod Nodes Quantity ● Use eventsizer.io to derive number of broker pods ● Adjust count to balance across AZs ● Each AZ is a rack Capacity ● Eventsizer.io output will derive CPU and memory per pod ● Set CPU and memory limits and requests Nodes ● Memory optimized node type ● Evaluate capacity of node based on how average load ● Enable auto scaling with average and peak range
- 33. Best Practices 33 ● Use Confluent for Kubernetes ● Choose the right storage that guarantees reliability, efficiency and speed e.g. SSD ○ Refer to ‘Building the K8s Cluster’ slide for other storage best practices ● Run health/liveness checks on: ○ Individual pod and bootstrap LBs ○ kube-system LBs ○ kube-dns service ● Use node affinity/pod anti-affinity to strategically place broker and ZK pods across AZs ● Use automation so infrastructure and CRDs can be deployed multi-region across all of your environments ● Follow best practices for tuning tcp socket buffers, replica fetcher, and clients for optimal performance with stretch clusters ● Minimum Durability Configuration ○ 2 Replicas and 2 Observers in each region ○ min.ISR=3 ● Monitor everything!
- 34. 34 ● Requires separate IP CIDR ranges ● CoreDNS exposed externally ● Restricted to single K8s implementation ● CRDs may get stuck if the finalizer logic in Operator is not finishing ● Manually restart stateful set if pods are erroring - known issue ● GCP VPCs are global, subnets are regional
- 35. Achieving your Desired Resilience 35 Active-Passive Active-Active ● RTO > 0, RPO > 0 ● Replicas in one Region ● Observers in another Region ● Under-replicated AOP ● RTO ~ 0, RPO = 0 ● 2 Replicas in each region ● 2 Observers in each region ● Under-replicated AOP
- 36. Demo and Q&A
- 37. References 37 1. https://assets.confluent.io/m/69c5ce7aff462f44/original/20180619-WP- Recommendations_for_Deploying_Apache_Kafka_on_Kubernetes.pdf 2. https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips 3. https://learn.hashicorp.com/tutorials/terraform/gke 4. https://chaos-mesh.org/docs/ 5. https://docs.confluent.io/operator/current/overview.html 6. https://docs.confluent.io/platform/current/multi-dc-deployments/multi-region.html 7. https://www.confluent.io/en-gb/events/kafka-summit-americas-2021/a-tale-of-2-n-data-centers- tuning-apache-kafka-clusters-to-combat-latency/
- 38. Your Apache Kafka® journey begins here developer.confluent.io 38