Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes ( Kamala Dasika, Pivotal and Michael Ng, Confluent) Kafka Summit London 2019
1 like513 views
The document discusses modern cloud-native streaming platforms, particularly focusing on event streaming microservices using Kafka on Kubernetes. It highlights the shift towards microservices, the importance of real-time data processing, and the benefits of automation for operational efficiency. The Confluent operator is introduced as a solution for managing Kafka on Kubernetes, enabling automated provisioning, scaling, and security configuration.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes ( Kamala Dasika, Pivotal and Michael Ng, Confluent) Kafka Summit London 2019
- 1. Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes Kamala Dasika Michael Ng Victor Gamov @DasikaKN @Micha8LNg @gAmUssA
- 2. Changing Model for Application Delivery Development Deployment Sparingly at designated times Ready for prod at any time Architecture Abstraction Layer “Day 2” Ops App Server on Machine App on “disposable” infrastructure App and Data Monolith Microservices / Composite app Linear / Sequential Short cycles, test driven, iterative Many tools, ad hoc automation Manage services, not servers
- 3. Driven by Modern Cloud Native Platform Cont inuous Del ivery DEV OPS Customer Feedback Customer Delivery An idea in the morning can ship by evening Microservices Release #1 Microservices Release #2
- 5. Hardware IaaS CaaS PaaS FaaS HPE, Dell, IBM, Lenovo Cloud providers: AWS, GCP, Microsoft Azure, VMware Kubernetes Cloud Foundry, Heroku AWS Lambda, Azure Functions, OpenWhisk, kubeless, PFS
- 6. Hardware IaaS CaaS PaaS FaaS • Data warehouses • Storage with regional implications • Hypervisors • Relational DBs • Packaged software • Virtual appliances • Application platforms (cloud abstraction) • Containerized Monoliths • Stateful workloads • Databases • Messaging • 12 -15 factor apps • stateful and stateless options • APIs • Streaming apps • DB copies • Change synchronization • Real time stream analytics
- 7. Hardware IaaS CaaS PaaS FaaS Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency
- 8. What’s best for my app?
- 9. Goal: Operational consistency AND multiple abstractions FREEDOM: Run Your App on Any Cloud AUTOMATION: Lowers Ops Cost and Increases Velocity RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt. GOVERNANCE: Common Developer-Self-Serviced Frameworks VISIBILITY: Full Stack Logging, Monitoring & Auditability SPEED & SCALE: Cut Cycle Time from Months to Days/Hours Built for Cloud Native New Initiatives Existing Workloads Modernize to Cloud-Native?
- 10. Outcomes trump technology bingo Needs High Dev Efficiency Needs Low Dev Efficiency App Rarely changes App Frequently changes • Traditional apps • Leave as-is • Re-architect • Replatform • CI/CD • Evaluate • Evaluate
- 11. Evaluating the Enterprise App Portfolio Business Technical Economic Score • Business criticality • Risk tolerance • Change frequency • Lifecycle stage • Domain expert availability • Licensing costs (HW / SW) • Time-to-market implications • Revenue impact-fullness • Codebase • Suitable framework / runtime • Relatively lightweight footprint • No “hard wired” file system dependencies • Usage / workload For each app
- 12. Application Transformation Strategies Re-Host Containerize aka “lift and shift” to CaaS and gain infrastructure + platform ops benefits Re-Platform Minimal readiness effort to run on PaaS (4 -7 factor apps + data service) that can run on PaaS Re-Factor Convert to cloud- native/microservice (12-15 factor app + data service) runs well on PaaS/FaaS Re-Build Decompose and rewrite from the ground up Application Portfolio Funneling Process Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments
- 13. Evolution of the enterprise application portfolio Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka
- 14. Now what about Data?
- 15. “The world’s most valuable resource is no longer oil, but data” - The Economist, May 6 2017
- 16. Apache Kafka is a Distributed Event Streaming Platform Process streams of events In real time, as they occur 110101 010111 001101 100010 Publish and subscribe to streams of events Similar to a message queue or enterprise messaging system 110101 010111 001101 100010 Store streams of events In a fault tolerant way 110101 010111 001101 100010
- 17. Events A Sale An Invoice A Trade A Customer Experience
- 18. What is a company? A business is a series of events and reacting to those events.
- 19. All Your Data is a Stream of Events
- 20. Events enable Microservice Architectures - Let’s refactor! Submit Order Orders Service Shipping Service Customer Service Webserver E-Commerce Microservices (with RPC) - Orders Service calls Shipping Service to tell it to ship item. - Shipping service looks up address to ship to (from Customer Service) - No Kafka !shipOrder() getCustomer()
- 21. Refactoring Orders and Shipping Message Broker (Kafka) Submit Order Order Created getCustomer() RPC Orders Service Shipping Service Customer Service Webserver - Orders Service no longer knows about the Shipping service (or any other service). - Events are fire and forget.
- 22. Refactoring Customers Customer Updated Submit Order Order Created Webserver KAFKA Orders Service Shipping Service Customer Service - Call to Customer service is gone. - Data is replicated, as events, into the shipping service, where it is queried locally.
- 23. Events are the key to scalable microservices Orders Service
- 25. Why run Kafka on Kubernetes? 33% of Confluent Customers plan to run Production Kafka on Kubernetes
- 26. Microservices - driving container and container orchestration adoption Breaking down the monolith - enables dev teams to work and scale independently µ Containers allow services to be independently deployable services ● Each service has its own purpose & implementation language ● Each service has its own dependencies Orchestration - running containers at scale ● Microservices = container deployments at scale ● Service discovery ● Scale services independently ● High Availability, Self Healing of services ● Hardware Abstraction
- 27. The world seems to think Kafka on Kubernetes is Difficult
- 28. There are some challenges Translating an existing architecture to Kubernetes: K8s pods, nodes, stateful sets, custom resources External access: What do we do with advertised listeners on Kubernetes? Managing configuration of Kafka, Zookeeper and all Confluent Components Persistent Storage options on prem and clouds Security Configuration and Upgrades
- 29. Confluent Operator enables you to: Automate provisioning of Kafka pods in minutes Monitor SLAs through Confluent Control Center or Prometheus Scale Kafka elastically & Automate rolling updates Built on our first hand knowledge of running Confluent at scale Confluent Operator is our Kubernetes Operator and our first step towards becoming cloud native CONFIDENTIAL
- 30. Confluent Operator - Automated Provisioning
- 31. Confluent Operator - Scale Horizontally Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources
- 32. Confluent Operator - Rolling Upgrade of all components Automated Rolling Upgrades of all components - Kafka Brokers, Zookeeper, Connect, Control Center Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker
- 33. Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
- 34. Demo – Ride Sharing Location Tracking 2 Web Apps powered by a microservice each Apps are a ride sharing driver and rider location tracking Rider hails a ride and the driver drives towards the rider Real time location events powered by Kafka on Kubernetes Rider Driver
- 35. Confluent Operator GA Plans We are in private Preview Release now 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech We are in the final stages of Preview and about to launch soon.
- 36. !!!
- 38. Kamala Dasika Product team, Platform and Ecosystem, Pivotal Michael Ng Product Manager, Confluent Thank You! Viktor Gamov Developer Advocate, Confluent
- 39. Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes Kamala Dasika Michael Ng Victor Gamov @DasikaKN @Micha8LNg @gAmUssA