Message Broker implementation in Kubernetes
Download as PPTX, PDF0 likes151 views
The document discusses the implementation of a message broker, specifically KubeMQ, within Kubernetes to address communication challenges in distributed microservices architectures. KubeMQ is highlighted as an enterprise-grade, lightweight, and easily deployable message broker that supports a variety of messaging patterns and has low maintenance requirements compared to alternatives like Apache Kafka. Key features include synchronous and asynchronous message relays, transport protocol support, and seamless integration with various development languages and observability tools.
Message Broker implementation in Kubernetes
- 1. Implementing Message Broker in Kubernetes
- 2. $whoami Suman Chakraborty - Senior Devops Engineer @SAP Labs Community member & Speaker - Docker Bangalore, CNCF Bangalore group Tech Blogger on PaaS, Cloud-Native & Microservices https://www.linkedin.com/in/ schakraborty007/ @itsmesumanc
- 3. Agenda Need for a Message Broker ? The Solution - KubeMQ Messaging Patterns Kubernetes Messaging Use-Cases Demo KubeMQ vs Other Message Brokers
- 4. Need for a message broker ? Kubernetes has become the de-facto standard for building and managing microservices, allowing seamless container management, scalability and service extensibility. Managing a distributed architecture means processing high volume of traffic flow of messages as each data model is disintegrated from the system. Problem arises, when the model grows (adding more data points, IoT hub,etc), that impact the communication process across the system. Enterprises attempt to solve this communication gap in Kubernetes using a Point- To-Pont connectivity system such as REST. However, REST is not effectively “cloud-native” in sense as its expensive, time- consuming and requires additional maintenance. Building an effective cloud-native messaging broker system for k8s involves re- architecting the stack and deploying a single focal point of communication for better communication. This ensures that each service communicates with the message queue broker in its own language
- 5. The Solution – KubeMQ Enterprise-grade message broker and message queue, scalable, high available and secured Written in Golang, created as a small light-weight container (~ 30 MB) Can be deployed through Operator for full LM operation Supports both synchronous and asynchronous message relays – for ‘At Most Once Delivery’ and ‘At Least Once Delivery models’ Supports gRPC, Rest and WebSocket Transport protocols with TLS support Scales instantly and work with or without persistent volume Supported on all Kubernetes environments ( Vanilla installation, Rancher k3s, Microk8s ) Supports message masticating and smart routing, authentication and authorization No Message broker configuration needed (i.e., queues, exchanges) Supported development languages -> .Net, Java, Python, Go and NodeJS SDKs Observability support: Prometheus, Jeager, Zipkin, AWS X-Ray, Datadog, Google Stack, Driver, HoneyComb
- 6. KubeMQ is part of CNCF landscape under Streaming and Messaging
- 8. Kubernetes messaging use-cases Multistage Pipeline :- This pattern processes the message between services in a coordinated workflow.
- 9. Tasks Queue : - An A-synchronic pattern that has multiple producers<-> many consumers relation to process a job. Each service is pulling messages from the queue to process it. The sequence of processing the messages is not important as each worker takes the task from the queue and process it in his own time.
- 10. Pub/Sub Stream: - This pattern streams data from many external sources (big data, IoT) and process it in a dedicated service such as database, pipeline, machine learning, storage, etc. Useful for multiple producers to small number of consumers relation.
- 11. Pub/Sub Real-time: - This A-Synchronic pattern is suitable for cases of a small number of producers sending real-time data to many consumers. The message is sent to a channel, where multiple services are subscribed to
- 12. Plans & Pricing
- 14. Advantages Robust messaging queue system Secured system Low DevOps maintenance system Well-connected ecosystem for logging in Kubernetes Rapid Deployment Configurable to handle change in cluster configuration
- 15. KubeMq vs other Message brokers KubeMq Apache Kafka Ultra-Simple & can be deployed by any DevOps Member Level Challenging, requires expert help Lightweight and Small Footprint Container (~90MB). No need for additional components Heavy and Resource-Demanding (~2.1 GB) (three Kafka containers + three ZooKeeper containers) KubeMq is faster as its built on Go Kafka requires dedicated setup on zookeeper, has a stateful setup Effortless, developer-oriented, no need to configure broker components separately Complex and dedicated setup required (additional knowledge on Java and Scala skillsets) All messaging patterns are Only Async messaging patterns are supported Off-the-Shelf integrations, CNCF- oriented Requires additional configurations and adjustments
- 17. THANK YOU