Building the Pivotal RabbitMQ for Kubernetes Beta

RabbitMQ & Kafka
October 7–10, 2019
Austin Convention Center
Madhav Sathe & Zoe Vance

Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Introduction
2

Madhav Sathe
Platform Architect
@madhav_sathe
Zoe Vance
Product Lead
zvance@pivotal.io

Beneﬁts
When to Use Each
Challenges
RabbitMQ Building Blocks
Developer Experiences
Live Coding
Kafka Building Blocks
Tips
Agenda
3

Beneﬁts

What is Rabbit
General purpose message broker, based around message queues,
designed with a smart broker / passive consumer model

Why RabbitMQ

Plugins
Tier 1 (19)
Community
- Routing
- Auth
- Mgmt
- Clustering
- Logging
- Queues
- Protocols
Why RabbitMQ
Client Libraries
Java (4)
Spring (3)
.Net (6)
Ruby (7)
Python (4)
PHP (7)
JavaScript & Node (4)
Rust (2)
Objective-C & Scala (1)
Other JVM (11)
C & C++ (4)
Go (3)… ﬁnd more at rmq docs

9

Why RabbitMQ
RabbitMQ Broker
Exchange(s)
Bindings
Queues

Easy to scale by adding/removing competing consumers on a single queue
Can be conﬁgured for consistency, high availability, low latency, high throughput
Cluster rolling upgrades via feature ﬂags
Easy to get started
Supports strict ordering
Wider use cases e.g., event driven microservices, RPC, ETL (with SCDF), enterprise message bus, pub-sub
messaging, real-time analytics (with Reactor and RabbitMQ Reactive API)
Why RabbitMQ

Distributed & partitioned commit log with messaging semantics
Distributed real-time streaming platform
What is Kafka?

Why Kafka
Very high throughput with data guarantees
Massive scale
Ability to replay events
De-facto standard for streaming platform
Ability to plug-n-play consumer groups on a topic
Supports strict ordering
Replace complex data architectures
Broad ecosystem
Wider use cases - pub-sub messaging, events driven microservices, logs store, streaming, event
sourcing, CDC, enterprise data pipelines

Why Kafka
Messaging API
Apps
Streaming API
Apps
Connect Sink
Connect Source

When to use each

If you don’t have speciﬁc Kafka requirements, then RabbitMQ gives you greater ﬂexibility, can meet
high throughput and real-time event-processing needs and has lower cost of operations
Evolving application requirements
Decoupled producer and consumers (using exchanges)
Consumers independently bring their own queue that binds to exchanges
Consuming applications don’t need to process messages that aren’t relevant
When to use RabbitMQ Over Kafka

When to use Kafka Over RabbitMQ
Streaming platform
Extremely high throughput
Joining multiple streams or streams and tables to enrich the data
Scale (typically beyond 5 brokers)
Replay

Challenges

RabbitMQ
Operational complexity in resolving network partitions
Queues are single-threaded
Scaling brokers >3 becomes complicated and can have negative performance impacts
No events replay
Does not natively support stateful streaming use cases (but can do so with Reactor +
RabbitMQ Reactive API with external store such as Redis)

Challenges in Kafka
No free lunch - operational complexity
Requires a separate Zookeeper cluster
Requires meticulous planning to select partition count
Storage management overheads
Careful coordination needed between teams writing consumer groups and producers
No out of box management & monitoring console
Streaming API support restricted mainly to Java
No out of box solution for upgrade*

Let us a play a little game
Let us say, we are introducing topic “Bar” for “appA” that requires parallelism of 20
consumers. So we deﬁne topic “Bar” with 20 partitions.
And YOU are a Kafka architect 😍
https://kafka.apache.org/20/documentation/streams/developer-guide/dsl-api.html#join-co-partitioning-requirements

At some later point “appB” too has a requirement to stream events from “Bar”. But “appB”
already streams events from topic “Foo”. “appB” actually needs to join the events from “Foo”
and “Bar”. However, “Foo” has only 10 partitions.
And YOU are a Kafka architect 😕

Now “appC” is interested in events from “Bar”. However, “appC” wants ordering across the
topic, so “appC” really needs “Bar” to have only one partition.
And YOU are a Kafka architect 😢

Now “appC” is interested in events from “Bar”. However, “appC” wants ordering across the
topic, so “appC” really needs “Bar” to have only one partition.
Both “Foo” and “Bar” events are becoming popular, so “appD” doesn’t want to be left
behind and wants both events. However, “appD” needs ordering guarantee across the all
events.
And YOU are a Kafka architect ……….. 😼 🤟

Credit: Jack Vanlightly @vanlightly

Building Blocks of
RabbitMQ

2
7

Messaging in RabbitMQ
BrokerProducers
Message
Exchange(s)
Bindings
Queues Consumers

Direct Exchange
Direct exchange delivers
messages to queues when the
message routing key exactly
matches the queue’s binding key.
images.crop cropper
Routing key
resizer
images.resize

Topic Exchange
Topic exchange delivers
wildcard matches between the
routing key and the queue’s
binding key.
*.*.error errors
Routing key
geos
eu.de.*

Fanout Exchange
Fanout exchange delivers
messages to all queues
regardless of routing keys or
pattern matching.

Clustering with RabbitMQ

Autoheal partition handling strategy optimizes for AP

Pause Minority partition handling strategy optimizes for CP

Publisher Conﬁrms
Broker
Producer
Exchange
Bindings
Queues
Ack

Consumers
Broker
Exchange
Bindings
Queues

RabbitMQ Roadmap
RabbitMQ
OSS
RMQ for Pivotal
Platform
RMQ on
Kubernetes
RabbitMQ 3.8 just out!
- Quorum queues using RAFT to
provide persistent and fault tolerant
messaging systems
- Mixed-version rolling upgrades
- Enhanced observability (new
metrics and built in plugin with
visualizations in grafana)
- OAuth 2.0 Support
In Closed Beta
Goal is to provide great developer and
day-2 operational experience (automated
reliable upgrades, problem resolution and
actionable observability)
RabbitMQ for PCF 1.18 just out!
- Support for oﬀ-platform
applications instances to
on-platform RabbitMQ instances
PAS PKSDIY

RabbitMQ Developer Experience

Spring Cloud
Stream
● Abstraction over
protocol
● Abstraction over
messaging vendor
● Same code
regardless of
messaging broker
Spring AMQP
● Provides a
"template" as a
high-level
abstraction for
sending and
receiving messages.
● Support for
Message-driven
POJOs with a
"listener container".
● Similar to the JMS
support in the Spring
Framework.
RabbitMQ AMQP
Client
● Low level API to
RabbitMQ
Options for Java developers
Reactor RabbitMQ
● Reactive API for
RabbitMQ based on
Reactor and RMQ
Java Client
● Functional APIs
enables messages to
be
published/consumed
with non-blocking
back-pressure and
very low overheads

Kafka Developer Experience

Spring Integration
Spring for Apache Kafka
Spring Cloud Stream Binder for
Kafka
Spring Cloud Stream Kafka Streams Binder
KStream, KTable & GlobalKTable
Spring Cloud Stream
Input & Output Message Channels Spring Cloud Kafka Stream
Spring Cloud Stream Microservice
Spring Cloud Stream Kafka Streams
Microservice
Binder

Demo

JustRide: Driver Behavior Driven Car Insurance
Car Events
Violations
Score Processor
Customer Score
Caches
Customer Info
Customer
Scores API
vSphere
Azure &
Azure Stack
Google CloudAWS
Dashboard
Pivotal Platform
Speed Check
Processor
Customer Score
Customer
Score Sink

Speed Check Processor
(SCSt + Kstream)
For the demo
Car Events
Violations
Car Events Load
Generator
(SCSt)
Score Processor
(SCSt + Kstream)
Main business logic
application.yaml to deﬁne input and output bindings with
topics as destinations

Model for the demo
CarEvent:
- uuid
- latitude
- longitude
- speed
Simple JSON friendly Pojo
Message Key: uuid

Model for the demo
CarEvent:
- uuid
- latitude
- longitude
- speed
Simple JSON friendly Pojo
Message Key: uuid
ViolationEvent:
- uuid
- List<CarEvent>
- violationCount
- start
- end
State Management:
- new() - ‘start’ timestamp, new ArrayList<CarEvent>
- addCarEvent() - check speed, add to list, increase
violationCount
- closeWindow() - ‘end’ timestamp
JSON friendly “self aware” data model
Message Key: uuid
(SCSt + Kstream)

Building Blocks
Kafka

Sequential I/O as opposed to random → huge benefits for disk
performance
Kafka is Log & Real-time*, How’s That Possible?
Extremely smart utilization of OS page cache → achieve read and writes
without disk IOPS in call path
Zero copy send files → kernel copies the data directly from the disk file to
the socket, without going through the application
* When evil scenarios are avoided :)
In-memory
performance

Broker A
File System
In-memory Cache
Producer
1 - Send message
2
3
4
5 - Receive ACK
ACK = 1
https://www.conﬂuent.io/kafka-summit-sf18/kafka-on-zfs

Broker A
File System
In-memory Cache
Consumer
2
4
1
Socket
Buﬀer
NIC Buﬀer
3
5
https://medium.com/@sunny_81705/what-makes-apache-kafka-so-fast-71b477dcbf0

Broker and Cluster
Broker A
Controller
- Monitoring other
brokers
- Broker shutdown
- Election of
partition leaders
- Tell brokers
about partition leaders
Broker B Broker C
Zookeeper Cluster
/controller → Broker A
/topic/A/0 → Broker A
/topic/A/1 → Broker B
/topic/A/2 → Broker C

Topic and Partitions Basics
Partition 0
Partition 1
Topic is divided in partitions, each partition is
essentially a log ﬁle
Partitions can have replicas for HA
One of replicas is chosen as a Leader
All reads and writes happen only on Leader
Publisher can only append to the partition
Ordering guarantees only in a partition
Topic

Broker B
Partition: 0
Leader
Partition: 1
Follower
Partition: 0
Follower
Partition: 1
Leader
Broker A
Producers
Producers
Producer API
Cluster: 2 brokers,
Topic: trades,
Partitions: 2,
Replicas: 2
Messages without a key
are shared across
partitions in round robin
fashion
Hash based partition
selection
Messages with same key
go to same partition
Batch Batch

Partition and Consumer Offset
0 1 2 3 4 5 6 7
Producers/Publishers
Alana’s Offset
Cody’s Offset Zoe’s Offset
P0
P0 P0 P0
1 3 6
Partition offset map topic
Just reset your offset to
re-play the messages

Kafka Streams
Cluster
Topic: trades
Kstream
Topic: company
proﬁles
Ktable
Topic:
recommendations
Kstream
Kafka managed topic
Kafka Streaming
Microservice
Fault tolerant state
store

Cluster
Partitions and Consumer Groups
Topic: trades,
Partition: 0
Topic: trades,
Partition: 1
Consumer Group
Consumer 1

Cluster
Topic: trades,
Partition: 0
Topic: trades,
Partition: 1
Consumer Group
Consumer 1
Consumer 2

Cluster
Topic: trades,
Partition: 0
Topic: trades,
Partition: 1
Consumer Group
Consumer 1
Consumer 2
Consumer 3

Cluster
Topic: trades,
Partition: 0
Topic: trades,
Partition: 1
Consumer Group
Consumer 1
🔥Consumer 2

Tips for RabbitMQ

Tips to eﬃciently use RabbitMQ
● Overall
○ Clarify requirements of your application
○ Monitor RabbitMQ
○ Use wide range of RabbitMQ Resources
● Queues
○ Happy Rabbit is an Empty Rabbit - keep queues short
○ For performance, use in-memory, non-mirrored queues
○ For HA & data safety, use Quorum Queues
● Producer/Consumers
○ Use multiple consumers if consumers are slow or there are too many producers
○ If messages can’t be lost, use acknowledgments
○ For performance, use autoack
● Resources
○ Every connection, channel, queue costs memory & CPU; more there are and harder they work, more resources
are required
○ For throughput, have as many queues as cores on the underlying nodes of a multi-core system
● Messages
○ For performance, keep messages in memory
○ If messages can’t be lost, ensure you use pre-fetch values with publisher conﬁrms/consumer acks

Tips for Kafka

Tips to efficiently using Kafka
● Use ACK = All if you love your data, ACK = 1 for balance between reliability & latency
● Route “hot keys” to a different topic to avoid “hot partitions”
● Producer efficiency
○ Pick optimal batch size that fills up fast so that you good mix of throughput and latency
○ Linger of 5ms is a usually a good thumb rule
○ Increase batch size for higher throughput
○ Lower linger for lower latency
● If you have large files to send consider following options
○ Put files on shared location and send location of the files on kafka
○ Break down file to right size and use keys to ensure ordered processing
● Tips on using keys
○ Setting Key=Null gives best performance and balanced partitions across cluster
○ Explore possibility of leveraging downstream stores to establish ordering
○ Use keys only if you need ordered messaging in real-time or joins across different topics
● A lot of Kafka’s performance depends availability of page cache and GC overhead so monitor these two
parameters with extra care

● Extremely large messages may block consumers if the consumer is not configured with adequate buffer size. If there
are some large messages, consider using a separate topic
● Ad-hoc offset reset/replays while writes and other reads are going on
● ACK=All will have some impact on latency so plan the partitions accordingly
○ In-Sync replicas lagging behind, always watch your ISR list
● Watch out for disk IOPS during reads and writes
○ Less available memory for page caching forcing more disk IOPS
○ Vastly lagging consumers working at different speeds forcing disk IOPS
● Brokers running different versions within a cluster can performance issue
○ Few brokers running much faster or much slower than rest of the cluster
● Watch out for Zombie brokers in older Kafka versions
● Old Kafka client libraries (producer/consumer) may have adverse impact on throughput and latency
● Check rebalance of partitions if broker goes down, if you don’t throttle it then it can use up all your n/w bandwidth
● Restarting cluster with large number of partitions, leader election takes time
● If consumer crashes or unable to send heartbeats, the partition reassignment will take place, and during this time no
consumer in the group can process any message
Watch out! Things causing adverse impact in Kafka

Thanks to
Jack Vanlightly
Marcial Rosales
Soby Chacko
Sina Sajoodi
Wayne Lund
Timothy Dalsing
Dan Carwin
Gerhard Lazu
Karl Nilsson
Arnaud Cogoluegnes
James Williams

Resources
3.8 Blog

Thank You!

Appendix

Cluster
Topic: trades,
Partition: 0
Topic: trades,
Partition: 1
Sink
Consumer Groups
Streaming, Joins, Aggregations,
Windowing
AI

● Running out of date software
● Opening/closing channels repeatedly
● Using lots of connections/channels (causes lots of overhead and is not performant)
● Long queues
● Large messages
● Conﬁguring RabbitMQ for scenarios you don’t need e.g. using ‘ha-all’ when mirroring to a
quorum is suﬃcient or using durable queues when non-durable queues work
Watch out! Things causing adverse impact in RabbitMQ

Cluster - Architecture
Exchange
routing_key1
routing_key2
routing_key3
…
Queue1 Metadata
Queue1 Contents
Exchange
routing_key1
routing_key2
routing_key3
…
Queue1 Metadata
Exchange
routing_key1
routing_key2
routing_key3
…
Queue1 Metadata

Broker B
Partition: 0
Leader
Partition: 1
Follower
Partition: 0
Follower
Partition: 1
Leader
Broker A
Partition Replicas and Brokers
Cluster: 2 brokers,
Topic: trades,
Partitions: 2,
Replicas: 2

Consistent Hashing
Routing key
½ hash
space
Routes messages to specific
queue based on the hash of the
routing key or header. Enables
order guarantees with multiple
consumers
½ hash
space

Connected Cars & Insurance
Car Events
Violations
Score Processor
Customer Score
Caches
Customer Info
Customer
Scores API
vSphere
Azure &
Azure Stack
Google CloudAWS
Dashboard
Pivotal Platform
Speed Check
Processor
Customer Score
Customer
Score Sink

(SCSt + Kstream)
For the demo
Car Events
Violations
Car Events Load
Generator
(SCSt)
Score Processor
(SCSt + Kstream)
Main business logic
application.yaml to deﬁne SCSt input and output bindings for
each of the components - 1. load generator, 2. speed check
processor & 3. score processor

Live code SCSt + KStream - stateful transformation logic
(SCSt + Kstream)
Main business logic
Consume KStream of events from “car-events” topic, apply business logic using stateful
transformations & send KStream of events to “violations” topic for cars that meet the criteria

Double click on KStream stateful transformation logic
(SCSt + Kstream)
Main business logic
1. Group By Key (car UUID)
2. Create tumbling window for 30 sec

(SCSt + Kstream)
Main business logic
3. Perform stateful transformation using Aggregator (this is where we use self
aware model) that creates a KTable for each Windowed<UUID>
a. Initialize ViolationEvent (model decides what is required on
initialization)
b. Call aggregator in ViolationEvent for each CarEvent (model decides
how to implement aggregation, this is where CarEvent is checked for
overspeeding)
c. Materialize the state (KTable) in a Kafka managed state store using
SerDe

(SCSt + Kstream)
Main business logic
3. Perform stateful transformation using Aggregator (this is where we use self aware
model) that creates a KTable for each Windowed<UUID>
a. Initialize ViolationEvent (model decides what is required on initialization)
b. Call aggregator in ViolationEvent for each CarEvent (model decides how to
implement aggregation, this is where CarEvent is checked for overspeeding)
c. Materialize the state (KTable) in a Kafka managed state store using SerDe
4. Update the value, let ViolationEvent close the window (again, let the model
decide how to manage itself)

(SCSt + Kstream)
Main business logic
4. Update the value, let ViolationEvent close the window (again, let the model decide how
to manage itself)
5. Convert the KTable to KStream

(SCSt + Kstream)
Main business logic
to manage itself)
6. Filter out the ViolationEvents that have less than “Threshold” violation
occurrences in a window

(SCSt + Kstream)
Main business logic
to manage itself)
6. Filter out the ViolationEvents that have less than “Threshold” violation occurrences in a
window
7. Map Windowed<Key> to Key for downstream apps

OLD SLIDES

Abstract
When should I use RabbitMQ and when should I use Kafka? This is a question we're
asked all the time on our team.
In this talk, we'll walk through the diﬀerent architecture and design of these two
popular messaging tools. We'll also explain what those design decisions mean for
performance, scaling, and diﬀerent use cases.

Provide guidance on how to think about how these two
powerful tools in your architectures.
Goal
8
7

“Events First” as a core design principle

AppA AppB❌
Dont tell what to do
AppA
AppB
Events
Share what happened via an event
Let others ﬁgure out what to do with it
AppC
Later, more apps may be interested in these events,
still nothing changes for app that generated events

Events driven systems are not new
However
The recent hyper growth is fueled by a shift from batch processing
to real-time stream processing and the rise of digital economy

ETL
EAI
Events Driven,
(near) real-time
Streaming
Platform
+
Scale
+
Events Driven,
Real-time
Diversity

RabbitMQ

Events ﬁrst as a core principle
RabbitMQ building blocks
Apache Kafka building blocks
Diﬀerences and Architectures
Design Considerations
Spring Boot 🧡 Kafka (live coding)
Spring Boot 🧡 RMQ (live coding)
Agenda
9
3

CG1
Topic Exchange A
App - CG1 with SCSt,
Binding = TopicA
App - CG2 with SCSt,
Binding = TopicACG2

9
5

What is RabbitMQ
● Open-source project written in Erlang backed by Pivotal
● A general purpose message broker designed with a smart broker / passive
consumer model
● Enables cross-language messaging with Advanced Message Queueing Protocol
speciﬁcation and support for range of additional protocols (STOMP, MQTT, HTTP,
AMQP 1.0)
● A tool with wide and extendable support (many client libraries, many plugins)

9
6

How does RabbitMQ work
BrokerProducers
Message
Exchange(s)
Bindings
Queues Consumers

Exchanges and
bindings are the routing
instructions for message
distribution.
Bindings are the rules
about which queues are
relevant for the
message.
Exchanges and bindings
Exchange
Bindings

Topic Exchange
Topic exchange delivers
wildcard matches between the
routing key and the queue’s
binding key.
*.*.error errors
Routing pattern
geos
eu.de.*

Publisher Confirms
Mechanism for producers to confirm successful handling by RabbitMQ. Publisher
confirms will inform if a message is dropped (e.g., not routed to a queue) or if a
message has been appended to the relevant queues or written to disk.
Increasing levels of message reliability
make a performance trade-off
Broker
Producer
Message
Exchange
Bindings
Queues
Ack

Consumer Acknowledgement
Mechanism for consumers to conﬁrm successful delivery and processing
of messages
Consume
No-Ack Consume without acknowledge
Ack Acknowledge message
Multi-Ack Acknowledge multiple messages at once
Reject
Reject Reject individual, delivered message
Nack Reject messages in bulk
Increasing levels of message reliability
make a performance trade-off

Apps and RabbitMQ on Pivotal Platform

Apache Kafka

Topic and Partitions Basics
Partition 0
Partition 1
Topic is divided in partitions, each partition is essentially a log ﬁle
Messages are immutable
Publisher can only append to the partition
All reads are sequential
Ordering guarantees only in a partition
Partitions can have replicas for HA

Log & Real-time*, How’s That Possible?
Extremely smart utilization of OS page cache →
broker writes to file system
OS actually writes to memory, which is flushed to disk later
disk IOPS used primarily (async & bulk) writes
broker effectively reads from memory
In-memory
performance,
minus overheads
of Java Heap
performance

Broker A
File System
In-memory Cache
Producer
1 - Send message
2
3
4
Replicas: 2
File System
In-memory Cache
Broker B
a
b
c
5 - Receive ACK

Broker A
File System
In-memory Cache
Consumer
2
File System
In-memory Cache
Broker B4
1
Socket
Buffer
NIC
Buffer
3
5

Log & Real-time*, How’s That Possible?
Extremely smart utilization of OS page cache →
broker writes to file system
OS actually writes to memory, which is flushed to disk later
disk IOPS used primarily (async & bulk) writes
broker effectively reads from memory
In-memory
performance,
minus overheads
of Java Heap
performance
Partitions, horizontally scalable brokers → parallelism

Tips to efficiently use Kafka
● Use ACK = All if you love your data
● Separate topics based on need of parallelism
● Producer efficiency
○ Optimal batch size that fills up fast so that you good mix of throughput and latency
○ Linger of 5ms is a usually a good thumb rule
○ Increase batch size for higher throughput
○ Lower linger for lower latency
● If you have large files to send consider following options
○ Put files on shared location and send location of the files on kafka
○ Break down file to right size and use keys to ensure ordered processing
● Tips on using keys
○ Setting Key=Null gives best performance and balanced partitions across cluster
○ Explore possibility of leveraging downstream stores to establish ordering
○ Use keys only if you need ordered messaging in real-time or joins across different
● A lot of Kafka’s performance depends availability of page cache and GC overhead so monitor these
two parameters with extra care

What can cause adverse impact on Kafka
●

Apps and Kafka on Pivotal Platform

Deﬁne Service Goals
Throughput
Latency
Durability
Availability

Throughput is important
However
It is meaningless in isolation

Throughput Considerations
What are the requirements/constraints of your application and infrastructure?
- What delivery guarantees do you need for your data from producer/consumer?
- Do you need replay?
- Will your messages have keys?
- Will your partitions stay balanced or become unbalanced?
- How can you optimize batch size by thinking about bytes/sec, messages/sec and
payload size?
- How will your requirements evolve over time?
- What will be the eﬀect of serialization/deserialization? Hint 💡: Throughput will be much
lower than plain Byte[]
Source: https://www.infoq.com/presentations/spring-xd-stream-processing/
by S Maldini, G Renfro and D Turanski

● Assume infrastructure failure will happen
○ What happens when a node goes down, a rack or AZ goes down? What happens
when there is a network split across racks or AZs?
● Can you aﬀord to lose a few messages? How many?
● Is ordered processing important?
● What will the state of system be if app processes duplicate messages?
● Despite broker guarantees always plan for idempotency in your consumer code.
○ What happens when a message is processed and saved in DB but ACK fails? What
happens when deserialization fails?
● Payload schema, will it evolve over time?

Understand Tradeoﬀs, No Free Lunch!
Smart Broker vs Dumb Broker
Throughput vs Ordering
Throughput vs Durability
Throughput vs Exactly Once ⚠
Broker Features vs Your Actual Business Needs
And
Cost of License & Operations vs Your Actual Business Needs

● Kafka Broker optimises for:
■ Very high throughput needs
■ Decoupled producer and
consumer applications to evolve
their data format independently
■ Native stateful stream processing
■ Multiple consumer groups per
topic
■ Replay
1
2
1

Where each shines
● RabbitMQ Broker optimises for:
■ Smart broker with advanced
content based routing
■ Competing consumers
■ Cross-protocol and language
support
■ Self service experience for
developers on PCF
■ Easy to manage

TEMPLATE THEME SLIDES

Comparison

Throughput Considerations
● Verify the throughput based your app requirements & infrastructure.
● Have you tested throughput with Serialization/Deserialization?
○ Hint 💡: It will be much lower than plain Byte[]
● Are partitions well balanced or largely unbalanced? Do messages have keys?
● Are there lagging consumers? Is replay a business need?
● What are the delivery guarantee requirements on producer/consumer?
● Measure throughput in bytes/sec vs messages/sec
● Consider bytes/sec, messages/sec and payload size to optimize batch size
● Will throughput requirement increase over time, what does it look like 2 years, 4 years
from now?
Source: https://www.infoq.com/presentations/spring-xd-stream-processing/
by S Maldini, G Renfro and D Turanski

1
2
5

Use cases
- Point to point microservice based architectures
- Pub-Sub microservices
- Real time analytics
- SQL table changes as events
- ETL

The Binder Abstraction
● Provides a programing API that removes all
dependencies from messaging vendors
● Same code regardless of messaging broker
Available binders
● RabbitMQ
● Kafka
● Gemﬁre
● IBM MQ
● JMS
● Custom binders with the Binder SPI
Spring Cloud Stream

Spring Cloud Stream
● Abstraction over protocol
● Abstraction over messaging
vendor
● Same code regardless of
messaging broker
Spring Cloud Abstractions for RabbitMQ & Kafka

1. Create a Sink interface
2. Declare an a method
returning a MessageChannel
3. Annotate the method with
@Input and give a name to
the queue to be notiﬁed with
Spring Cloud Stream Programming Model - Consumer

1. Declare a Service Bean
2. Annotate a handler method
with @StreamListener
3. Provide the queue name to
the StreamListener

1. Activate the binding of Sink
with @EnableBinding
2. Bootstrap the application like
a regular Spring Boot
Application

Spring Cloud Stream Programming Model - Producer
1. Create a Source interface
2. Declare an a method
returning a MessageChannel
3. Annotate the method with
@Output and give a name to
the queue notify

1. Declare a Service Bean
2. Inject the Source that Spring
Cloud Stream will Auto
Conﬁgure
3. Use the source to produce
messages

Learn More: RabbitMQ Patterns Simulator

1
3
6

How does Kafka work: Topics and Partitions
Producers
Message
Consumers
Partitions
Topic
0 1 2 3 4 5 ... n
0 1 ... z
0 1 2 3 ... x

1
3
7

Clusters in Kafka
Broker
Partitions
0 1 2 3
.
.
.
x
Broker
Partitions
0 1 2 3 4 5
.
.
.
n
0 1
.
.
.
z
0 1 2 3
.
.
.
x
Broker
Partitions
0 1 2 3 4 5
.
.
.
n
0 1
.
.
.
z
0 1 2 3
.
.
.
x
Broker
Partitions
0 1 2 3 4 5
.
.
.
n
0 1 2 3
.
.
.
x

STOP! Download Fonts Now
PLEASE INSTALL PROXIMA NOVA FONTS BEFORE CREATING YOUR PRESENTATION
Download: https://brandfolder.com/pivotal/springone-platform
Guest Password: keepitsimple
Instructions on how to install fonts:
http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-mac
http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-windows-pc

Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of
Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal
and is subject to change. This information is provided without warranty or any kind, express or implied, and
is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on
features currently available. The development, release, and timing of any features or functionality described
for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to
update forward looking information in this presentation.
If you are a Pivotal employee and your talk contains forward facing information,
please include this slide at the beginning or conclusion of the presentation.
Remove this text box before presenting.

Slide Templates

Body Slide — Light Background
All body text is Proxima Nova Regular
• Subhead
• Level Two
• Level Three
• Level Four
1
4
1

Body Slide — Dark Background
All body text is Proxima Nova Regular
• Subhead
• Level Two
• Level Three
• Level Four
1
4
2

14
3

// This is Roboto Mono: 42pt or higher please
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}
…
}

14
4
// This is Roboto Mono: 42pt or higher please
public class TransferServiceImpl implements TransferService {
public TransferServiceImpl(AccountRepository ar) {
this.accountRepository = ar;
}
…
}

Design Assets

Pivotal Product Logo Library
One-color versions

Spring Logo and Project Icons
Spring
Framework
Spring
Security
Spring
Data
Spring
Batch
Spring
Integration
Project
Reactor
Spring
AMQP
Spring
Hateoas
Spring
Mobile
Spring
Android
Spring
Social
Spring
Web Services
Spring
Web Flow
Spring
XD
Spring
Boot
Spring
LDAP
Spring Tool
Suite
Spring Cloud
Data Flow
Spring
Kafka
Spring
Cloud

Icon Library
Router Team Metrics Service Broker Server Permissions Database
Cloud Mobile Workstation Browser Laptop IoT Security
CLI User 1 User 2 User 3 Alert
Star Deploy/Ship
Agile Container Microservice Monolith Application
File / Log
Checklist

Icon Library

Event Logos
On dark
Horizontal Vertical
On light
On mint

Looking for other logos?
Visit our BrandFolder >
(pw: keepitsimple)
There you’ll ﬁnd all our:
• Product Logos
• OSS Logos
• PCF Service Icons
• Brand Fonts & Colors
• And more!

Brand Colors
SpringOne Platform 2019
Primary Colors
Spring
Brand Color
Pivotal
Brand Color
#1ab9a5ff
#13377dff
#ffffffff
#6db33fff #1ab9a5ff
#00253eff

Sample Table
15
3
2007 2008 2009 2010
Category 1 2.4 6.4 4.1 6.6
Category 2 8.2 4.5 3.2 3.8
Category 3 4.6 3.2 1.9 9.6
Category 4 6.7 3.3 3.4 2.2
Category 5 4.3 5.6 7.1 3.4

Sample Table
15
4
2007 2008 2009 2010
Category 1 2.4 6.4 4.1 6.6
Category 2 8.2 4.5 3.2 3.8
Category 3 4.6 3.2 1.9 9.6
Category 4 6.7 3.3 3.4 2.2
Category 5 4.3 5.6 7.1 3.4

Stay Connected.
<Your CTA>
<Related Session>
<Related Session>
#springone@s1p

Building the Pivotal RabbitMQ for Kubernetes Beta

More Related Content

What's hot (12)

Similar to Building the Pivotal RabbitMQ for Kubernetes Beta (20)

More from VMware Tanzu (20)

Recently uploaded (20)

Building the Pivotal RabbitMQ for Kubernetes Beta