Developer Week - Reacting to an event-driven world

Editor's Notes

• #4: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better Clement’s session C3 – 4pm It is possible to do http requests without blocking the thread, but even with that switch you are still approaching from a request/response perspective
• #5: Event Driven Architecture (EDA) is a popular architectural approach that enables events to be placed at the heart of our systems Consists of Events Events are records of something that has happened, a change in state - immutable and are ordered in sequence of their creation. Interested parties can be notified of these state changes by subscribing to published events and then acting on information using their chosen business logic. An event-driven architecture, refers to a system of loosely coupled microservices that exchange information between each other through the production and consumption of events.
• #6: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #8: No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application?Short answer: NOWhile Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
• #9: No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application?Short answer: NOWhile Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
• #10: No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application?Short answer: NOWhile Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
• #11: No! This isn’t reasonable! Kafka is a good tool, but it isn’t enough to have a good tool, you need to use it in the right way You also need to think about your applications and other services, Kafka isn’t your whole architecture – integration between components is key! Can we just use Kafka to create a Reactive application?Short answer: NOWhile Kafka look after the messaging part, we still need a Reactive Microservice implementation, for instance, using the actor model to replace thread synchronization with queued message processing or the supervisor model to handle failures and self-healing. We definitely need both Akka and Kafka to build Reactive Microservices based responsive, resilient and elastic systems.
• #12: So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
• #13: Talking about message driven vs event driven
• #14: Talking about message driven vs event driven
• #16: Open sourced distributed streaming platform, often being adopted as the “de-facto” event streaming technology Arrived at the right time, captured mindshare among developers and so exploded in popularity Kafka has deliberately moved away from the word “events”… instead uses records now
• #17: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #18: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #19: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #20: Talking about message driven vs event driven
• #21: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #22: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #23: Use this as an example -> there are plenty of existing demos showing that using event driven vs e.g. http is much better
• #24: Talking about message driven vs event driven
• #25: Talking about message driven vs event driven
• #26: Talking about message driven vs event driven
• #27: How do we actually achieve this when using event-driven architecture and tools like Kafka? Kafka is naturally resilient : Multiple brokers with replication of events within (Leader and at least two followers) Leader election if leader broker goes down, don’t loose connection between app and Kafka Kafka configuration can also help  RESILIENT PRODUCER  Delivery Guarantees (Using “at least once”, so that you guarantee delivery of event or message to Kafka) and Retries (in case sending fails) RESILIENT CONSUMER  Use of manual offset commit rather than automatic, line of code in app that determines when offset should be committed so messages aren’t lost during processing by consumers. Kafka is naturally scalable: Ability to scale out work for each topic across brokers  scaling load on system Use of Consumer Groups to scale out number of consumers while keeping the order guarantee and preventing duplication of processing of same message or event. BUT NOT DESIGNED FOR REACTIVE – THAT’S WHERE WE CAN UTILISE REACTIVE FRAMEWORKS AND TOOLKITS TO REALLY ENABLE ALL OF THESE BAHVIOURS, BUILDING UPON WHAT COMES AS STANDARD WITH KAFKA.
• #28: All of these frameworks take the role of handling polling, then provide different ways for us to handle the processing and flow. MicroProfile and Alpakka provide a more constrained way of doing backpressure and flow control, whereas vertx provides the tools for doing that separation but leaves it up to the developer to decide exactly how to implement it. We will describe flow control for MP and alpakka, “how they improve flow control and allow application to follow reactive streams spec” Others: Alpakka Kafka Connector Spring WebFlux Project Reactor
• #29: Eclipse MicroProfile is an open-source community specification for Enterprise Java microservices A community of individuals, organizations, and vendors collaborating within an open source (Eclipse) project to bring microservices to the Enterprise Java community
• #30: The MicroProfile Reactive Messaging specifiMicroProfile Reactive Messaging makes use of and interoperates with two other specifications: Reactive Streams is a specification for doing asynchronous stream processing with back pressure. It defines a minimal set of interfaces to allow components which do this sort of stream processing to be connected together. MicroProfile Reactive Streams Operators is a MicroProfile specification which builds on Reactive Streams to provide a set of basic operators to link different reactive components together and to perform processing on the data which passes between them. When you use the MicroProfile Reactive Messaging @Incoming and @Outgoing annotations, Open Liberty creates a Reactive Streams component for each method and joins them up by matching the channel names. cation aims to deliver applications embracing the characteristics of reactive systems
• #31: A method with an @Incoming annotation consumes messages from a channel. A method with an @Outgoing annotation publishes messages to a channel. A method with both an @Incoming and an @Outgoing annotation is a message processor, it consumes messages from a channel, does some transformation to them, and publishes messages to another channel. When you use the MicroProfile Reactive Messaging @Incoming and @Outgoing annotations, Open Liberty creates a Reactive Streams component for each method and joins them up by matching the channel names.
• #32: Polyglot Java, Javascript, Groovy, Ceylon, Scala and Kotlin The reactor pattern is one implementation technique of event-driven architecture. In simple terms, it uses a single threaded event loop blocking on resource-emitting events and dispatches them to corresponding handlers and callbacks. It receives messages, requests, and connections coming from multiple concurrent clients and processes these posts sequentially using event handlers. The purpose of the Reactor design pattern is to avoid the common problem of creating a thread for each message, request, and connection. Then it receives events from a set of handlers and distributes them sequentially to the corresponding event handlers. It’s single-threaded – so you must not block the thread! The Kafka client is becoming more popular and e.g. it is used by SmallRye Reactive messaging
• #33: So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
• #34: Demo the starter app working Key takeaways: Choosing a reactive framework makes it easier to work with Kafka Strimzi, cool open source project that provides a Kubernetes operator for Kafka, just been accepted into CNCF (Cloud Native Computer Foundation) Kate active contributer to Strimzi and I was interested in Vert.x
• #35: Demo the starter app working Key takeaways: Choosing a reactive framework makes it easier to work with Kafka Strimzi, cool open source project that provides a Kubernetes operator for Kafka, just been accepted into CNCF (Cloud Native Computer Foundation) Kate active contributer to Strimzi and I was interested in Vert.x
• #38: In the original version of the application, we were having to use a for loop inside a while loop that resulted in the following flow when consuming records: Poll for records. Poll function returns with a batch of records. Iterate through each record in the batch. For each record, send the record along the WebSocket to the frontend. After iteration is complete, return to Step 1. So, while we were processing the current batch of records, no new records could be fetched. 
• #39: In the new application using the Vert.x client instead, you simply write a handler function that will be called every time a new record arrives. This allows asynchronous consuming and processing of records which fits in much better with the reactive paradigm. This change makes the flow a lot simpler. Vert.x now handles the step of polling for new records. The application code does the following: Receive a new record. Send the record along the WebSocket to the frontend. This not only allows the application to process on a per-record basis but also leaves the developer free to focus on the processing of records, rather than the work of consuming them from Kafka.
• #40: We wanted the ability to pause and resume both producing and consuming records from the frontend. In the original application, where the default Java Kafka client was used, the while loop that calls the poll for the consumer needed to be exited and resumed based on commands from the frontend. Because this client is single threaded, the loop had to be executed in a separate thread to the main flow of the application. As we wanted to produce records on a timer, a similar while loop in a separate thread was required for the producer. Therefore, we had to deal with the headache of writing logic to control the thread lifecycle and synchronize data across threads just to get control over the flow of records. In the Vert.x application, we were able to use a combination of Vert.x features and Vert.x Kafka client features to make this flow control much easier. For the producer, we used a feature in Vert.x that allows a handler to be called on a timer basis. The handler could be called once, or multiple times, and it was easy for us to pause and resume the timer when start and stop requests arrived from the WebSocket. For the consumer in the Vert.x application, we could use the built-in back-pressure support that allows consumers to be paused and resumed. When a stop command comes from the WebSocket, the consumer is paused and the Vert.x client will stop polling for new records from Kafka. At a later time, the consumer can be resumed and the Vert.x client will resume polling and the existing handler will start receiving new records.
• #42: So Kafka claims to have scalable consumption and resiliency, do I just get that for free when I start Kafka? How does it work?
• #46: Creating reactive Java Microservices Testing reactive Java microservices Consuming RESTful services asynchronously with template interfaces Integrating RESTful services with a reactive system Acknowledging messages using MicroProfile Reactive Messaging
• #49: How do we actually achieve this when using event-driven architecture and tools like Kafka? Kafka is naturally resilient – (brokers go through
• #50: Talking about message driven vs event driven
• #51: Imagine a broker goes down, this means the leader of Topic A, partition 1 is offline
• #52: Can’t do fire and forget if you want full resiliency cause if the broker goes down your messages get lost Two different guarantees, way you get them is through confirguartion At most once, you may lose some messages (not completely relisient) At least once, guaranteed delivery but may get duplicates Retries is if acks times out/fails – how many times do you retry producing the event (how will the retry affect ordering)
• #54: Talking about message driven vs event driven
• #55: Elasticity in Kafka itself Scale out brokers, can’t scale down (where do events go if you did?) Can scale out partitions but can’t scale them down again Can add topics, and delete topics if you don’t care about them
• #56: To allow scalability of consumers, consumers are grouped into consumer groups. Consumer declare what group they are in using a group id For consumers we use Consumer groups to enable elasticity
• #57: Elasticity in Kafka itself Scale out brokers, can’t scale down (where do events go if you did?) Can scale out partitions but can’t scale them down again Can add topics, and delete topics if you don’t care about them