Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
2 likes2,305 views
Spring Cloud Stream is a framework for building microservices that connect and integrate using streams of events. It supports Kafka, RabbitMQ, and other middleware. Kafka Streams is a client library for building stateful stream processing applications against Apache Kafka clusters. With Spring Cloud Stream, developers can write Kafka Streams applications using Java functions and have their code deployed and managed. This allows building stream processing logic directly against Kafka topics in a reactive, event-driven style.
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
- 1. Streaming with Spring and Apache Kafka Spring Cloud Stream Apache Kafka Kafka Streams
- 2. About me… Soby Chacko Committer - Spring Cloud Stream/Spring Cloud Data Flow Twitter: @sobychacko Github: github.com/sobychacko
- 3. Spring Cloud Stream Overview • General purpose framework for writing event driven/stream processing micro services • Destination based bindings on your choice of middleware
- 7. Apache Kafka • Single Source of Truth for data • Fault Tolerant • Based on the ubiquitous log data structure • Producer/Consumer/Streams API • Transactions • Exactly once semantics processing guarantees
- 8. Application Core Spring Boot Spring Kafka Spring Integration/Kafka Streams Spring Cloud Stream Inputs Outputs Spring Cloud Stream Application Model
- 9. Application Core Spring Cloud Stream Application Inputs Outputs Binder Middleware Zooming in on the Application Model
- 10. Spring Cloud Stream Event-Driven microservices framework Pluggable Binder Implementations Opportunities: Same code; Same tests; Drop-in replacement for a variety of Message Brokers Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace
- 11. Spring Cloud Stream Application level features • Message conversions are handled by the framework • Support for error handling - DLQ • Auto provisioning of topics (destinations) • Schema evolution • Health indicators • And a whole lot of other convenient features….
- 12. Spring Cloud Stream Programming model • Java 8 based functional programming model • Spring Cloud Function support as the foundation • Reactive programming support using Project Reactor • Framework provided annotations based programming (EnableBinding, StreamListener)
- 13. Spring Cloud Stream application types • Sources - java.util.function.Supplier • Sinks - java.util.function.Consumer • Processors - java.util.function.Function
- 14. A simple processor @SpringBootApplication … @Bean public Function<String, String> toUpperCase() { return String::toUpperCase; }
- 15. A simple source @SpringBootApplication … @Bean public Supplier<Long> currentTime() { return System::currentTimeMillis; }
- 16. A simple sink @SpringBootApplication … @Bean public Consumer<String> cons() { return x -> { System.out.println("Consumer"); }; }
- 17. Stream processing with Spring Cloud Stream and Kafka Streams
- 18. “We call an application data- intensive if data is its primary challenge—the quantity of data, the complexity of data, or the speed at which it is changing.”
- 19. Kafka Streams • Client library • No need for dedicated processing cluster • All guarantees of Kafka are applicable • Per record processing • Stateful stream processing • And many more features…
- 20. Spring Cloud Stream with Kafka Streams Spring Boot Application Kafka Streams binder from Spring Cloud Stream Kafka Streams library Kafka Broker Spring Kafka
- 21. Major types available for stream processing • KStream • KTable • GlobalKTable Spring Cloud Stream provides binding capabilities for all these three types.
- 22. Stream processing concepts • Stream-Table duality • Time based windowing • Joins and Aggregation • Stateless map and filter like operations on data
- 23. Stateful Stream Processing • Kafka Streams provide built in capabilities for stateful stream processing • Default state store - RocksDB • Spring Cloud Stream touch points for interactive queries in Kafka Streams
- 24. Spring Cloud Stream programming model for Kafka Streams apps • Java 8 based functional programming model • Processors can be written as java.util.function.Function or java.util.function.Consumer. • Multiple input/output binding capabilities
- 25. Kafka Streams app as a function… @Bean public Function<KStream<Object, String>, KStream<?, WordCount>> process() { return input -> input .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("W+"))) .map((key, value) -> new KeyValue<>(value, value)) .groupByKey(Serialized.with(Serdes.String(), Serdes.String())) .windowedBy(TimeWindows.of(5000)) .count(Materialized.as("foo-WordCounts")) .toStream() .map((key, value) -> new KeyValue<>(null, new WordCount(key.key(), value, new Date(key.window().start()), new Date(key.window().end())))); }
- 26. Quick Demo…
- 27. Resources • Code used for the demo • Samples Repository • Project page • Dataflow microsite • Confluent Blogs on Spring with Kafka: Part 1 | Part 2 | Part 3 | Part 4