![Kafka Streams is just an API
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
builder.stream(”caterpillars")
.map((k, v) -> coolTransformation(k, v))
.to(“butterflies”);
new KafkaStreams(builder.build(), props()).start();
}
24](https://image.slidesharecdn.com/81550confluent-171124114049/85/Big-Data-LDN-2017-The-Streaming-Transformation-24-320.jpg)
Big Data LDN 2017: The Streaming Transformation
- 3. Evolution!
- 4. KAFKA Serving Layer (Cassandra etc) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
- 6. CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 7. CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 8. CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 9. CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 10. CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 11. CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; authorization_attempts possible_fraud
- 12. Streaming == Manipulating Data in Flight • Join • Aggregate • Map • Reduce • Peek • Transform • (any arbitrary code) • Window • Transactions
- 13. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 14. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 15. What is a Distributed Log?
- 16. Shard on the way in Producing Services Kafka Consuming Services
- 17. Each shard is a queue Producing Services Kafka Consuming Services
- 18. Share Load / Fault Tollerant Producing Services Kafka Consuming Services
- 19. Retain datasets in the log. “Rewind & Replay”. Rewind & Replay
- 20. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 22. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 23. SELECT card_number, count(*) FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3; KSQL is SQL over Kafka Streams
- 24. Kafka Streams is just an API public static void main(String[] args) { StreamsBuilder builder = new StreamsBuilder(); builder.stream(”caterpillars") .map((k, v) -> coolTransformation(k, v)) .to(“butterflies”); new KafkaStreams(builder.build(), props()).start(); } 24
- 26. Windows / Retention – Handle Late Events KAFKA Buffer 5 mins Join by Key KStream orders = builder.stream(“Orders”); KStream payments = builder.stream(“Payments”); orders.join(payments, KeyValue::new, JoinWindows.of(10 * MIN)) .peek((key, pair) -> emailer.sendMail(pair));
- 28. A KTable is just a stream with infinite retention KStream orders = builder.stream(“Orders”); KStream payments = builder.stream(“Payments”); KTable customers = builder.table(“Customers”); orders.join(payments, EmailTuple::new, JoinWindows.of(10*MIN)) .join(customers, (tuple, cust) -> tuple.setCust(cust)) .peek((key, tuple) -> emailer.sendMail(tuple)); KAFKA Join Materialize a table in two lines of code! Dataset Moves to Client
- 29. Streaming is about 1. Processing data incrementally 2. Moving data to where it needs to be processed (quickly and efficiently)
- 30. Kafka: a Streaming Platform The Log ConnectorsConnectors Producer Consumer Streaming Engine
- 33. SOA / Microservices / EDA Customer Service Shipping Service
- 34. The Problem is DATA
- 35. Most services share the same core facts. Catalog Most services live in here
- 36. Buying an iPad with REST Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
- 37. Buying an iPad with Events Message Broker (Kafka) Notification Data is replicated (incrementally) Submit Order Order Created Customer Updated Orders Service Shipping Service Customer Service Webserver KAFKA
- 38. Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
- 39. Events for Data Locality Customer Updated Submit Order Order Created Data is replicated (incrementally)Orders Service Shipping Service Customer Service Webserver KAFKA
- 40. Events have two hats Notification Data replication
- 41. Apply Stream Processing Streaming is about manipulating data in flight
- 42. Kafka is a high throughput Messaging & Storage System Orders Service Shipping Service Customer Service KAFKA Web App
- 43. Kafka Streams API (or KSQL) An embedded API for data in flight Orders Service Shipping Service Customer Service KAFKA Web App
- 44. Streaming platforms optimize for moving data to code!?!
- 45. Add a more data intensive use case
- 46. A Scrollable Grid Orders Service Customer Service Web App Scrollable Grid Customer & Order in each row Many rows
- 47. Add caching -> problems of its own Orders Service Customer Service Web App Scrollable Grid
- 48. The Streaming Way Orders Service Customer Service KAFKA Web App Scrollable Grid (with RocksDB) KStreams API Select * from orders, customers where…
- 49. Streams & Tables Orders Service Customer Service KAFKA Web App Scrollable Grid Orders provide Notification Customers are replicated
- 50. Add Payments Service & Window Orders Service Customer Service KAFKA Web App Scrollable Grid Orders provide Notification Customers are replicated Payments Service Buffer / Window
- 51. Orders Service Customer Service KAFKA Web App Scrollable Grid Payments Service Query Runs INSIDE the webserver
- 52. Orders Service Customer Service KAFKA Web App Scrollable Grid Payments Service Events are stored in Kafka (e.g. Customers)
- 53. Orders Service Customer Service KAFKA Web App Scrollable Grid Payments Service Streaming is about Data Movement Materialized View
- 54. POST GET Load Balancer ORDERSORDERS OVTOPIC Order Validations KAFKA INVENTORY Orders Inventory Fraud Service Order Details Service Inventory Service (see previous figure) Order Created Order Validated Orders View Q in CQRS Orders Service C is CQRS Services in the Micro: Orders Service Find the code online!
- 55. Orders Customers Payments Stock Query Engine (Kstreams/KSQL) Larger Ecosystems HISTORICAL EVENT STREAMS
- 56. Kafka KAFKA New York Tokyo London Global / Disconnected Ecosystems
- 57. WIRED Principals • Windowed: Use an API built for async events • Immutable: Store events in an immutable log • Repeatable: Compose from side-effect free functions • Evolutionary: Be pluggable. Have data available in the log. • Data-Enabled: Push data to services where necessary
- 58. Makes it easier to evolve!
- 60. References • Confluent Microservices Series: https://www.confluent.io/blog/tag/microservices • Code examples: https://github.com/confluentinc/kafka-streams- examples