Kafka Summit NYC 2017 - Building Advanced Streaming Applications using the Latest from Apache Flink & Kafka
1 like888 views
The document discusses the latest features of Apache Flink including rescalable state, asynchronous I/O support, streaming SQL, flexible deployment options, and enhanced security. It provides examples of how these features enable advanced event-driven applications like a trading engine that processes stock quotes in real-time and makes trading decisions based on complex time-oriented rules.
![Asynchronous I/O Support
// create the original stream
val stream: DataStream[String] = ...
// apply the async I/O transformation
val resultStream: DataStream[(String, String)] =
AsyncDataStream.unorderedWait(
input = stream,
asyncFunction = new AsyncDatabaseRequest(),
timeout = 1000,
timeUnit = TimeUnit.MILLISECONDS,
concurrentRequests = 100)](https://image.slidesharecdn.com/advancedstreaming-flink-170524225825/85/Kafka-Summit-NYC-2017-Building-Advanced-Streaming-Applications-using-the-Latest-from-Apache-Flink-Kafka-15-320.jpg)
![Asynchronous I/O Support
class AsyncDatabaseRequest extends AsyncFunction[String, (String, String)] {
override def asyncInvoke(str: String, asyncCollector: AsyncCollector[(String, String)]): Unit = {
// issue the asynchronous request, receive a future for the result
val resultFuture: Future[String] = client.query(str)
// set the callback to be executed once the request by the client is complete
// the callback simply forwards the result to the collector
resultFuture.onSuccess {
case result: String => asyncCollector.collect(Iterable((str, result)));
}
}
}](https://image.slidesharecdn.com/advancedstreaming-flink-170524225825/85/Kafka-Summit-NYC-2017-Building-Advanced-Streaming-Applications-using-the-Latest-from-Apache-Flink-Kafka-16-320.jpg)
Kafka Summit NYC 2017 - Building Advanced Streaming Applications using the Latest from Apache Flink & Kafka
- 1. 1 Jamie Grier @jamiegrier data-artisans.com Building Advanced Streaming Applications: The Latest and Greatest of Flink and Kafka
- 2. 2 Original creators of Apache Flink® Providers of the dA Platform, a supported Flink distribution
- 3. The Latest Features - Quick Overview ▪ Rescalable State ▪ Async I/O Support ▪ Streaming SQL (Dynamic Tables) ▪ Flexible Deployment Options ▪ Enhanced Security
- 4. The Latest Features ▪ ProcessFunction API ▪ Queryable State API ▪ Excellent support for advanced applications that are: ▪ Flexible ▪ Stateful ▪ Event Driven ▪ Time Driven
- 5. Rescalable State ▪ Separates state parallelism from task parallelism ▪ Enables autoscaling integrations while maintaining stateful computations ▪ Handled efficiently via key groups
- 6. Rescalable State Map Filter Window State State State Source Sink State is partitioned by key
- 7. Rescalable State Source Sink Map Filter Window State State State Map Filter Window State State State State is partitioned by key
- 8. Rescalable State Source Sink Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State State is partitioned by key
- 9. Rescalable State Source Sink Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State Map Filter Window State State State State is partitioned by key
- 10. Asynchronous I/O Support ▪ Make aynchronous calls to external services from streaming job ▪ Efficiently keeps configurable number of asynchronous calls in flight ▪ Correctly handles failure scenarios - restarts failed async calls, etc
- 12. Asynchronous I/O Support Little’s Law: throughput = occupancy / latency
- 13. Asynchronous I/O Support a b c d b c d x a a b b a Sync. I/O Async. I/O database database sendRequest(x) x receiveResponse(x) wait
- 15. Asynchronous I/O Support // create the original stream val stream: DataStream[String] = ... // apply the async I/O transformation val resultStream: DataStream[(String, String)] = AsyncDataStream.unorderedWait( input = stream, asyncFunction = new AsyncDatabaseRequest(), timeout = 1000, timeUnit = TimeUnit.MILLISECONDS, concurrentRequests = 100)
- 16. Asynchronous I/O Support class AsyncDatabaseRequest extends AsyncFunction[String, (String, String)] { override def asyncInvoke(str: String, asyncCollector: AsyncCollector[(String, String)]): Unit = { // issue the asynchronous request, receive a future for the result val resultFuture: Future[String] = client.query(str) // set the callback to be executed once the request by the client is complete // the callback simply forwards the result to the collector resultFuture.onSuccess { case result: String => asyncCollector.collect(Iterable((str, result))); } } }
- 17. Streaming SQL: Querying Dynamic Tables
- 18. Streaming SQL: Stream to Table Append Mode
- 19. Streaming SQL: Stream to Table Update Mode
- 21. Streaming SQL: Windowed Queries
- 22. Streaming SQL: Table to Stream Redo Stream
- 23. Streaming SQL: Table to Stream Undo / Redo Stream
- 24. Flexible Deployment Options ▪ YARN ▪ Mesos ▪ Docker Swarm ▪ Kubernetes
- 25. Flexible Deployment Options ▪ DC/OS ▪ Amazon EMR ▪ Google Dataproc
- 26. Enhanced Security ▪ SSL ▪ Kerberos ▪ Kafka ▪ Zookeeper ▪ Hadoop
- 27. Advanced Event-Driven Applications ▪ ProcessFunction API ▪ Queryable State API ▪ Excellent support for advanced applications that are: ▪ Flexible ▪ Stateful ▪ Event Driven ▪ Time Driven
- 28. Example: FlinkTrade ▪ Overall Requirements: ▪ Consume “starting position” and “quote” streams ▪ Process complex, time-oriented, trading rules ▪ Trade out of positions to our advantage if possible ▪ Provide a dashboard of currently held positions to traders and asset managers ▪ Complex Rules: ▪ We only make trades where the Bid Price is above our current Ask Price ▪ When a trade is made we increase our Ask Price — looking to optimize our profits ▪ Positions have a set time-to-live until we try to trade out of them more aggressively by decreasing the Ask Price over time
- 29. Example: FlinkTrade Quotes Starting Positions Trading Engine Trades Positions SYMBOL SHARES BUY PRICE ASK PRICE LAST TRADE PRICE Profit AAPL 10,000 140.40 140.50 140.40 $10,921.00 GOOG 20,000 846.81 846.91 846.81 $12,021.00 TWTR 8,000 15.12 15.22 15.12 $4,032.00 ● Event Driven Processing ● Complex Trading Rules ● Time Based Logic Trader Dashboard
- 30. Example: FlinkTrade Quotes Starting Positions Trades Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions Trading Engine Positions SYMBOL SHARES BUY PRICE ASK PRICE LAST TRADE PRICE Profit AAPL 10,000 140.40 140.50 140.40 $10,921.00 GOOG 20,000 846.81 846.91 846.81 $12,021.00 TWTR 8,000 15.12 15.22 15.12 $4,032.00
- 31. Example: FlinkTrade Let’s look at the code