Real time analytics with Kafka and SparkStreaming

Editor's Notes

• #2: Canonical architecture of real-time stream processing
• #3: It is easy to get carried away, because real-time sounds cool. But it is totally fair if you are skeptical about the business value of real time stream processing. After all, this is not a new concept. There have been tools provided by traditional enterprise software vendors to do real-time data processing. They were formerly known as “Complex Event Processing” or CEP systems. In fact, even now the traditional vendors refer to their real-time stream processing tools as CEP systems. If CEP systems have been around, but never really took off, are we just seeing the
• #6: Lets look at an end to end architecture of putting together open source tools to do real time stream processing. Lets start with the sources of data.
• #7: You want to write this data to a reliable high-throughput low latency messaging system, Kafka and Flume are popular choices, but there are many options out there, like ActiveMQ, RabbitMQ,etc. Kafka is the system that is gaining the most popularity right now. ====== With this architecture, the real-time processed data only gets leveraged when the next application query comes in. But often you want to take some action based on the real-time analysis of your data. For proactive actions, write relevant events out to Kafka. Again, based on yoru stream processign engine you will find libraries that make this easy. You can have an application that is continusouly listeing on your event queue, and can issues alerts, emails, etc
• #8: A stream processing system like Spark Streaming can then read your data streams from the messaging system. Filter Enrich or embellish your data with relevant metadata Transform Compute statistics based on moving windows of time Feature Engineering + Predictive Analytics … and much more
• #9: Almost always, you want to take your full fidelity raw data, and put it in HDFS, or an object store if your are running in the cloud. The raw data can then be used in batch jobs where you may want to do deep complex processing that can not be done in a streaming fashion. Or you may have a team of data scientists who may want to explore the data and uncover new insights. Why the dotted line: how you dump your data to HDFS depends on your messaging system. Almost all messaging systems will provide a way to transfer your data to HDFS
• #10: All this real-time processing is great, but not very useful if you can not serve the processed data to your application in real-time. Your need a system that can enable a lot of fast reads and writes. That is where NoSql stores come in. There are many choices here. Hbase, Cassnadra and MongoDb are popular choices. All those end applications Also, for most stream procsssing engine and NoSql store pairs, there are libraries available that make it easy to read from or write to your NoSql store from the stream processing engine: for example, the SparkOnHbase library makes it easy to write to Hbase from spark streamign jobs.
• #11: Another common scenario is indexing your data, in real-time, into a search system. This is great if the data your are dealing with is textual data. There are libararies that enable real-time indexing of your data in your stream proocessing engine, and writing it to a Search Engine.
• #12: Now the data is ready to be queried by your application. This is a very common and popular architecture, and I am guessing this is in keeping with what most of you would have expected.
• #13: Again, write your processed output to HDFS. Again, why the dotter arrow. Weather or not you need to dump data to HDFS depends upon your serving system of choice. If you write it to Hbase, you may not need to duplicate it in HDFS. But if you are indexing the data in search or writing to a system like Redis, you may want to also write the processed otuptut to HDFS. Why? If nothing else, for auditing purposes. Errors will happen. And you may need to go back and audit what was done in your stream processing engine. Hence, put the data in hdfs and keep it there are some amount of time.
• #14: With this architecture, the real-time processed data only gets leveraged when the next application query comes in. But often you want to take some action based on the real-time analysis of your data. For proactive actions, write relevant events out to Kafka. Again, based on yoru stream processign engine you will find libraries that make this easy. You can have an application that is continusouly listeing on your event queue, and can issues alerts, emails, etc
• #15: By writing it to a message queue, you enable multiple downstream applications to consume the data as its produced, including enabling furthur processing of your data with a stream processing engine. Such multi-stage architectures, where you cosnume from say Kafka, process the data, produce a new stream in Kafka, and process
• #26: For moment like these, streaming systems provide the capability to rewind, at least they should.