Windowing data in big data streams
1 like1,934 views
The document discusses windowing data in big data streams. It describes how windowing allows analyzing data based on a time interval, such as the number of invalid login attempts in the last 5 minutes. It then covers different streaming data technologies like Spark Streaming, Spark Structured Streaming, Kafka Streams, Flink, and Akka Streams that can be used to perform windowing. Each technology handles aspects such as event time, watermarks, triggers, and state management slightly differently. In the end, the document notes that windowing is one part of streaming and other factors like processing guarantees and state management also need consideration.
Windowing data in big data streams
- 1. WINDOWING DATA IN BIG DATA STREAMS ADAM WARSKI, WOLVESSUMMIT
- 2. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI BIG DATA? FAST DATA? ▸ What is big data? ▸ Shift of focus ▸ Processing speed ▸ Fast data -> streaming
- 3. A TYPE OF DATA PROCESSING ENGINE THAT IS DESIGNED WITH INFINITE DATA SETS IN MIND Tyler Akidau, Google ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI WHAT IS STREAMING?
- 4. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI WINDOWING ▸ Time becomes the focus point ▸ How many invalid password errors where there in the last 5 minutes ▸ During which 30-minute window did we get most traffic? ▸ What’s the average 5-minute speed on a section of a highway throughout the day?
- 5. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI HOW TO DO STREAMING? WITH WINDOWS? ▸ Many possibilities: ▸ Spark Streaming ▸ Spark Structured Streaming ▸ Kafka Streams ▸ Flink ▸ Akka Streams ▸ …
- 6. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI WHICH ONE TO CHOOSE? LET’S FIND OUT
- 7. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI /ME ▸ coder @ ▸ Lightbend, Confluent, Datastax consulting partner ▸ mainly Scala ▸ open-source: MacWire, ElasticMQ, Quicklens, … ▸ http://www.warski.org / @adamwarski
- 8. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI WHAT’S THE TIME? ▸ How to associate time with an event: ▸ event time: “logical”, data-dependent ▸ ingestion time: when the event entered the system ▸ processing time: when the event is being processed
- 9. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI TYPES OF WINDOWS ▸ Time-based ▸ fixed/tumbling ▸ sliding ▸ Session-based time
- 10. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI OUT-OF-ORDER: WATERMARKS, LATENESS ▸ Windows GC ▸ At some point, enough is enough ▸ Watermark: ▸ all events before X have been observed ▸ heuristics
- 11. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI TRIGGERS ▸ When to emit window results ▸ Watermark progress ▸ Event time progress ▸ Processing time progress ▸ Punctuations
- 12. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI ACCUMULATION OF RESULTS ▸ If we trigger many times … ▸ discard ▸ accumulate ▸ retract & accumulate
- 13. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI FINALLY … HOW TO MANIPULATE THE DATA ▸ map, flatMap, filter … ▸ stateful computation ▸ fold, reduce ▸ past-dependent operations ▸ where to store the state
- 14. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SUMMING UP ▸ Event/ingestion/processing time ▸ Tumbling/sliding/session windows ▸ Watermarks ▸ Triggers ▸ Accumulation of results ▸ State management
- 15. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SPARK STREAMING ▸ Micro-batches (DStream) ▸ .window() API: ▸ tumbling/sliding windows ▸ only processing time ▸ no watermarks ▸ triggers at the end of the window ▸ state persisted in cluster (e.g. updateStateByKey())
- 16. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SPARK STREAMING - WHY BOTHER? ▸ Popular ▸ Not only streaming ▸ ML ▸ SQL ▸ GraphX ▸ but …
- 17. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SPARK STRUCTURED STREAMING ▸ Alpha in Spark 2.0 ▸ Micro-batches not exposed ▸ groupBy(window(…)) ▸ Event-time support ▸ No watermarks, session windows (2.1?) ▸ Trigger: processing time; outputs changed windows ▸ Exactly-once processing*
- 18. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI FLINK ▸ Mostly with keyed streams (parallelism) ▸ TimeCharacteristic: event/ingestion/processing ▸ TimestampAssigner: also generates watermarks ▸ WindowAssigner: arbitrary, built-in tumbling, sliding, session ▸ Trigger: event/processing time, count, single/continuous ▸ Window function: fold/reduce/with-kv-state ▸ Exactly-once* / at-least-once
- 19. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI KAFKA STREAMS ▸ State: Kafka topics/local key-value backed by a topic for resiliency ▸ Watermarks: no, but windows are retained for 1 day ▸ Time: event/ingestion/processing; TimestampExtractor ▸ Tumbling/sliding windows ▸ Trigger: after every element ▸ aggregate by key&window into an ever-updating KTable ▸ At-least-once
- 20. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI AKKA STREAMS ▸ Single-node, no clustering ▸ No OOTB support, but quite easy to implement: ▸ Windows: arbitrary, assign windows to each element ▸ Trigger: only window-close ▸ State: local ▸ Watermarks: can be implemented
- 21. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SUMMING UP ▸ Spark: widely used, some features missing ▸ Flink: versatile ▸ Kafka: simple model ▸ Akka: single-node
- 22. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SUMMING UP ▸ Windowing is just one of the aspects ▸ Other: ▸ State management ▸ Work distribution ▸ Processing guarantees
- 23. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI SUMMING UP ▸ Other stream processing systems out there! ▸ Apache Storm ▸ Google Cloud Dataflow ▸ Amazon Kinesis ▸ Apache Beam ▸ …
- 24. ADAM WARSKI, SOFTWAREMILL, @ADAMWARSKI LINKS ▸ Streaming 101 & 102: ▸ https://www.oreilly.com/ideas/the-world-beyond-batch- streaming-101 ▸ https://www.oreilly.com/ideas/the-world-beyond-batch- streaming-102 ▸ https://softwaremill.com/windowing-data-in-akka-streams/