February 2016 HUG: Apache Apex (incubating): Stream Processing Architecture and Applications
2 likes5,278 views
The document provides an overview of the Apache Apex stream processing architecture, including its application programming model, advanced windowing, and dynamic partitioning features. It details how data tuples are processed through operators, the fault tolerance mechanisms in place, and the guarantees for data processing. Additionally, it highlights the ability to monitor and visualize real-time data processing within the platform.
February 2016 HUG: Apache Apex (incubating): Stream Processing Architecture and Applications
- 1. Pramod Immaneni <[email protected]> PPMC Member, Architect @DataTorrent Inc Dec 2nd, 2015 Stream Processing Architecture and Applications Apache Apex (incubating)
- 2. © 2015 DataTorrent Apex Platform Overview 2
- 3. © 2015 DataTorrent Apache Malhar Library 3
- 4. © 2015 DataTorrent Native Hadoop Integration 4 • YARN is the resource manager • HDFS used for storing any persistent state
- 5. © 2015 DataTorrent Application Programming Model 5 A Stream is a sequence of data tuples An Operator takes one or more input streams, performs computations & emits one or more output streams • Each Operator is YOUR custom business logic in java, or built-in operator from our open source library • Operator has many instances that run in parallel and each instance is single-threaded Directed Acyclic Graph (DAG) is made up of operators and streams Directed Acyclic Graph (DAG) Output StreamTuple Tuple er Operator er Operator er Operator er Operator
- 6. © 2015 DataTorrent Advanced Windowing Support 6 Application window Sliding window and tumbling window Checkpoint window No artificial latency
- 7. © 2015 DataTorrent Application Specification 7
- 8. © 2015 DataTorrent Partitioning and unification 8
- 9. © 2015 DataTorrent Advanced Partitioning 9
- 10. © 2015 DataTorrent Dynamic Partitioning 10 • Partitioning change while application is running ᵒ Change number of partitions at runtime based on stats ᵒ Determine initial number of partitions dynamically • Kafka operators scale according to number of kafka partitions ᵒ Supports re-distribution of state when number of partitions change ᵒ API for custom scaler or partitioner unifiers not shown 1a 2a 1b 2b 3 2b 1b 2c 3 2a 2d 1a 2b 1b 2c 3b 2a 2d 3a1a
- 11. © 2015 DataTorrent How tuples are partitioned 11 • Tuple hashcode and mask used to determine destination partition ᵒ Mask picks the last n bits of the hashcode of the tuple ᵒ hashcode method can be overridden • StreamCodec can be used to specify custom hashcode for tuples ᵒ Can also be used for specifying custom serialization tuple: { Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x11) Partition 00 1 01 2 10 3 11 4
- 12. © 2015 DataTorrent Custom partitioning 12 • Custom distribution of tuples ᵒ E.g.. Broadcast tuple:{ Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x00) Partition 00 1 00 2 00 3 00 4
- 13. © 2015 DataTorrent Fault Tolerance 13 • Operator state is checkpointed to a persistent store ᵒ Automatically performed by engine, no additional work needed by operator ᵒ In case of failure operators are restarted from checkpoint state ᵒ Frequency configurable per operator ᵒ Asynchronous and distributed by default ᵒ Default store is HDFS • Automatic detection and recovery of failed operators ᵒ Heartbeat mechanism • Buffering mechanism to ensure replay of data from recovered point so that there is no loss of data • Application master state checkpointed
- 14. © 2015 DataTorrent Processing Guarantees 14 Atleast once • On recovery data will be replayed from a previous checkpoint ᵒ Messages will not be lost ᵒ Default mechanism and is suitable for most applications • Can be used in conjunction with following to ensure data is written once to store in case of fault recovery ᵒ Transactions with meta information, Rewinding output, Feedback from external entity, Idempotent operations Atmost once • On recovery the latest data is made available to operator ᵒ Useful in use cases where some data loss is acceptable and latest data is sufficient Exactly once • Operators checkpointed every window ᵒ Can be combined with transactional mechanisms to ensure end-to-end exactly once behavior
- 15. © 2015 DataTorrent Stream Locality 15 • By default operators are deployed in containers (processes) randomly on different nodes across the Hadoop cluster • Custom locality for streams ᵒ Rack local: Data does not traverse network switches ᵒ Node local: Data is passed via loopback interface and frees up network bandwidth ᵒ Container local: Messages are passed via in memory queues between operators and does not require serialization ᵒ Thread local: Messages are passed between operators in a same thread equivalent to calling a subsequent function on the message
- 16. © 2015 DataTorrent Data Processing Pipeline Example App Builder 16
- 17. © 2015 DataTorrent Monitoring Console Logical View 17
- 18. © 2015 DataTorrent Monitoring Console Physical View 18
- 19. © 2015 DataTorrent Real-Time Dashboards Real Time Visualization 19
- 20. © 2015 DataTorrent Resources 20 Apache Apex Community Page - http://apex.incubator.apache.org/
- 21. © 2015 DataTorrent We Need Your Vote (Today) 21 Introducing Apache Apex - Not Just Another Stream Processing Platform Next Gen Big Data Analytics with Apache Apex Enterprise-grade streaming under 2ms on Hadoop
- 22. End 22
- 23. © 2015 DataTorrent Extra Slides
- 24. © 2015 DataTorrent Partitioning and Scaling Out 24 • Operators can be dynamically scaled • Flexible Streams split • Parallel partitioning • MxN partitioning • Unifiers
- 25. © 2015 DataTorrent Fault Tolerance Overview 25 Stateful Fault Tolerance Processing Semantics Data Locality Supported out of the box – Application state – Application master state – No data loss Automatic recovery Lunch test Buffer server At least once At most once Exactly once Stream locality for placement of operators Rack local – Distributed deployment Node local – Data does not traverse NIC Container local – Data doesn’t need to be serialized Thread local – Operators run in same thread Data locality
- 26. © 2015 DataTorrent Machine Data Application Logical View 26
- 27. © 2015 DataTorrent Machine Data Application Physical View 27