Introduction to Apache Apex and writing a big data streaming application
Download as PPTX, PDF2 likes829 views
The document introduces Apache Apex, a platform for developing scalable and fault-tolerant distributed applications on Hadoop. It emphasizes the use of open-source technologies and collaboration among industry experts for professional development. Key features include support for streaming and batch processing, a robust operator library, fault tolerance mechanisms, and dynamic partitioning capabilities.
Introduction to Apache Apex and writing a big data streaming application
- 1. Pramod Immaneni <[email protected]> Apache Apex PMC, Architect @DataTorrent Inc May 7th, 2016 The next generation native Hadoop platform Introduction to Apache Apex
- 2. Guru Virtues Organizer and sponsor
- 3. Guru Virtues 3 • Focus on open source new age technology software development • Architects with varied Industry experience including Telecom, R&D, Technology Consulting, DevOps and Product Development • Architects with deep product exposures to new age technologies like Virtualization, Cloud Computing, Embedded Development, Advanced Analytics, Hadoop/Spark Computing, Sales Force • Research & Development Data Center in Tyngsboro, MA
- 4. Philosophy 4 Collaborate Skills Development by bringing specialized experts together for deeper Knowledge Sharing to enable Professional Development and deliver cutting edge solutions for businesses using open source technologies Develop rich-man software on poor-man hardware
- 5. Apache Apex
- 6. What is Apex 6 • Platform and runtime engine that enables development of scalable and fault-tolerant distributed applications • Hadoop native • Process streaming or batch big data • High throughput and low latency • Library of commonly needed business logic • Write any custom business logic in your application
- 7. Applications on Apex 7 • Distributed processing • Application logic broken into components called operators that run in a distributed fashion across your cluster • Scalable • Operators can be scaled up or down at runtime according to the load and SLA • Fault tolerant • Automatically recover from node outages without having to reprocess from beginning • State is preserved • Long running applications • Operators • Use library to build applications quickly • Write your own in Java using the API • Operational insight – DataTorrent RTS • See how each operator is performing and even record data
- 9. Apex Operator Library - Malhar 9
- 10. Native Hadoop Integration 10 • YARN is the resource manager • HDFS used for storing any persistent state
- 11. Application Development Model 11 A Stream is a sequence of data tuples A typical Operator takes one or more input streams, performs computations & emits one or more output streams • Each Operator is YOUR custom business logic in java, or built-in operator from our open source library • Operator has many instances that run in parallel and each instance is single-threaded Directed Acyclic Graph (DAG) is made up of operators and streams Directed Acyclic Graph (DAG) Output Stream Tupl e Tupl e er Operator er Operator er Operator er Operator er Operator er Operator
- 12. Advanced Windowing Support 12 Application window Sliding window and tumbling window Checkpoint window No artificial latency
- 14. Operators 14
- 16. Partitioning and unification 16 NxM PartitionsUnifier 0 1 2 3 Logical DAG 0 1 2 1 1 Unifier 1 20 Logical Diagram Physical Diagram with operator 1 with 3 partitions 0 Unifier 1a 1b 1c 2a 2b Unifier 3 Physical DAG with (1a, 1b, 1c) and (2a, 2b): No bottleneck Unifier Unifier0 1a 1b 1c 2a 2b Unifier 3 Physical DAG with (1a, 1b, 1c) and (2a, 2b): Bottleneck on intermediate Unifier
- 17. Advanced Partitioning 17 0 1a 1b 2 3 4Unifier Physical DAG 0 4 3a2a1a 1b 2b 3b Unifier Physical DAG with Parallel Partition Parallel Partition Container uopr uopr1 uopr2 uopr3 uopr4 uopr1 uopr2 uopr3 uopr4 dopr dopr doprunifier unifier unifier unifier Container Container NICNIC NICNIC NIC Container NIC Logical Plan Execution Plan, for N = 4; M = 1 Execution Plan, for N = 4; M = 1, K = 2 with cascading unifiers Cascading Unifiers 0 1 2 3 4 Logical DAG
- 18. Dynamic Partitioning 18 • Partitioning change while application is running ᵒ Change number of partitions at runtime based on stats ᵒ Determine initial number of partitions dynamically • Kafka operators scale according to number of kafka partitions ᵒ Supports re-distribution of state when number of partitions change ᵒ API for custom scaler or partitioner 2b 2c 3 2a 2d 1b 1a1a 2a 1b 2b 3 1a 2b 1b 2c 3b 2a 2d 3a Unifiers not shown
- 19. How tuples are partitioned 19 • Tuple hashcode and mask used to determine destination partition ᵒ Mask picks the last n bits of the hashcode of the tuple ᵒ hashcode method can be overridden • StreamCodec can be used to specify custom hashcode for tuples ᵒ Can also be used for specifying custom serialization tuple: { Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x11) Partition 00 1 01 2 10 3 11 4
- 20. Custom partitioning 20 • Custom distribution of tuples ᵒ E.g.. Broadcast tuple:{ Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x00) Partition 00 1 00 2 00 3 00 4
- 21. Fault Tolerance 21 • Operator state is checkpointed to a persistent store ᵒ Automatically performed by engine, no additional work needed by operator ᵒ In case of failure operators are restarted from checkpoint state ᵒ Frequency configurable per operator ᵒ Asynchronous and distributed by default ᵒ Default store is HDFS • Automatic detection and recovery of failed operators ᵒ Heartbeat mechanism • Buffering mechanism to ensure replay of data from recovered point so that there is no loss of data • Application master state checkpointed
- 22. Processing Guarantees - Recovery 22 Atleast once • On recovery data will be replayed from a previous checkpoint ᵒ Messages will not be lost ᵒ Default mechanism and is suitable for most applications • Can be used in conjunction with following mechanisms to achieve exactly-once behavior in fault recovery scenarios ᵒ Transactions with meta information, Rewinding output, Feedback from external entity, Idempotent operations Atmost once • On recovery the latest data is made available to operator ᵒ Useful in use cases where some data loss is acceptable and latest data is sufficient Exactly once • At least once + state recovery + operator logic to achieve end-to-end exactly once
- 23. Stream Locality 23 • By default operators are deployed in containers (processes) randomly on different nodes across the Hadoop cluster • Custom locality for streams ᵒ Rack local: Data does not traverse network switches ᵒ Node local: Data is passed via loopback interface and frees up network bandwidth ᵒ Container local: Messages are passed via in memory queues between operators and does not require serialization ᵒ Thread local: Messages are passed between operators in a same thread equivalent to calling a subsequent function on the message
- 24. Data Processing Pipeline Example App Builder 24
- 27. Real-Time Dashboards Real Time Visualization 27
- 28. Resources 28 • Apache Apex website - http://apex.apache.org/ • Subscribe - http://apex.apache.org/community.html • Download - http://apex.apache.org/downloads.html • Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex • Facebook - https://www.facebook.com/ApacheApex/ • Meetup - http://www.meetup.com/topics/apache-apex • Free Enterprise License for Startups - https://www.datatorrent.com/product/startup- accelerator/
- 29. We Are Hiring 29 • [email protected] • Developers/Architects • QA Automation Developers • Information Developers • Build and Release • Community Leaders
- 30. End 30