SlideShare a Scribd company logo
Pramod Immaneni <pramod@datatorrent.com>
PPMC Member, Architect @DataTorrent Inc
Dec 2nd, 2015
Stream Processing Architecture and Applications
Apache Apex (incubating)
© 2015 DataTorrent
Apex Platform Overview
2
© 2015 DataTorrent
Apache Malhar Library
3
© 2015 DataTorrent
Native Hadoop Integration
4
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state
© 2015 DataTorrent
Application Programming Model
5
 A Stream is a sequence of data tuples
 An Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output StreamTuple Tuple
er
Operator
er
Operator
er
Operator
er
Operator
© 2015 DataTorrent
Advanced Windowing Support
6
 Application window
 Sliding window and tumbling window
 Checkpoint window
 No artificial latency
© 2015 DataTorrent
Application Specification
7
© 2015 DataTorrent
Partitioning and unification
8
© 2015 DataTorrent
Advanced Partitioning
9
© 2015 DataTorrent
Dynamic Partitioning
10
• Partitioning change while application is running
ᵒ Change number of partitions at runtime based on stats
ᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitions
ᵒ Supports re-distribution of state when number of partitions change
ᵒ API for custom scaler or partitioner
unifiers not shown
1a 2a
1b 2b
3
2b
1b 2c
3
2a
2d
1a 2b
1b 2c 3b
2a
2d
3a1a
© 2015 DataTorrent
How tuples are partitioned
11
• Tuple hashcode and mask used to determine destination partition
ᵒ Mask picks the last n bits of the hashcode of the tuple
ᵒ hashcode method can be overridden
• StreamCodec can be used to specify custom hashcode for tuples
ᵒ Can also be used for specifying custom serialization
tuple: {
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x11)
Partition
00 1
01 2
10 3
11 4
© 2015 DataTorrent
Custom partitioning
12
• Custom distribution of tuples
ᵒ E.g.. Broadcast
tuple:{
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x00)
Partition
00 1
00 2
00 3
00 4
© 2015 DataTorrent
Fault Tolerance
13
• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint state
ᵒ Frequency configurable per operator
ᵒ Asynchronous and distributed by default
ᵒ Default store is HDFS
• Automatic detection and recovery of failed operators
ᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so
that there is no loss of data
• Application master state checkpointed
© 2015 DataTorrent
Processing Guarantees
14
Atleast once
• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lost
ᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following to ensure data is written
once to store in case of fault recovery
ᵒ Transactions with meta information, Rewinding output, Feedback from
external entity, Idempotent operations
Atmost once
• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is
sufficient
Exactly once
• Operators checkpointed every window
ᵒ Can be combined with transactional mechanisms to ensure end-to-end
exactly once behavior
© 2015 DataTorrent
Stream Locality
15
• By default operators are deployed in containers (processes) randomly
on different nodes across the Hadoop cluster
• Custom locality for streams
ᵒ Rack local: Data does not traverse network switches
ᵒ Node local: Data is passed via loopback interface and frees up network
bandwidth
ᵒ Container local: Messages are passed via in memory queues between
operators and does not require serialization
ᵒ Thread local: Messages are passed between operators in a same thread
equivalent to calling a subsequent function on the message
© 2015 DataTorrent
Data Processing Pipeline Example
App Builder
16
© 2015 DataTorrent
Monitoring Console
Logical View
17
© 2015 DataTorrent
Monitoring Console
Physical View
18
© 2015 DataTorrent
Real-Time Dashboards
Real Time Visualization
19
© 2015 DataTorrent
Resources
20
Apache Apex Community Page - http://apex.incubator.apache.org/
© 2015 DataTorrent
We Need Your Vote (Today)
21
Introducing Apache Apex - Not Just Another Stream Processing Platform
Next Gen Big Data Analytics with Apache Apex
Enterprise-grade streaming under 2ms on Hadoop
End
22
© 2015 DataTorrent
Extra Slides
© 2015 DataTorrent
Partitioning and Scaling Out
24
• Operators can be dynamically
scaled
• Flexible Streams split
• Parallel partitioning
• MxN partitioning
• Unifiers
© 2015 DataTorrent
Fault Tolerance Overview
25
Stateful Fault Tolerance Processing Semantics Data Locality
 Supported out of the box
– Application state
– Application master state
– No data loss
 Automatic recovery
 Lunch test
 Buffer server
 At least once
 At most once
 Exactly once
 Stream locality for placement of
operators
 Rack local – Distributed
deployment
 Node local – Data does
not traverse NIC
 Container local – Data
doesn’t need to be
serialized
 Thread local – Operators
run in same thread
 Data locality
© 2015 DataTorrent
Machine Data Application
Logical View
26
© 2015 DataTorrent
Machine Data Application
Physical View
27

More Related Content

What's hot (20)

PPTX
Smart Partitioning with Apache Apex (Webinar)
Apache Apex
 
PPTX
Stream Processing with Apache Apex
Pramod Immaneni
 
PDF
Apex as yarn application
Chinmay Kolhatkar
 
PDF
Extending The Yahoo Streaming Benchmark to Apache Apex
Apache Apex
 
PPTX
Introduction to Apache Apex
Apache Apex
 
PPTX
Apache Apex Introduction with PubMatic
Apache Apex
 
PPTX
Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming
Apache Apex
 
PDF
Introduction to Apache Apex - CoDS 2016
Bhupesh Chawda
 
PDF
Integrating Apache NiFi and Apache Apex
Apache Apex
 
PPTX
Fault Tolerance and Processing Semantics in Apache Apex
Apache Apex Organizer
 
PPTX
Architectual Comparison of Apache Apex and Spark Streaming
Apache Apex
 
PPTX
University program - writing an apache apex application
Akshay Gore
 
PDF
Windowing in apex
Yogi Devendra Vyavahare
 
PPTX
Introduction to Real-Time Data Processing
Apache Apex
 
PPTX
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Apache Apex
 
PDF
Building your first aplication using Apache Apex
Yogi Devendra Vyavahare
 
PPTX
Intro to Apache Apex - Next Gen Native Hadoop Platform - Hackac
Apache Apex
 
PPTX
Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)
Apache Apex
 
PDF
Developing streaming applications with apache apex (strata + hadoop world)
Apache Apex
 
PDF
Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra Tagare
Apache Apex
 
Smart Partitioning with Apache Apex (Webinar)
Apache Apex
 
Stream Processing with Apache Apex
Pramod Immaneni
 
Apex as yarn application
Chinmay Kolhatkar
 
Extending The Yahoo Streaming Benchmark to Apache Apex
Apache Apex
 
Introduction to Apache Apex
Apache Apex
 
Apache Apex Introduction with PubMatic
Apache Apex
 
Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming
Apache Apex
 
Introduction to Apache Apex - CoDS 2016
Bhupesh Chawda
 
Integrating Apache NiFi and Apache Apex
Apache Apex
 
Fault Tolerance and Processing Semantics in Apache Apex
Apache Apex Organizer
 
Architectual Comparison of Apache Apex and Spark Streaming
Apache Apex
 
University program - writing an apache apex application
Akshay Gore
 
Windowing in apex
Yogi Devendra Vyavahare
 
Introduction to Real-Time Data Processing
Apache Apex
 
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Apache Apex
 
Building your first aplication using Apache Apex
Yogi Devendra Vyavahare
 
Intro to Apache Apex - Next Gen Native Hadoop Platform - Hackac
Apache Apex
 
Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)
Apache Apex
 
Developing streaming applications with apache apex (strata + hadoop world)
Apache Apex
 
Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra Tagare
Apache Apex
 

Similar to Apache Apex - Hadoop Users Group (20)

PPTX
Intro to Apache Apex - Next Gen Platform for Ingest and Transform
Apache Apex
 
PDF
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
In-Memory Computing Summit
 
PDF
Introduction to Apache Apex by Thomas Weise
Big Data Spain
 
PDF
Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
PPSX
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex (Hadoop)
Apache Apex
 
PPTX
Hadoop Summit SJ 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
PPTX
Next Gen Big Data Analytics with Apache Apex
DataWorks Summit/Hadoop Summit
 
PPTX
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Dataconomy Media
 
PPTX
Big Data Berlin v8.0 Stream Processing with Apache Apex
Apache Apex
 
PPTX
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations
Apache Apex
 
PDF
Artur Borycki - Beyond Lambda - how to get from logical to physical - code.ta...
AboutYouGmbH
 
PPTX
Intro to Apache Apex @ Women in Big Data
Apache Apex
 
PDF
Fast SQL on Hadoop, Really?
DataWorks Summit
 
PPTX
Hive Performance Dataworks Summit Melbourne February 2019
alanfgates
 
PPTX
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
PDF
Real-time Stream Processing using Apache Apex
Apache Apex
 
PPTX
Spark+flume seattle
Hari Shreedharan
 
ODP
Zero Downtime JEE Architectures
Alexander Penev
 
PPTX
The Past, Present, and Future of Apache Flink
Aljoscha Krettek
 
PPTX
Flink Forward Berlin 2018: Aljoscha Krettek & Till Rohrmann - Keynote: "A Yea...
Flink Forward
 
Intro to Apache Apex - Next Gen Platform for Ingest and Transform
Apache Apex
 
IMCSummit 2015 - Day 1 IT Business Track - Designing a Big Data Analytics Pla...
In-Memory Computing Summit
 
Introduction to Apache Apex by Thomas Weise
Big Data Spain
 
Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex (Hadoop)
Apache Apex
 
Hadoop Summit SJ 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
Next Gen Big Data Analytics with Apache Apex
DataWorks Summit/Hadoop Summit
 
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Dataconomy Media
 
Big Data Berlin v8.0 Stream Processing with Apache Apex
Apache Apex
 
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations
Apache Apex
 
Artur Borycki - Beyond Lambda - how to get from logical to physical - code.ta...
AboutYouGmbH
 
Intro to Apache Apex @ Women in Big Data
Apache Apex
 
Fast SQL on Hadoop, Really?
DataWorks Summit
 
Hive Performance Dataworks Summit Melbourne February 2019
alanfgates
 
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
Real-time Stream Processing using Apache Apex
Apache Apex
 
Spark+flume seattle
Hari Shreedharan
 
Zero Downtime JEE Architectures
Alexander Penev
 
The Past, Present, and Future of Apache Flink
Aljoscha Krettek
 
Flink Forward Berlin 2018: Aljoscha Krettek & Till Rohrmann - Keynote: "A Yea...
Flink Forward
 
Ad

Recently uploaded (20)

PDF
Revenue streams of the Wazirx clone script.pdf
aaronjeffray
 
PDF
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
PPTX
OpenChain @ OSS NA - In From the Cold: Open Source as Part of Mainstream Soft...
Shane Coughlan
 
PDF
Online Queue Management System for Public Service Offices in Nepal [Focused i...
Rishab Acharya
 
PDF
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
PPTX
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
PDF
The 5 Reasons for IT Maintenance - Arna Softech
Arna Softech
 
PPTX
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pptx
Varsha Nayak
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PPTX
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
PDF
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
PDF
Wondershare PDFelement Pro Crack for MacOS New Version Latest 2025
bashirkhan333g
 
PDF
[Solution] Why Choose the VeryPDF DRM Protector Custom-Built Solution for You...
Lingwen1998
 
PDF
Open Chain Q2 Steering Committee Meeting - 2025-06-25
Shane Coughlan
 
PDF
SciPy 2025 - Packaging a Scientific Python Project
Henry Schreiner
 
PPTX
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
PPTX
Hardware(Central Processing Unit ) CU and ALU
RizwanaKalsoom2
 
PDF
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
PDF
AI + DevOps = Smart Automation with devseccops.ai.pdf
Devseccops.ai
 
PPTX
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
Revenue streams of the Wazirx clone script.pdf
aaronjeffray
 
Digger Solo: Semantic search and maps for your local files
seanpedersen96
 
OpenChain @ OSS NA - In From the Cold: Open Source as Part of Mainstream Soft...
Shane Coughlan
 
Online Queue Management System for Public Service Offices in Nepal [Focused i...
Rishab Acharya
 
유니티에서 Burst Compiler+ThreadedJobs+SIMD 적용사례
Seongdae Kim
 
Change Common Properties in IBM SPSS Statistics Version 31.pptx
Version 1 Analytics
 
The 5 Reasons for IT Maintenance - Arna Softech
Arna Softech
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pptx
Varsha Nayak
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
Tally_Basic_Operations_Presentation.pptx
AditiBansal54083
 
MiniTool Partition Wizard 12.8 Crack License Key LATEST
hashhshs786
 
Wondershare PDFelement Pro Crack for MacOS New Version Latest 2025
bashirkhan333g
 
[Solution] Why Choose the VeryPDF DRM Protector Custom-Built Solution for You...
Lingwen1998
 
Open Chain Q2 Steering Committee Meeting - 2025-06-25
Shane Coughlan
 
SciPy 2025 - Packaging a Scientific Python Project
Henry Schreiner
 
Agentic Automation: Build & Deploy Your First UiPath Agent
klpathrudu
 
Hardware(Central Processing Unit ) CU and ALU
RizwanaKalsoom2
 
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
AI + DevOps = Smart Automation with devseccops.ai.pdf
Devseccops.ai
 
AEM User Group: India Chapter Kickoff Meeting
jennaf3
 
Ad

Apache Apex - Hadoop Users Group

  • 1. Pramod Immaneni <[email protected]> PPMC Member, Architect @DataTorrent Inc Dec 2nd, 2015 Stream Processing Architecture and Applications Apache Apex (incubating)
  • 2. © 2015 DataTorrent Apex Platform Overview 2
  • 3. © 2015 DataTorrent Apache Malhar Library 3
  • 4. © 2015 DataTorrent Native Hadoop Integration 4 • YARN is the resource manager • HDFS used for storing any persistent state
  • 5. © 2015 DataTorrent Application Programming Model 5  A Stream is a sequence of data tuples  An Operator takes one or more input streams, performs computations & emits one or more output streams • Each Operator is YOUR custom business logic in java, or built-in operator from our open source library • Operator has many instances that run in parallel and each instance is single-threaded  Directed Acyclic Graph (DAG) is made up of operators and streams Directed Acyclic Graph (DAG) Output StreamTuple Tuple er Operator er Operator er Operator er Operator
  • 6. © 2015 DataTorrent Advanced Windowing Support 6  Application window  Sliding window and tumbling window  Checkpoint window  No artificial latency
  • 10. © 2015 DataTorrent Dynamic Partitioning 10 • Partitioning change while application is running ᵒ Change number of partitions at runtime based on stats ᵒ Determine initial number of partitions dynamically • Kafka operators scale according to number of kafka partitions ᵒ Supports re-distribution of state when number of partitions change ᵒ API for custom scaler or partitioner unifiers not shown 1a 2a 1b 2b 3 2b 1b 2c 3 2a 2d 1a 2b 1b 2c 3b 2a 2d 3a1a
  • 11. © 2015 DataTorrent How tuples are partitioned 11 • Tuple hashcode and mask used to determine destination partition ᵒ Mask picks the last n bits of the hashcode of the tuple ᵒ hashcode method can be overridden • StreamCodec can be used to specify custom hashcode for tuples ᵒ Can also be used for specifying custom serialization tuple: { Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x11) Partition 00 1 01 2 10 3 11 4
  • 12. © 2015 DataTorrent Custom partitioning 12 • Custom distribution of tuples ᵒ E.g.. Broadcast tuple:{ Name, 24204842, San Jose } Hashcode: 00101010001 0101 Mask (0x00) Partition 00 1 00 2 00 3 00 4
  • 13. © 2015 DataTorrent Fault Tolerance 13 • Operator state is checkpointed to a persistent store ᵒ Automatically performed by engine, no additional work needed by operator ᵒ In case of failure operators are restarted from checkpoint state ᵒ Frequency configurable per operator ᵒ Asynchronous and distributed by default ᵒ Default store is HDFS • Automatic detection and recovery of failed operators ᵒ Heartbeat mechanism • Buffering mechanism to ensure replay of data from recovered point so that there is no loss of data • Application master state checkpointed
  • 14. © 2015 DataTorrent Processing Guarantees 14 Atleast once • On recovery data will be replayed from a previous checkpoint ᵒ Messages will not be lost ᵒ Default mechanism and is suitable for most applications • Can be used in conjunction with following to ensure data is written once to store in case of fault recovery ᵒ Transactions with meta information, Rewinding output, Feedback from external entity, Idempotent operations Atmost once • On recovery the latest data is made available to operator ᵒ Useful in use cases where some data loss is acceptable and latest data is sufficient Exactly once • Operators checkpointed every window ᵒ Can be combined with transactional mechanisms to ensure end-to-end exactly once behavior
  • 15. © 2015 DataTorrent Stream Locality 15 • By default operators are deployed in containers (processes) randomly on different nodes across the Hadoop cluster • Custom locality for streams ᵒ Rack local: Data does not traverse network switches ᵒ Node local: Data is passed via loopback interface and frees up network bandwidth ᵒ Container local: Messages are passed via in memory queues between operators and does not require serialization ᵒ Thread local: Messages are passed between operators in a same thread equivalent to calling a subsequent function on the message
  • 16. © 2015 DataTorrent Data Processing Pipeline Example App Builder 16
  • 17. © 2015 DataTorrent Monitoring Console Logical View 17
  • 18. © 2015 DataTorrent Monitoring Console Physical View 18
  • 19. © 2015 DataTorrent Real-Time Dashboards Real Time Visualization 19
  • 20. © 2015 DataTorrent Resources 20 Apache Apex Community Page - http://apex.incubator.apache.org/
  • 21. © 2015 DataTorrent We Need Your Vote (Today) 21 Introducing Apache Apex - Not Just Another Stream Processing Platform Next Gen Big Data Analytics with Apache Apex Enterprise-grade streaming under 2ms on Hadoop
  • 24. © 2015 DataTorrent Partitioning and Scaling Out 24 • Operators can be dynamically scaled • Flexible Streams split • Parallel partitioning • MxN partitioning • Unifiers
  • 25. © 2015 DataTorrent Fault Tolerance Overview 25 Stateful Fault Tolerance Processing Semantics Data Locality  Supported out of the box – Application state – Application master state – No data loss  Automatic recovery  Lunch test  Buffer server  At least once  At most once  Exactly once  Stream locality for placement of operators  Rack local – Distributed deployment  Node local – Data does not traverse NIC  Container local – Data doesn’t need to be serialized  Thread local – Operators run in same thread  Data locality
  • 26. © 2015 DataTorrent Machine Data Application Logical View 26
  • 27. © 2015 DataTorrent Machine Data Application Physical View 27