SlideShare a Scribd company logo
Stream API For Apex
June 2016
Apex Overview
Apex Overview
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state
Current Development Model
Directed Acyclic Graph (DAG)
Output
Stream
Tupl
e
Tupl
e
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
● Stream is a sequence of data tuples
● Typical Operator takes one or more input streams, performs computations & emits one or more output streams
● Each operator is your custom business logic in java, or built-in operator from our open source library
● Operator has many instances that run in parallel and each instance is single-threaded
● Directed Acyclic Graph (DAG) is made up of operators and streams
Current Application Example
@ApplicationAnnotation(name="WordCountDemo")
public class Application implements StreamingApplication
{
@Override
public void populateDAG(DAG dag, Configuration conf)
{
WordCountInputOperator input = dag.addOperator("wordinput", new WordCountInputOperator());
UniqueCounter<String> wordCount = dag.addOperator("count", new UniqueCounter<String>());
ConsoleOutputOperator consoleOperator = dag.addOperator("console", new ConsoleOutputOperator());
dag.addStream("wordinput-count", input.outputPort, wordCount.data);
dag.addStream("count-console",wordCount.count, consoleOperator.input);
}
}
o Easier for beginners to start with
o Fluent API
o Smaller learning curve
o Transform methods in one place vs operator library
o Operator API provides flexibility while high-level API provides ease of use
Why we need high-level API
Stream API
map(..)
filter(..)
…
addOperator(...)
with(prop, val)
…
window(Opt...)
ApexStream<T>
group(..)
groupByKey(...)
reduce(..)
fold(..)
join(..)
count(..)
…
window(Opt...)
WindowedStream<T>
<<interface>> <<interface>>
Stream API (Application Example)
@ApplicationAnnotation(name = "WordCountStreamingApiDemo")
public class ApplicationWithStreamAPI implements StreamingApplication
{
@Override
public void populateDAG(DAG dag, Configuration configuration)
{
String localFolder = "./src/test/resources/data";
ApexStream<String> stream = StreamFactory
.fromFolder(localFolder)
.flatMap(new Split())
.window(new WindowOption.GlobalWindow(), new
TriggerOption().withEarlyFiringsAtEvery(Duration.millis(1000)).accumulatingFiredPanes())
.countByKey(new ConvertToKeyVal()).print();
stream.populateDag(dag);
}
}
How it works
o ApexStream<T> literally means bounded/unbounded data set of type T
o ApexStream<T> also holds a graph data struture of all operator and
connections between operators from input to current point
o Each transform method attach one or more operators to current graph
data structure and return a new Apex Stream object
o The graph data structure won’t be translated to Apex DAG until
populateDag or run method are called
How it works (Con’t)
○ Method chain for readability
○ Stateless transform(map, flatmap, filter)
○ Some input and output are available (file, console, Kafka)
○ Some interoperability (addOperator, getDag, set property/attributes etc)
○ Local mode and distributed mode
○ Annonymous function class support
○ Extensible
Current Status
○ WindowedStream is in pull request along with Operators that support it
○ A few window transforms (count, reduce, etc)
○ 3 Window types (fix window, sliding window, session window)
○ 3 Trigger types (early trigger, late trigger, at watermark)
○ 3 Accumulation modes(accumulate, discard, accumulation_retraction)
○ In memory window state (checkpointed)
Current Status (Con’t)
Roadmap
○ Persistent window state for windowed operators (large state)
○ Fully follow Beam model (window, trigger, watermark)
○ Rich selection of windowed transform (group, combine, join)
○ Support custom window assignor
○ Support custom trigger
○ More input/output (hbase, cassendra, jdbc, etc)
○ Better schema support
○ More language support (java 8, scala, etc...)
○ What the community asks for
Resources
○ Apache Apex website - http://apex.apache.org/
○ Subscribe - http://apex.apache.org/community.html
○ Download - http://apex.apache.org/downloads.html
○ Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex
○ Facebook - https://www.facebook.com/ApacheApex/
○ Meetup - http://www.meetup.com/topics/apache-apex
○ SlideShare - http://www.slideshare.net/ApacheApex/presentations
○ More Examples - https://github.com/DataTorrent/examples
○ Pull request
https://github.com/apache/apex-malhar/pull/319
https://github.com/apache/apex-malhar/pull/327
Demo & Code Example
○ Word Count
○ AutoComplete
Thank You!
June
2016
Comments/Questions
siyuan@datatorrent.com

More Related Content

What's hot (20)

PDF
Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra Tagare
Apache Apex
 
PPTX
Deep Dive into Apache Apex App Development
Apache Apex
 
PDF
Building your first aplication using Apache Apex
Yogi Devendra Vyavahare
 
PPTX
Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming
Apache Apex
 
PPTX
Smart Partitioning with Apache Apex (Webinar)
Apache Apex
 
PPTX
Introduction to Real-Time Data Processing
Apache Apex
 
PPTX
Apache Big Data 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
PPTX
Apache Apex: Stream Processing Architecture and Applications
Thomas Weise
 
PDF
Low Latency Polyglot Model Scoring using Apache Apex
Apache Apex
 
PDF
Developing streaming applications with apache apex (strata + hadoop world)
Apache Apex
 
PDF
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Apex
 
PPTX
DataTorrent Presentation @ Big Data Application Meetup
Thomas Weise
 
PPTX
Ingestion and Dimensions Compute and Enrich using Apache Apex
Apache Apex
 
PPTX
Introduction to Apache Apex and writing a big data streaming application
Apache Apex
 
PDF
Introduction to Apache Apex - CoDS 2016
Bhupesh Chawda
 
PPTX
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations
Apache Apex
 
PPTX
Big Data Berlin v8.0 Stream Processing with Apache Apex
Apache Apex
 
PDF
Apex as yarn application
Chinmay Kolhatkar
 
PPTX
Ingesting Data from Kafka to JDBC with Transformation and Enrichment
Apache Apex
 
PPTX
Fault Tolerance and Processing Semantics in Apache Apex
Apache Apex Organizer
 
Actionable Insights with Apache Apex at Apache Big Data 2017 by Devendra Tagare
Apache Apex
 
Deep Dive into Apache Apex App Development
Apache Apex
 
Building your first aplication using Apache Apex
Yogi Devendra Vyavahare
 
Intro to Apache Apex (next gen Hadoop) & comparison to Spark Streaming
Apache Apex
 
Smart Partitioning with Apache Apex (Webinar)
Apache Apex
 
Introduction to Real-Time Data Processing
Apache Apex
 
Apache Big Data 2016: Next Gen Big Data Analytics with Apache Apex
Apache Apex
 
Apache Apex: Stream Processing Architecture and Applications
Thomas Weise
 
Low Latency Polyglot Model Scoring using Apache Apex
Apache Apex
 
Developing streaming applications with apache apex (strata + hadoop world)
Apache Apex
 
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Apex
 
DataTorrent Presentation @ Big Data Application Meetup
Thomas Weise
 
Ingestion and Dimensions Compute and Enrich using Apache Apex
Apache Apex
 
Introduction to Apache Apex and writing a big data streaming application
Apache Apex
 
Introduction to Apache Apex - CoDS 2016
Bhupesh Chawda
 
Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data Transformations
Apache Apex
 
Big Data Berlin v8.0 Stream Processing with Apache Apex
Apache Apex
 
Apex as yarn application
Chinmay Kolhatkar
 
Ingesting Data from Kafka to JDBC with Transformation and Enrichment
Apache Apex
 
Fault Tolerance and Processing Semantics in Apache Apex
Apache Apex Organizer
 

Similar to Java High Level Stream API (20)

PDF
Building Your First Apache Apex Application
Apache Apex
 
PPTX
Apache Apex Introduction with PubMatic
Apache Apex
 
PDF
Introduction to Apache Apex by Thomas Weise
Big Data Spain
 
PPTX
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Dataconomy Media
 
PDF
#GeodeSummit - Apex & Geode: In-memory streaming, storage & analytics
PivotalOpenSourceHub
 
PDF
Apex & Geode: In-memory streaming, storage & analytics
Ashish Tadose
 
PDF
BigDataSpain 2016: Stream Processing Applications with Apache Apex
Thomas Weise
 
PDF
Stream Processing use cases and applications with Apache Apex by Thomas Weise
Big Data Spain
 
PDF
Introduction to Apache Apex
Chinmay Kolhatkar
 
PPSX
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex (Hadoop)
Apache Apex
 
PDF
Visualizing Big Data in Realtime
DataWorks Summit
 
PPTX
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
PDF
Real-time Stream Processing using Apache Apex
Apache Apex
 
PDF
BigDataSpain 2016: Introduction to Apache Apex
Thomas Weise
 
PPTX
Next Gen Big Data Analytics with Apache Apex
DataWorks Summit/Hadoop Summit
 
POTX
Apache Spark Streaming: Architecture and Fault Tolerance
Sachin Aggarwal
 
PDF
Apache Spark Overview part2 (20161117)
Steve Min
 
PDF
Windowing in Apache Apex
Apache Apex
 
PDF
Windowing in apex
Yogi Devendra Vyavahare
 
PPTX
Stream processing - Apache flink
Renato Guimaraes
 
Building Your First Apache Apex Application
Apache Apex
 
Apache Apex Introduction with PubMatic
Apache Apex
 
Introduction to Apache Apex by Thomas Weise
Big Data Spain
 
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Dataconomy Media
 
#GeodeSummit - Apex & Geode: In-memory streaming, storage & analytics
PivotalOpenSourceHub
 
Apex & Geode: In-memory streaming, storage & analytics
Ashish Tadose
 
BigDataSpain 2016: Stream Processing Applications with Apache Apex
Thomas Weise
 
Stream Processing use cases and applications with Apache Apex by Thomas Weise
Big Data Spain
 
Introduction to Apache Apex
Chinmay Kolhatkar
 
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex (Hadoop)
Apache Apex
 
Visualizing Big Data in Realtime
DataWorks Summit
 
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
Real-time Stream Processing using Apache Apex
Apache Apex
 
BigDataSpain 2016: Introduction to Apache Apex
Thomas Weise
 
Next Gen Big Data Analytics with Apache Apex
DataWorks Summit/Hadoop Summit
 
Apache Spark Streaming: Architecture and Fault Tolerance
Sachin Aggarwal
 
Apache Spark Overview part2 (20161117)
Steve Min
 
Windowing in Apache Apex
Apache Apex
 
Windowing in apex
Yogi Devendra Vyavahare
 
Stream processing - Apache flink
Renato Guimaraes
 
Ad

More from Apache Apex (10)

PPTX
Hadoop Interacting with HDFS
Apache Apex
 
PPTX
Introduction to Yarn
Apache Apex
 
PPTX
Introduction to Map Reduce
Apache Apex
 
PPTX
HDFS Internals
Apache Apex
 
PPTX
Intro to Big Data Hadoop
Apache Apex
 
PPTX
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Apache Apex
 
PPTX
Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)
Apache Apex
 
PPTX
Apache Beam (incubating)
Apache Apex
 
PPTX
Making sense of Apache Bigtop's role in ODPi and how it matters to Apache Apex
Apache Apex
 
PPTX
Apache Apex & Bigtop
Apache Apex
 
Hadoop Interacting with HDFS
Apache Apex
 
Introduction to Yarn
Apache Apex
 
Introduction to Map Reduce
Apache Apex
 
HDFS Internals
Apache Apex
 
Intro to Big Data Hadoop
Apache Apex
 
Building Your First Apache Apex (Next Gen Big Data/Hadoop) Application
Apache Apex
 
Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)
Apache Apex
 
Apache Beam (incubating)
Apache Apex
 
Making sense of Apache Bigtop's role in ODPi and how it matters to Apache Apex
Apache Apex
 
Apache Apex & Bigtop
Apache Apex
 
Ad

Recently uploaded (20)

PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PDF
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
PDF
AI Agents in the Cloud: The Rise of Agentic Cloud Architecture
Lilly Gracia
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PDF
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
PDF
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
“Voice Interfaces on a Budget: Building Real-time Speech Recognition on Low-c...
Edge AI and Vision Alliance
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PPT
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
Bitcoin for Millennials podcast with Bram, Power Laws of Bitcoin
Stephen Perrenod
 
AI Agents in the Cloud: The Rise of Agentic Cloud Architecture
Lilly Gracia
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
“Voice Interfaces on a Budget: Building Real-time Speech Recognition on Low-c...
Edge AI and Vision Alliance
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
“NPU IP Hardware Shaped Through Software and Use-case Analysis,” a Presentati...
Edge AI and Vision Alliance
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 

Java High Level Stream API

  • 1. Stream API For Apex June 2016
  • 3. Apex Overview • YARN is the resource manager • HDFS used for storing any persistent state
  • 4. Current Development Model Directed Acyclic Graph (DAG) Output Stream Tupl e Tupl e er Operator er Operator er Operator er Operator er Operator er Operator ● Stream is a sequence of data tuples ● Typical Operator takes one or more input streams, performs computations & emits one or more output streams ● Each operator is your custom business logic in java, or built-in operator from our open source library ● Operator has many instances that run in parallel and each instance is single-threaded ● Directed Acyclic Graph (DAG) is made up of operators and streams
  • 5. Current Application Example @ApplicationAnnotation(name="WordCountDemo") public class Application implements StreamingApplication { @Override public void populateDAG(DAG dag, Configuration conf) { WordCountInputOperator input = dag.addOperator("wordinput", new WordCountInputOperator()); UniqueCounter<String> wordCount = dag.addOperator("count", new UniqueCounter<String>()); ConsoleOutputOperator consoleOperator = dag.addOperator("console", new ConsoleOutputOperator()); dag.addStream("wordinput-count", input.outputPort, wordCount.data); dag.addStream("count-console",wordCount.count, consoleOperator.input); } }
  • 6. o Easier for beginners to start with o Fluent API o Smaller learning curve o Transform methods in one place vs operator library o Operator API provides flexibility while high-level API provides ease of use Why we need high-level API
  • 8. Stream API (Application Example) @ApplicationAnnotation(name = "WordCountStreamingApiDemo") public class ApplicationWithStreamAPI implements StreamingApplication { @Override public void populateDAG(DAG dag, Configuration configuration) { String localFolder = "./src/test/resources/data"; ApexStream<String> stream = StreamFactory .fromFolder(localFolder) .flatMap(new Split()) .window(new WindowOption.GlobalWindow(), new TriggerOption().withEarlyFiringsAtEvery(Duration.millis(1000)).accumulatingFiredPanes()) .countByKey(new ConvertToKeyVal()).print(); stream.populateDag(dag); } }
  • 9. How it works o ApexStream<T> literally means bounded/unbounded data set of type T o ApexStream<T> also holds a graph data struture of all operator and connections between operators from input to current point o Each transform method attach one or more operators to current graph data structure and return a new Apex Stream object o The graph data structure won’t be translated to Apex DAG until populateDag or run method are called
  • 10. How it works (Con’t)
  • 11. ○ Method chain for readability ○ Stateless transform(map, flatmap, filter) ○ Some input and output are available (file, console, Kafka) ○ Some interoperability (addOperator, getDag, set property/attributes etc) ○ Local mode and distributed mode ○ Annonymous function class support ○ Extensible Current Status
  • 12. ○ WindowedStream is in pull request along with Operators that support it ○ A few window transforms (count, reduce, etc) ○ 3 Window types (fix window, sliding window, session window) ○ 3 Trigger types (early trigger, late trigger, at watermark) ○ 3 Accumulation modes(accumulate, discard, accumulation_retraction) ○ In memory window state (checkpointed) Current Status (Con’t)
  • 13. Roadmap ○ Persistent window state for windowed operators (large state) ○ Fully follow Beam model (window, trigger, watermark) ○ Rich selection of windowed transform (group, combine, join) ○ Support custom window assignor ○ Support custom trigger ○ More input/output (hbase, cassendra, jdbc, etc) ○ Better schema support ○ More language support (java 8, scala, etc...) ○ What the community asks for
  • 14. Resources ○ Apache Apex website - http://apex.apache.org/ ○ Subscribe - http://apex.apache.org/community.html ○ Download - http://apex.apache.org/downloads.html ○ Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex ○ Facebook - https://www.facebook.com/ApacheApex/ ○ Meetup - http://www.meetup.com/topics/apache-apex ○ SlideShare - http://www.slideshare.net/ApacheApex/presentations ○ More Examples - https://github.com/DataTorrent/examples ○ Pull request https://github.com/apache/apex-malhar/pull/319 https://github.com/apache/apex-malhar/pull/327
  • 15. Demo & Code Example ○ Word Count ○ AutoComplete