Next gen tooling for building streaming analytics apps: code-less development, unit and integration testing, continuous integration and delivery

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Next Gen Tooling for Building
Stream Analytics App
Berlin Talk
George Vetticaden
VP of Emerging Products at Hortonworks

Streaming Analytics Manager (SAM)
What is it?
 Helps users build and deploy complex
stream analytics apps without writing
code using graphical interface
 An Open Source ASF Licensed tool
– Project Name: Streamline,
https://github.com/hortonworks/streamline
Key Design Principles
 Build stream analytics apps w/o specialized
skillsets.
 Support multiple underlining streaming
engine (Storm, Spark Streaming, Flink)
 Extensibility – Provide SDK to plug in custom
sources/sinks/processors/udfs
 Schema is a first class citizen

Who Uses SAM?

SAM is All about Doing Real-Time Analytics on the Stream
Real-Time
Prescriptive
Analytics
Real-Time Analytics
Real-Time
Predictive
Analytics
Real-Time
Descriptive
Analytics
What should we do
right now?
What could happen
now/soon?
What is happening
right now?

Showcasing SAM with an Use Case

Trucking company w/ large fleet of international trucks
A truck generates millions of events for a given route;
an event could be:
 'Normal' events: starting / stopping of the vehicle
 ‘Violation’ events: speeding, excessive acceleration and
breaking, unsafe tail distance
 ‘Speed’ Events: The speed of a driver that comes in every
minute.
Company uses an application that monitors truck
locations and violations from the truck/driver in real-
time
Route?
Truck?
Driver?
Analysts query a broad
history to understand if
today’s violations are
part of a larger problem
with specific routes,
trucks, or drivers

Two Streaming Sources: Geo Event and Speed Stream
eventType
truckId
driverId
driverName longitude
eventTime routeId
eventSource
2018-01-22 15:00:58.493|1516654858493| truck_geo_event |40| 23 | G Vetticaden | 1090292248 | Peoria to Ceder Rapids Route 2 | Lane Departure| 40.7 | -89.52 |1|
route
latitude
correlationId
speed
truckId
driverId
driverName
eventTime routeIdeventSource
2018-01-22 15:00:58.673|1516654858673| truck_speed_event|40| 23 | G Vetticaden| 1090292248 | Peoria to Ceder Rapids Route 2 | 73 |
route
 Each Truck emits different event stream
– Truck Geo Event
– Truck Speed Event
Truck Geo Event:
Truck Speed Event:
eventTimeLong
eventTimeLong

Common Streaming Analytics Requirements
Streaming Analytics
Requirement #
Requirement Description
Req. #1 Create streams consuming from the two Kafka topics that NiFi delivered the
enriched geo and speed streams to.
Req. #2 Join the streams of the Geo and Speed sensors over a time based aggregation
window.
Req. #3 Apply rules on the stream to filter on events of interest.
Req. #4 Calculate the average speed of driver over 3 minute window and create alert
for speeding driver
Req. #5 Enrich the stream with features required for a machine learning (ML) model.
The enrichment entails performing lookups for driver HR info, hours/miles
driven in the past week, weather info.
Req. #6 Normalize the events in the stream to feed into the PMML model.
Req. #7 Execute a predictive logistical regression model on the stream built with
Spark ML to predict if a driver is in danger going to commit a violation.
Req. #8 Alert and feed into real-time dashboard if model predicted a violation.

Implementing these 8 Requirements in Storm – Complex and Lots of
Java Code

Implementing this with Spark Structured Streaming – Complex and
Lots of Scala Code

Implementing this with SAM – Easy as Pie

Demo Implementing
Streaming Reqs 1 - 4 with SAM

Common Streaming Analytics Requirements
Streaming Analytics
Requirement #
Requirement Description
Req. #1 Create streams consuming from the two Kafka topics that NiFi delivered the
enriched geo and speed streams to.
Req. #2 Join the streams of the Geo and Speed sensors over a time based aggregation
window.
Req. #3 Apply rules on the stream to filter on events of interest.
Req. #4 Calculate the average speed of driver over 3 minute window and create alert
for speeding driver
Req. #5 Enrich the stream with features required for a machine learning (ML) model.
The enrichment entails performing lookups for driver HR info, hours/miles
driven in the past week, weather info.
Req. #6 Normalize the events in the stream to feed into the PMML model.
Req. #7 Execute a predictive logistical regression model on the stream built with
Spark ML to predict if a driver is in danger going to commit a violation.
Req. #8 Alert and feed into real-time dashboard if model predicted a violation.

Real-Time Predictive Analytics
 Question: No violation events but what might happen that I need to be worried about?
 My data science team has a model that can predict that based on
– Weather
– Roads
– Driver HR info like driver certification status, wagePlan
– Driver timesheet info like hours, and miles logged over the last week

Building the Predictive Model
Identify suitable ML algorithms to train a model – we will
use classification algorithms as we have labeled events
data
2
Transform enriched events data to a format that is
friendly to Spark MLlib – many ML libs expect
training data in a certain format
3
Train a logistic classification Spark model on YARN, with
above events as training input, and iterate to fine tune
generated model
4
Explore small subset of events to identify predictive
features and make a hypothesis. E.g. hypothesis: “foggy
weather causes driver violations”
1

Logistical Regression Model

Scoring the Predictive Model using SAM
Use SAM’s enrich/custom processors to enrich the event
with the features required for the model6
Enrich with Features
Use SAM’s projection/custom processors to
transform/normalize the streaming event and the
features required for the model
7
Transform/Normalize
Use SAM’s PMML processor to score the model for each
stream event with its required features8
Score Model
Use SAM’s rule and notification processors to alert,
notify and take action using the results of the model9
Alert / Notify / Action
Export the Spark Mllib model and import into the HDF’s
Model Registry
5 Model
Registry

Demo Implementing
Streaming Reqs 5 - 8

New SAM “Test” Mode
Key Highlights
 Persona User: Developer
 Allows Developers to test SAM app
locally without deploying to cluster
 SAM Test Mode Lifecycle
1. Create Named Test
2. Mock out sources (e.g: Kafka) with test
data
3. Execute the Test Case
4. Visualize Validate the data as it traverses
through the SAM DAG Visualization
5. Download the Test Case Results
 Can do steps 1-5 via UI or REST
 Enables writing automated test
(Junit)

Build Automated Junit Tests and CI & CD
Pipelines with SAM

Visualizing Output of Test Mode is Great…But want CI/CD
Key Highlights
 What Users Really Want?
– Automated Unit Tests – Write Junit tests
for streaming apps like their traditional
apps
– Continuous Integration - Incorporate
streaming apps into their CI Envs with
Jenkins
– Continuous Delivery – Deliver to business
new features in a continuous fashion.
 SAM Addresses these needs. How?
– SAM has first class support for REST.
– Everything in UI can be done with SAM
REST. This includes SAM Test Mode.
 SAM REST allows you to
– Create service pools, create envs, create
test case, setup test data, download test
results, validate, deploy app to diff envs

Next gen tooling for building streaming analytics apps: code-less development, unit and integration testing, continuous integration and delivery

More Related Content

What's hot (20)

Similar to Next gen tooling for building streaming analytics apps: code-less development, unit and integration testing, continuous integration and delivery (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Next gen tooling for building streaming analytics apps: code-less development, unit and integration testing, continuous integration and delivery

Editor's Notes