The State of Stream Processing

1Confidential
The State of Stream Processing
Neil Avery, @avery_neil,
March 2018

2
The state of stream processing
The rise of streaming...
business has changed:
customer 360, digital transformation,
IoT, machine learning, data science

3
Before stream processing
Before:
OLTP - RPC/Request-response - Messaging - ESB
Almost:
- OLTP: TRIGGERs + materialized views, stored procedures…
….CEP….Big data…
Acknowledge:
Dr Michael Stonebraker - CEP, volt-db, Streaming-SQL etc 2005+

… it didn’t happen until now ...
… WHY NOT? ...
4

5
Because Stream Processing is hard
● Scaling
● Semantics
● Correctness
● Accuracy (determinism)
(eos)
● Transactions
● Partitioning
● Fault tolerance
● Re-processing
● Unifying tables and streams
(stream-relational)
● Time

6
Why stream processing is different
We are solving different problems today.
Businesses want to be real-time, correlated events, react and adapt
Turning off batch and moving to real-time, scaling beyond the limits of traditional
storage system, applying data science or machine learning (fashion, fraud or retailer,
suggestions, advertising)
processing the ‘now’ has great value

7
Who is using it
Companies: Uber, Netflix, Audi, Banks, Slack… everyone
...obvious use-cases…
dumb pipes,
Reactive microservices,
ML, streaming ETL etc
….there are ALSO, ‘data-type’ problems we can solve...

8
Stream processing == data processing for events
● Correlation (user register and signed up, bad password)
● Scaling out the data flow of a company (connecting data-sources)
● Moving to the Central nervous system (Data flow system)
● Transaction processing: The Database unbundled, AKA, Turning the
database inside out…
…. why is this so important? [2014]

9
Turning the database inside out….
“The database is a cache of a subset of the log”
● Pat Helland, 2007
With Kafka we can do this at scale…
… using the LOG and partitions ..

10
The 12 stage Adoption JOURNEY
● Dumb pipes
● Simple streaming (filtering)
● Streaming ETL
● Analytics
● Monitoring & notifications
● Anomaly detection
● Correlation
● Reactive microservices
● Transaction processing
● Database inside out
● Central nervous system (platform)
● Global, central nervous system
(platForm)

11
Turning the database inside out
● Event sourcing
● Source of truth
● Data flow systems
● Central nervous system
● Txns At ‘web-scale’ scale!

12
At scale….
Single Deployments with
● Hundreds of brokers
● Hundreds of thousands of topics
● Millions of partitions
Hundreds of millions of Events per second….
AWS Death Star diagram, circa 2008 as per Werner Vogels tweet

14
the streaming platform stack
A new breed of data platform - built on unix principles (do lots of simple
things)
- Messaging
- Storage (topics + partitions + log based)
- Correctness (EoS, txns)
- Processing (KSQL, kafka streams)
- Governance
- Operations
- Elastic scale up/down
Kind of a like a modern database but...it can do these clever things, fast and
at scale - unbundled using stream processing

15
the streaming platform in action
KSQL
Stream processing here...

16
State of the union
stream-Relational Players
● Kafka Streams
● Flink
● spark
Stream Only players
● Apex
● Samza
● Storm
● Google Dataflow
A stream-relational processing platform has the following capabilities:
● Relations (or tables) are first-class citizens, i.e. each has an independent identity.
● Relations can be transformed into other relations.
● Relations can be queried in an ad-hoc manner.

..Is a..
- Sequence of facts, records or events
- Captures the order of events
- Time sensitive
But it needs to be stored as a stream => in a LOG
A stream….
18
{ Joe pays Mary $1m }

..Is a..
- ‘Materialized’ view of a stream
- uses Change Data Capture (CDC)
A Table….
19
insert delete update
stream

Your code here
Processing….
20

Event sourcing
Table
Stream
Stream-table join
Your code
The next level down
21

Do you think that’s a
table you are querying?
23

24
Streams and Tables
● STREAM and TABLE as first-class
citizens
● Interpretations of Topic content
● STREAM - data in motion
● TABLE - collected state of a stream
(aggregations)
○ One record per key (per window)
○ Current values (compacted topic)
○ Changelog

25
Windows: Aggregations & Tables
●
Source: https://www.rittmanmead.com/blog/2017/10/ksql-streaming-sql-for-apache-kafka/

28
● Scaling
● Semantics
● Correctness
● Accuracy (determinism)
(eos)
● Transactions
● Partitioning
● Fault tolerance
● Re-processing
● Unifying tables and streams
(stream-relational)
● Time
Stream Processing (the hard parts)

30
Or some SQL
CREATE STREAM notifications AS SELECT code,
definition FROM ticks LEFT JOIN opens ON
opens.id = ticks.code;
CREATE TABLE metrics AS SELECT * FROM
notifications WINDOW TUMBLING (size 30 second)
GROUP BY id, value WHERE value = ‘Closed’;

32
Applied Stream processing
1. Data processing: enriching streams of data, filtering, cleaning, joining,
splitting: ETL, Machine learning
2. Analytics: asking questions of data
3. Apps: Build reactive (stateful) microservice apps with materialized views
4. Transaction processing: Stream proc + log-store (Kafka), handle txns
across multiple partitions
(replacing 40 years of slow ACID with Async, fast, correct behaviour)
80% of stream processing is for doing simple stuff, like ETL

33
Stream processing using SQL
80% of stream processing is for doing ‘simple stuff’, like ETL
SQL == winning!
=> Simple, repeatable, easy to deploy & run….
=> Accessible: anyone can do stream processing!
https://github.com/confluentinc/ksql

34
KSQL: Streaming ETL
● Streaming ETL
○ Kafka is popular for data pipelines.
○ KSQL enables easy transformations of data within the pipe
CREATE STREAM vip_actions AS
SELECT userid, page, action
FROM clickstream c
LEFT JOIN users u ON c.userid =
u.user_id
WHERE u.level = 'Platinum';

35
KSQL: Monitoring
● Real-Time Monitoring
○ Infrastructure/data flows/apps monitoring, tracking, and alerting (e.g.
logs, metrics)
○ Sensor / IoT data
CREATE TABLE error_counts AS
SELECT error_code, count(*)
FROM monitoring_stream
WINDOW TUMBLING (SIZE 1 MINUTE)
WHERE type = 'ERROR’
GROUP BY error_code;

36
KSQL: Anomaly detection
● Anomaly Detection
○ Identifying patterns or anomalies in real-time data, surfaced in
milliseconds
CREATE TABLE possible_fraud AS
SELECT card_number, COUNT(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING COUNT(*) > 3;

37
Use case: Anomaly detection - fraud
Goal: Detect when ‘new’ transactions occur against suspicious recipients
Real world: once a fraudster has access to your account - they need to take the
money. This is done by trying to make multiple transactions of small amounts against
common recipients. There is usually a group of them
I.e. take $100 per month for a broadband or mobile phone provider etc

38
Use case: Anomaly detection - fraud
3x Data Sources:
1) realtime txns, 2) set of suspicious recipients, 3) set of valid recipients
Check ‘previous use’
processor - str-table join
(by account-recipient)
{
Type: txn
req_id:1234,
joe -> mary,
joe_account: 1234
mary_account: 5678
amount: $1m
}
Check ‘is suspicious’
processor - str-table join
(by recipient)
{
Type: suspicious_account
account: 1234
recipient: verizon
}
{
Type: valid_account
account: 1234
recipient: verizon
}
table
table
{
Type:
detected_multi-sus-txns
account: 1234
}
Check ‘multi-sus-24h’
processor - count-by-
(by recipient)
For a complete solution (including data generators) see:
https://github.com/bluemonk3y/ksql-recipe-fraudulent-txns
Window - 24h - count(recipient)

39
Use case: Web-scale Transaction processing
> Joe pays mary $1,000,000
Nb: requires a log based streaming platform to support determinism
(kafka!)
A Traditional OLTP would require atomic commit on all tables
Log-based stream processing scales by using partitioning
(partitions are log-append, atomic and provide the source of truth)

40
Use case: Web-scale Transaction processing
1. Request #1234 Joe -> mary $1m
2. Payer account (joe)
3. Payee account (mary)
Requests processor
(by request#)
{
req_id:1234,
joe -> mary,
joe_account: 1234
mary_account: 5678
amount: $1m
}
Debit processor
(by payer id)
De-dup req-id
Credit processor
(by payee id)
de-dup req-id
{ req:1234, type: debit,
who: joe, amount: - $1m }
{ req:1234, type: credit,
who: mary, amount: + $1m }
Payment balance processor
(by account number)
{req:1234, joe - $1m } {req:1234, mary + $1m }
Why is this fast? Why is it horizontally scalable?

41
Use case: Central nervous system
Human resources
B2B txns
Webscale Payments
& ETL
Regulations, compliance and audit
Data science, ETL

42
What (interesting) problems do we still need
to solve?
- SQL (for everyone)
- Improved auto-scaling of analytics (cloud)
- Improved workload optimisation (calcite)
- Large scale real-time analytics (multi-stage - Hyperloglog-reduce)
- Improved streaming-relational support
- Improved data modelling & constraint handling
- Data lineage

43
The future
- Continued adoption, acceleration of “dataflow systems & central nervous system”.
- In 2022, 60% of business will be using Event Driven architectures
- We grow up the stack, we grow out the stack

We have a long road ahead of us….
44

45
Questions?
Recommended reading:
Confluent Blog: https://www.confluent.io/blog/
Kafka Summit San Francisco: https://kafka-summit.org/
Free ebook: Making Sense of Stream Processing, Martin Kleppmann
Safari books: “The Log.”
“Designing Data Intensive Applications”,
Google dataflow paper (very old now!)
& lots more….
Neil Avery, @avery_neil
Technologist, Office of the CTO

46
Questions?
Demo applications: https://github.com/confluentinc/examples
• Interactive Queries, Joins, Security, Windowing, Avro
integration, …
Confluent documentation: http://docs.confluent.io/current/streams/
• Quickstart, Concepts, Developer Guide, FAQ
Recorded talks
• Introduction to Kafka’s Streams API:
http://www.youtube.com/watch?v=o7zSLNiTZbA
• Application Development and Data in the Emerging
World of Stream Processing (higher level talk):
https://www.youtube.com/watch?v=JQnNHO5506w
Neil Avery, @avery_neil
Technologist, Office of the CTO

Bio for Neil
Neil is a Technologist in the Office of the CTO at Confluent, the company founded by the creators of
Apache Kafka. He has over 20 years of expertise of working on distributed computing, messaging and
stream processing. He has built or redesigned commercial messaging platforms, distributed caching
products as well as developed large scale bespoke systems for tier-1 banks. After a period at
ThoughtWorks, he went on to build some of the first distributed risk engines in financial services. In
2008 he launched a startup that specialised in distributed data analytics and visualization. Prior to
joining Confluent he was the CTO at a fintech consultancy.
Font:
https://github.com/google/fonts/blob/master/apache/permanentmarker/PermanentMarker-Regular.ttf
49

Synopsis -The State of Stream processing
This session walks through the journey to how stream processing arrived at its current
state and a look towards the future. It explores the use-cases and concepts that drive the
evolution of stream processing and the need for fast data, at scale. This includes the
evolution of how Apache Kafka has grown to become the goto streaming platform that is
the Confluent Platform, where it supports the concepts that underpin many processing
frameworks. Learn about the current challenges being faced by Stream processing
frameworks including real-time semantics surrounding windowing, joining, correctness,
elasticity, and accessibility. Finally, learn how KSQL is evolving to shape the future of
what is to come.
50

The State of Stream processing
Stream processing is now at the forefront of many company strategies. Over the last
couple of years we have seen streaming use cases explode and now proliferate the
landscape of any modern business.
Use-cases including digital transformation, IoT, real-time risk, payments microservices
and machine learning are all built on the fundamental that they need fast data and they
need it at scale.
Apache Kafka has long been the streaming platform of choice, its origins of being dumb
pipes for big data have long since been left behind and now it is the goto-streaming
platform of choice.
Stream processing beckons as being the vehicle for driving those streams, and along
with it brings a world of real-time semantics surrounding windowing, joining, correctness,
elasticity, and accessibility. The ‘current state of stream processing’ walks through the
origins of stream processing, applicable use cases and then dives into the challenges
currently facing the world of stream processing as it drives the next data revolution.
51

The State of Stream Processing

More Related Content

What's hot (20)

Similar to The State of Stream Processing (20)

More from confluent (20)

Recently uploaded (20)

The State of Stream Processing