End to End Akka Streams / Reactive Streams - from Business to Socket

Konrad `ktoso` Malawski & Johan Andrén
pic: 1:1 scale Gundam model @ Odaiba, Tokyo
streams / Reactive Streams
End to end
– from Business to Socket

Konrad `ktoso` Malawski
Akka Team,
Reactive Streams TCK,
Persistence, HTTP

Konrad `@ktosopl` Malawski
work: akka.io lightbend.com
personal blog: http://kto.so
communities: geecon.org Java.pl / KrakowScala.pl sckrk.com GDGKrakow.pl lambdakrk.pl

Special thanks to
Johan Andrén
for helping prepare the examples.
Akka Team
Stockholm Scala User Group

Nice to meet you!
Who are you?

Make building powerful concurrent &
distributed applications simple.
Akka is a toolkit and runtime
for building highly concurrent,
distributed, and resilient
message-driven applications
on the JVM

Actors – simple & high performance concurrency
Cluster / Remoting – location transparency, resilience
Cluster Sharding – and more prepackaged patterns
Streams – back-pressured stream processing
Persistence – Event Sourcing
HTTP – complete, fully async and reactive HTTP Server
Oﬀicial Kafka, Cassandra, DynamoDB integrations, tons
more in the community
Complete Java & Scala APIs for all features (since day 1)
Typed coming soon…

End to End Akka Streams / Reactive Streams - from Business to Socket

“Stream”
What does it mean?!

Suddenly everyone jumped on the word “Stream”.
Akka Streams / Reactive Streams started end-of-2013.
“Streams”
* when put in “” the word does not appear in project name, but is present in examples / style of APIs / wording.

Suddenly everyone jumped on the word “Stream”.
Akka Streams / Reactive Streams started end-of-2013.
“Streams”
Akka Streams
Reactive Streams
RxJava “streams”*
Spark Streaming
Apache Storm “streams”*
Java Steams (JDK8)
Reactor “streams”*
Kafka Streams
ztellman / Manifold (Clojure)
* when put in “” the word does not appear in project name, but is present in examples / style of APIs / wording.
Apache GearPump “streams”
Apache [I] Streams (!)
Apache [I] Beam “streams”
Apache [I] Quarks “streams”
Apache [I] Airﬂow “streams” (dead?)
Apache [I] Samza
Scala Stream
Scalaz Streams, now known as FS2
Swave.io
Java InputStream / OutputStream / … :-)

No no no…!
Not THAT Back-pressure!
No no no…!
Not THAT Back-pressure!
What is back-pressure?

Why back-pressure?
So you’ve built your app and it’s awesome.

Why back-pressure?
Let’s not smash it horribly under load.

Reactive Streams - story: 2013’s impls
~2013:
Reactive Programming
becoming widely adopted on JVM.
- Play introduced “Iteratees”
- Akka (2009) had Akka-IO (TCP etc.)
- Ben starts work on RxJava
http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspx
http://infoscience.epﬂ.ch/record/176887/ﬁles/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky
https://github.com/ReactiveX/RxJava/graphs/contributors
https://github.com/reactor/reactor/graphs/contributors
https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy
Teams discuss need for back-pressure
in simple user API.
Play’s Iteratee / Akka’s NACK in IO.
}

Play Iteratees – pull back-pressure, difficult API
http://blogs.msdn.com/b/rxteam/archive/2009/11/17/announcing-reactive-extensions-rx-for-net-silverlight.aspx
http://infoscience.epfl.ch/record/176887/files/DeprecatingObservers2012.pdf - Ingo Maier, Martin Odersky
https://github.com/ReactiveX/RxJava/graphs/contributors
https://github.com/reactor/reactor/graphs/contributors
https://medium.com/@viktorklang/reactive-streams-1-0-0-interview-faaca2c00bec#.69st3rndy
Akka-IO – NACK back-pressure; low-level IO (Bytes); messaging API
RxJava – no back-pressure, nice API

Reactive Streams - expert group founded
October 2013
Roland Kuhn (Akka) and Erik Meijer (Rx .NET) meet in Lausanne,
while recording “Principles of Reactive Programming” Coursera Course.
Viktor Klang (Akka), Erik Meijer, Ben Christensen (RxJava)
and Marius Eriksen (Twitter) meet at Twitter HQ.
The term “reactive non-blocking asynchronous back-pressure” gets coined.
Afterwards more organisations are invited to join the effort, including Pivotal, RedHat etc.

October 2013
The term “reactive non-blocking asynchronous back-pressure” gets coined.Goals:
- asynchronous
- never block (waste)
- safe (back-threads pressured)
- purely local abstraction
- allow synchronous impls.

October 2013
December 2013
Stephane Maldini & Jon Brisbin (Pivotal Reactor) contacted by Viktor.

October 2013
December 2013
Soon after, the “Reactive Streams” expert group is formed.
Also joining the efforts: Doug Lea (Oracle), EndreVarga (Akka), Johannes Rudolph &  
Mathias Doenitz (Spray), and many others, including myself join the effort soon after.

October 2013
December 2013
Soon after, the “Reactive Streams” expert group is formed.
Also joining the efforts: Doug Lea (Oracle), EndreVarga (Akka), Johannes Rudolph &  
Mathias Doenitz (Spray), and many others, including myself join the effort soon after.
I ended up implementing much of the TCK.
Please use it, let me know if it needs improvements :-)

2014–2015:
Reactive Streams Spec & TCK
development, and implementations.
1.0 released on April 28th 2015,
with 5+ accompanying implementations.
2015
Proposed to be included with JDK9 by Doug Lea
via JEP-266 “More Concurrency Updates”
http://hg.openjdk.java.net/jdk9/jdk9/jdk/ﬁle/6e50b992bef4/src/java.base/share/classes/java/util/concurrent/Flow.java

But what does it do!?
Reactive Streams

Publisher[T] Subscriber[T]
Back-pressure explained

Fast Publisher Slow Subscriber
Push model

Subscriber usually has some kind of buffer.
Push model

What if the buffer overﬂows?
Push model

Use bounded buffer,
drop messages + require re-sending
Push model

Kernel does this!
Routers do this!
(TCP)
Use bounded buffer,
drop messages + require re-sending
Push model

Increase buffer size…
Well, while you have memory available!
Push model

Reactive Streams explained
Reactive Streams
explained in 1 slide

Fast Publisher will send at-most 3 elements.
This is pull-based-backpressure.
Reactive Streams: “dynamic push/pull”

JEP-266 – soon…!
public final class Flow {
private Flow() {} // uninstantiable
@FunctionalInterface
public static interface Publisher<T> {
public void subscribe(Subscriber<? super T> subscriber);
}
public static interface Subscriber<T> {
public void onSubscribe(Subscription subscription);
public void onNext(T item);
public void onError(Throwable throwable);
public void onComplete();
}
public static interface Subscription {
public void request(long n);
public void cancel();
}
public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> {
}
}

Reactive Streams: goals
1) Avoiding unbounded buffering across async boundaries
2)Inter-op interfaces between various libraries

Argh, implementing a correct RS Publisher
or Subscriber is so hard!

Argh, implementing a correct
RS Publisher or Subscriber is so hard!

Argh, implementing a correct
RS Publisher or Subscriber is so hard!
You should be using
Akka Streams instead!

Already made a huge industry impact!
Reactive Streams

Inspiring other technologies
It’s been a while since Java inspired
other modern technologies, hasn’t it?
The JVM ecosystem is having
a small renaissance one might say?

The implementation.
Complete and awesome Java and Scala APIs.
As everything since day 1 in Akka.
Akka Streams

Akka Streams in 20 seconds:
// types:
Source<Out, Mat>
Flow<In, Out, Mat>
Sink<In, Mat>
// generally speaking, it's always:
val ready =
Source.from…(???).via(flow).map(i -> i * 2).to(sink)
val mat: Mat = ready.run()
// the usual example:
val f: Future<String> =
Source.single(1).map(i -> i.toString).runWith(Sink.head)
Proper static typing!

Source.single(1).map(i -> i.toString).runWith(Sink.head())
// types: _
Source<Int, NotUsed>
Flow<Int, String, NotUsed>
Sink<String, Future<String>>

Source.single(1).map(i -> i.toString).runWith(Sink.head())
// types: _
Source<Int, Unit>
Flow<Int, String, Unit>
Sink<String, Future<String>>

Materialization
Gears from GeeCON.org, did I mention it’s an awesome conf?

What is “materialization” really?

Codename:
Alpakka
// these are “Alpacasso”

A community for Streams connectors
Alpakka – a community for Stream connectors
Alp

Threading & Concurrency in Akka Streams Explained (part I)
Mastering GraphStages (part I, Introduction)
Akka Streams Integration, codename Alpakka
A gentle introduction to building Sinks and Sources using GraphStage APIs
(Mastering GraphStages, Part II)
Writing Akka Streams Connectors for existing APIs
Flow control at the boundary of Akka Streams and a data provider
Akka Streams Kafka 0.11

Existing examples:
MQTT
AMQP
Streaming HTTP
Streaming TCP
Streaming FileIO
Cassandra Queries
“Reactive Kafka” (akka-stream-kafka)
S3, SQS & other Amazon APIs
Streaming JSON
Streaming XML

“Composing building blocks”

Reactive tracking of job progress

Akka Streams & HTTP
streams
& HTTP

A core feature not obvious to the untrained eye…!
Akka Streams / HTTP
Quiz time!
TCP is a ______ protocol?

A core feature not obvious to the untrained eye…!
Akka Streams / HTTP
Quiz time!
TCP is a STREAMING protocol!

Streaming in Akka HTTP
DEMO
http://doc.akka.io/docs/akka/2.4.7/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming” https://github.com/akka/akka/pull/20778
HttpServer as a:
Flow[HttpRequest, HttpResponse]

DEMO
HttpServer as a:
HTTP Entity as a:
Source[ByteString, _]

DEMO
HttpServer as a:
HTTP Entity as a:
Source[ByteString, _]
Websocket connection as a:
Flow[ws.Message, ws.Message]

It’s turtles buffers all the way down!

Streaming from Akka HTTP
No demand from TCP
=
No demand upstream
=
Source won’t generate tweets
=>
Bounded memory
stream processing!

Streaming from Akka HTTP (Java)
public static void main(String[] args) {
final ActorSystem system = ActorSystem.create();
final Materializer materializer = ActorMaterializer.create(system);
final Http http = Http.get(system);
final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));
final Route tweetsRoute =
path("tweets", () ->
completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json())
);
final Flow<HttpRequest, HttpResponse, NotUsed> handler =
tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler,
ConnectHttp.toHost("localhost", 8080),
materializer
);
System.out.println("Running at http://localhost:8080");
}

Streaming from Akka HTTP (Scala)
object Example extends App
with SprayJsonSupport with DefaultJsonProtocol {
import akka.http.scaladsl.server.Directives._
implicit val system = ActorSystem()
implicit val mat = ActorMaterializer()
implicit val jsonRenderingMode = EntityStreamingSupport.json()
implicit val TweetFormat = jsonFormat1(Tweet)
def tweetsStreamRoutes =
path("tweets") {
complete {
Source.repeat(Tweet(""))
}
}
Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080)
System.out.println("Running at http://localhost:8080");
}

Next steps for Akka
Completely new Akka Remoting (goal: 700.000+ msg/s (!)),
(it is built using Akka Streams, Aeron).
More integrations for Akka Streams stages, project Alpakka.
Reactive Kafka polishing with SoftwareMill.
Akka Typed progressing again, likely towards 3.0.
Akka HTTP 2.0 Proof of Concept in progress.
Of course, continued maintenance of Cluster and others.

Is now the time to adopt
Akka Streams / Reactive Streams?

Akka <3 contributions
• Easy to contribute:
• https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute
• https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22
• Akka Stream Contrib
• https://github.com/akka/akka-stream-contrib
• (above contains Alpakka, but will be split)
• Mailing list:
• https://groups.google.com/group/akka-user
• Public chat rooms:
• http://gitter.im/akka/dev developing Akka
• http://gitter.im/akka/akka using Akka

Pics
Gundam pictures from: http://www.wallpaperup.com/tag/gundam/3

Free e-book and printed report.
Available soon.
Covers what reactive actually is.
Implementing in existing architectures.
Thoughts from the team that’s building
reactive apps since more than 6 years.
Obligatory “read my book!” slide :-)

ktoso @ typesafe.com
twitter: ktosopl
github: ktoso
team blog: letitcrash.com
home: akka.io

Q/A
ktoso @ lightbend.com
twitter: ktosopl
github: ktoso
team blog: blog.akka.io
home: akka.io

End to End Akka Streams / Reactive Streams - from Business to Socket

More Related Content

What's hot (20)

Viewers also liked (14)

Similar to End to End Akka Streams / Reactive Streams - from Business to Socket (20)

Recently uploaded (20)

End to End Akka Streams / Reactive Streams - from Business to Socket