Agile Lab_BigData_Meetup_AKKA
1 like•525 views
This document discusses reactive and parallel programming using Akka. It introduces Akka, an open-source toolkit for building distributed, concurrent applications on the JVM. Akka supports actors, futures, software transactional memory, and message passing to handle concurrency. Akka cluster allows building fault-tolerant distributed systems. Examples demonstrate using Akka for microservices, streaming engines, authentication platforms, and IoT applications. The document concludes with an example of how Agile Lab uses Akka in their applications.
Agile Lab_BigData_Meetup_AKKA
- 1. Reactive and Parallel Programming Presents... Enrico Pigozzi Vito Ressa
- 2. Who is Agile Lab? GO BIG (data) or GO HOME
- 3. Summary • Concurrency problem Recurrent issues Handle concurrency • Akka Overview How it works Akka cluster Expansions and use-cases How we use Akka • Demo time
- 4. Parallel and concurrent Programming Ⓒ2016 Agile Lab S.r.l. “We need to stop coding our concurrency and just let our code run concurrently”
- 5. Shared/mutable state • Threads (pool) usage to make code “faster” + synchronization features (primitives, locks, defensive copies, etc.) to make it “safe”. • We tend to create the problem first, and (don’t) fix it up later (N.B. no real concurrency allowed)
- 6. Issues • Threads • expensive (startup, memory, context switching) • limited (by OS) • CPU Vs. Synchronus IO balance required • optimizated usage requires non-blocking API (cumbersome to write and test) • Design • bottlenecks, race conditions, deadlocks • traditional errors management approches (exceptions and error codes) don’t translate well • maintenance / refactoring costs can increase to a fairy high degree (and therefore avoided)
- 7. Software Transactional Memory • Turns memory heap in a transactional data set (Atomic, Consistent, Isolated) • Based on Transactional References: Memory cells, holding an immutable value (separates identity from value) Implement CAS (Compare-And-Swap) semantic (old value still availabe after change) Can only be altered within a transaction Very fast (near constant time) Thread and iteration safe
- 8. Futures future(a,b)futureB(a) Asynchronous values Composable functional abstraction Error propagation and recover allowed Callback on termination available for { a ← futureA() b ← futureB(a) c ← futureC(a,b) } yield c futureA()
- 9. Actors & Message passing • Unit of code organization: “policy” separated from “logic”, incapsulate state, behaviour, mailbox • Weakly / Adaptive typed: achitectural flexibility enhanced (decupled design) • Safe: one thing a time in isolated context (actor accessed by reference, interaction by messages) • Immutable structures/messages managed by actors in isolated/safe context
- 10. Finite State Machine • Applied to actor: must be in one of the defined (in type system) state all the time • A data object can be attached to each state (M:N relation) • A starting state (and eventually data) has to be defined • Event insted of simple messages: provides current state info • 'When/goto/stay' model: define actor reaction (partial function) to event for a state S1 S2 S3 S4S5 A B A A A A B B B B A,B = possible messages S1,…S5 = States
- 11. Microservices • Bounded Context: Each element of functionality into a separate service • Fault tolerance • Deployment independency • DataStore independency • Loose coupling • Service Discovery • Sync/async messaging between services • Scalability
- 12. Akka Ⓒ2016 Agile Lab S.r.l.
- 13. Overview Event-driven systems with Actors High Performance Easy concurrency handling Simple distribution High flexibility Resilience Reactive Manifesto http://www.reactivemanifesto.org/ Commercial Support
- 14. Strenghts
- 15. How it works? • Actor as an essential code unit • Messages for actors interaction • Hierarchical organization • Supervision Hierarchy
- 16. Actors Lifecycle • Actor instance is hidden • Actor restarting transparent to other actors • Mailbox bounded to the incarnation
- 17. Messages • Any kind of message but immutable! • Hint: In scala use case classes and take advantage of pattern matching (see next slide) Tell or Ask
- 18. Mailbox • Landing point for messages • Bounded to the incarnation of the actor (not the instance!) • Allows different configurations (FIFO, LIFO, custom priority, ...) • A mailbox can be shared in case of balancing pools MAILBOX STASH unstash()
- 20. Akka Cluster Ⓒ2016 Agile Lab S.r.l.
- 21. Akka Cluster Node A Node B Node C Node D Leader CLUSTER Cluster membership service • Peer to peer • Decentralized • Fault tolerant • No single point of failure • No single point of bottleneck A Node is identified by a tuple hostname:port:uid • Gossip Protocol • Automatic failure detector
- 22. Failure detection An implementation of The Phi Accrual Failure Detector Failure Detector input Nodes heartbeats phi-value Is a node unreachable or not? Depends on a treshold
- 23. Extensions and use-cases Ⓒ2016 Agile Lab S.r.l.
- 24. Extensions Community projects available at http://akka.io/community/ • Akka persistence - http://doc.akka.io/docs/akka/current/scala/persistence.html • Akka HTTP - http://doc.akka.io/docs/akka-stream-and-http- experimental/current/scala/http/ • Akka Streams - http://doc.akka.io/docs/akka-stream-and-http- experimental/current/scala/http/ • Akka Camel - http://doc.akka.io/docs/akka/current/scala/camel.html • Akka FSM - http://doc.akka.io/docs/akka/current/scala/fsm.html
- 25. Use cases: GearPump Ref: https://www.typesafe.com/resources/case-studies-and-stories/intels-gearpump-real- time-streaming-engine-using-akka • Real-time streaming engine • Multiple stream datasources • Need of HA, scalability, performance • "Once at a time" transactional message processing With 4 nodes GearPump can process up to 11 million messages per seconds with an average latency of 17ms (100 bytes messages)
- 26. Use cases: Paycasso • Three-factor authentication platform • High speed authentication is mandatory • Scala, Akka and Cassandra at the core as a reactive stack • Make use of Akka HTTP, Akka Persistence, Akka Cluster Sharding. Ref: https://www.typesafe.com/resources/case-studies-and-stories/paycasso-conquers- speed-gap-in-three-factor-authentications-with-reactive-stack
- 27. Use cases: SAMI • Data broker in the cloud for IoT • Need for reactiveness and resiliency • Microservices-based platform with Akka Akka usages • Actors dedicated to clients for each WebSocket. • At the transformation layer, where data is streamed into the system and normalized. Ref: https://www.typesafe.com/resources/case-studies-and-stories/samsung-strategy- and-innovation-center-executes-iot-vision-at-startup-speed-with-reactive-architecture
- 28. Other use cases Interesting Discussion http://stackoverflow.com/questions/4493001/good-use-case-for-akka Online Gaming Finance Statistics Betting Social Media Telecom Web Media
- 29. How we use Akka Ⓒ2016 Agile Lab S.r.l.
- 30. Akka in Action with Microservices Mosaico RCS Reverse Proxy Akka Srv1 Akka Srv2 Akka Srv3 Akka Srv4 Akka Srv5 Akka Srv6 Mongo DB Mongo DB My SQL S3 Akka Custering
- 31. Akka in Action with BigData Wasp AgileLab Akka Custering master Producer Guardian Consumer Gardian API Guardian E P E P E P E P K R K R K R R T S P S P WS TCP Legend: EP: EndPoint DataIn KR:KafkaRouter SP: Spark Pipeline RT: Real Time WS: Web Socket API DataIn Control Plane
- 32. Demo time... Ⓒ2016 Agile Lab S.r.l.
- 33. We are hiring... Ⓒ2016 Agile Lab S.r.l. [email protected]