Using Redis Streams To Build Event Driven Microservices And User Interface In Clojure Bobby Calderwood
0 likes519 views
The document summarizes Bobby Calderwood's presentation on using Redis Streams to build event-driven microservices and user interfaces in Clojure(Script). The presentation covers how Redis Streams were used to facilitate asynchronous processing and distributed consistency for a customer project. It also discusses how Carmine, the Clojure Redis client, was updated to support Redis Streams shortly after their release. The presentation concludes with a demo of how Redis Streams can be used to retrofit an existing system with asynchronous integration.
Using Redis Streams To Build Event Driven Microservices And User Interface In Clojure Bobby Calderwood
- 1. PRESENTED BY Using Redis Streams to build Event-driven Microservices and User Interface in Clojure(Script) Bobby Calderwood Evident Systems, Founder [email protected] @bobbycalderwood
- 2. PRESENTED BY • Software Engineer @Cognitect for 4+ years, Consulting and Datomic • Distinguished Engineer @CapitalOne for 3+ years, helping redesign and move mission-critical banking systems to the cloud • Founder @Evident_Systems, helping smaller banks redesign and move mission-critical banking systems to the cloud • Meet with me! http://meet.evident.systems/founder Hello, I’m Bobby Calderwood!
- 3. PRESENTED BY 1 Story of a real customer system Event-Sourcing + Redis + Clojure(Script) = Joy 2 Aside: Memoir of an occasional Open Source contributor How we got Clojure’s Carmine to speak Redis Streams 3 Our Story Continues: The (not so) Big Refactor Retrofitting Redis Streams into a working system Agenda: 4 Demo System Let’s pretend we have to build a system, and then modify it with an asynchronous integration.
- 4. PRESENTED BY • Major automobile manufacturer • Greenfield system for new capability, political proving ground for stack and process • Highly asynchronous image processing and significant UI display/manipulation • Possible need for data re-processing/re-play, undo/redo, audit, and sharing with other teams • We decided to architect it using Event Sourcing on Streaming Data using Redis, and implemented in Clojure(Script) I was solving a customer problem
- 5. PRESENTED BY Why Event Sourcing & Streaming?
- 6. PRESENTED BY • At very least, we knew we had to process images asynchronously • We suspected that we’d need to keep this pipeline flexible, to try different processing steps and techniques • We knew we’d need to convey the results of this pipeline back to the UI (the oft-forgotten participant in the distributed system) To facilitate our asynchronous processing pipeline
- 7. PRESENTED BY • Integration among microservices becomes complex and burdensome – Balkanization of data stores – Data “owner” determines data access pattern – Non-determinism == impossibility of reasoning about system: • state • causality/time • modes of failure Because HTTP APIs aren’t enough
- 8. AWS Death Star diagram, circa 2008 as per Werner Vogels tweet
- 9. PRESENTED BY • Request/response protocols face a fundamental epistemological problem – What is happening, anything new? – When should I check for new data? – Who’s responsible to tell me? Because HTTP APIs aren’t enough
- 10. PRESENTED BY • Event Sourcing provides as-of consistency across distributed participants • Streaming Data protocols convey new data/events to interested participants (subscribers) • Any participant can simulate synchrony and consistency by blocking and waiting for a specific event Streaming Data + Event Sourcing Facilitate Distributed Consistency Image by Alan Light CC BY-SA 3.0
- 11. PRESENTED BY • “Situated Programs” a la Rich Hickey: – interact with other systems – often interact with humans – remain in use for long periods of time – are situated in a changing world – use other people’s code • In large enterprises – audit/compliance/governance – certain eventual integration with many new possible systems and data consumers – across many teams and organizational units who (basically) trust each other More generally, to solve a specific type of problem
- 14. PRESENTED BY Why Redis? (for streaming data)
- 15. PRESENTED BY ‘cause your last slide definitely said “Kafka”
- 16. PRESENTED BY • Redis is ubiquitous: you (or your customer) already has it in the stack • Redis is battle-tested, and Ops is comfortable with it • Redis is (often) easier to operate than Kafka (with some tradeoffs) Redis is Already There, and Just Works
- 17. PRESENTED BY • After Kafka Popularized, Salvatore noticed gap in Redis, as he described: http:// antirez.com/news/114 • Existing data structures got almost there – Sorted Sets: memory hungry, clients can’t block awaiting new records, order not stable – Lists: linear time to scan, can’t seek to specific position, no fan-out to multiple consumers – Pub/Sub: doesn’t maintain history, can’t do windowed aggregation, etc. • Redis Streams maintains range-queryable history, and efficiently conveys to awaiting clients Redis recently added Streams
- 19. PRESENTED BY • Clojure is a fantastic all-purpose programming language whose philosophy aligns well with event sourcing, streaming data • We needed to build all components quickly, with a small team of Clojurists: – UI in ClojureScript – React + Reagent + re-frame – Access to JavaScript libraries with Clojure language semantics – HTTP API in Clojure + Pedestal (+ GraphQL via Lacinia) – Workers in Clojure • Clojure’s core.async library provide in-process semantics aligned with streaming data Clojure(Script) is powerful and full-stack
- 20. PRESENTED BY • Very solid, simple Redis client • Generated from Redis artifacts (on which more to come) • Stays close to Redis semantics Clojure has Carmine
- 21. PRESENTED BY But Carmine didn’t have Streams…
- 22. PRESENTED BY • ~1 hour after Salvatore’s blog post dropped, I filed https://github.com/ ptaoussanis/carmine/issues/208 • But I felt Open Source guilt, so I opened https://github.com/ptaoussanis/ carmine/pull/210 • But Carmine is generated from Redis Docs JSON, which didn’t yet include Streams operations, so I filed https://github.com/antirez/redis-doc/issues/943, and opened https://github.com/antirez/redis-doc/pull/963 …so we added support for Streams
- 23. PRESENTED BY And support for all the above was added incredibly quickly!
- 24. PRESENTED BY • We had already built the HTTP API and processing pipeline • Both components were producers and consumers of events • Our implementation was limited in its scalability/fault-tolerance, so we needed to re-work • Ugh, how much re-work to support Streams?!? …but not quickly enough to prevent re-work
- 25. PRESENTED BY …including re-tooled deployment pipeline to support multiple workers How much rework? $ git diff --stat master <snip> 16 files changed, 576 insertions(+), 598 deletions(-)
- 26. PRESENTED BY • XADD — Produce to a Stream • XRANGE, XREVRANGE — Query stream per range of existing offsets, non- blocking, batch-ish • XREAD — Optionally-blocking read (conveys novelty), realtime-ish • XREADGROUP — Optionally-blocking read by a group of consumers, ack-ing messages to advance consumer offset (but watch message ordering!) But How to Redis Stream? https://redis.io/topics/streams-intro
- 27. PRESENTED BY And Now, A Completely Unrelated Demo App!
- 28. PRESENTED BY • Always use Clojure, so that your major distributed systems refactorings can be a net code reduction! • Sharpen your tools: contribute to open source! • Use Event Sourcing! – Kafka for big/important/durable/expensive things – Redis for smaller things https://redis.io/topics/streams-intro • Check out my demo app! https://github.com/bobby/redisconf19-demo Conclusion
- 30. PRESENTED BY