Download as PDF, PPTX
![@crichardson
OO = State + Behavior
creditLimit
creditReservations : Map[OrderId, Money]
Customer
List<Event> processCommand (
Command aCommand) { … }
void applyEvent (Event anEvent) { … }
State
Behavior](https://image.slidesharecdn.com/developingevent-drivenmicroserviceswitheventsourcingandcqrssvcc2015-151004160756-lva1-app6891/75/Developing-event-driven-microservices-with-event-sourcing-and-CQRS-svcc-svcc2015-59-2048.jpg)
![@crichardson
Event Store API
T is a subclass of Aggregate[T]
Rx Observable = Future++](https://image.slidesharecdn.com/developingevent-drivenmicroserviceswitheventsourcingandcqrssvcc2015-151004160756-lva1-app6891/75/Developing-event-driven-microservices-with-event-sourcing-and-CQRS-svcc-svcc2015-64-2048.jpg)
Uploaded byChris Richardson
Developing event-driven microservices with event sourcing and CQRS (svcc, svcc2015)
The document provides a comprehensive overview of developing event-driven microservices utilizing event sourcing and Command Query Responsibility Segregation (CQRS). It discusses the challenges of traditional monolithic architectures and presents solutions through microservices architecture, emphasizing event-driven design principles for maintaining data consistency and scalability. The details include practical implementations, benefits, and drawbacks of event sourcing and CQRS in the context of modern application development.
More Related Content
Similar to Developing event-driven microservices with event sourcing and CQRS (svcc, svcc2015)
Developing event-driven microservices with event sourcing and CQRS (svcc, svcc2015)
- 1. @crichardson Developing event-driven microservices withevent sourcing and CQRS Chris Richardson Author of POJOs in Action Founder of the original CloudFoundry.com @crichardson [email protected] http://plainoldobjects.com http://microservices.io
- 2. @crichardson Presentation goal Show howEvent Sourcing and Command Query Responsibility Segregation (CQRS) are a great way to implement microservices
- 3.
- 4. @crichardson About Chris Consultant andtrainer focusing on microservices (http://www.chrisrichardson.net/)
- 5. @crichardson About Chris Founder ofa startup that is creating a platform that makes it easy for application developers write microservices (http://bit.ly/trialeventuate)
- 6.
- 7. @crichardson Agenda Why event sourcing? Overviewof event sourcing ACID-free design Designing a domain model based on event sourcing Implementing queries in an event sourced application Event sourcing and microservices
- 8. @crichardson Tomcat Traditional monolithic architecture Browser/ Client WAR/EAR RDBMS … Customers Orders StoreFrontUI develop test deploy Simple to Load balancer scale Spring MVC Spring Hibernate HTML REST/JSON ACID Shopping cart
- 9. @crichardson But that leads*to monolithic hell For large and/or complex applications…
- 10. @crichardson Today: use amicroservice, polyglot architecture Shopping UI Orders Service Customer Service Order Database Customer Database Standalone services Sharded SQLNoSQL DB
- 11. @crichardson But now wehave distributed data management problems
- 12. @crichardson Example: placing anorder Order Service Customer Service Order Database Customer Database Order #1 Customer #1 No 2PC No ACID NoSQL SQL
- 13. @crichardson Customer management How tomaintain invariants? Order management Order Service placeOrder() Customer Service updateCreditLimit() Customer creditLimit ... has ordersbelongs toOrder total Invariant: sum(open order.total) <= customer.creditLimit ?
- 14.
- 15. @crichardson ….Use an event-driven architecture Servicespublish events when something important happens, e.g. state changes Services subscribe to events and update their state Maintain eventual consistency across multiple aggregates (in multiple datastores) Synchronize replicated data
- 16. @crichardson Order Management Order id :4567 total: 343 state = CREATED Customer Management Customer creditLimit : 12000 creditReservations: {} Customer creditLimit : 12000 creditReservations: { 4567 -> 343} Order id : 4567 total: 343 state = OPEN Eventually consistent credit checking Message Bus createOrder() Publishes: Subscribes to: Subscribes to: publishes: OrderCreatedEvent CreditReservedEvent OrderCreatedEvent CreditReservedEvent
- 17. @crichardson Now there aretwo problems to solve….
- 18. @crichardson Problem #1: Howto design eventually consistent business logic? More on that later….
- 19. @crichardson Problem #2: Howatomicity update database and publish an event Order Service Order Database Message Broker insert Order publish OrderCreatedEvent dual write problem ?
- 20. @crichardson Update and publishusing 2PC Guaranteed atomicity BUT Need a distributed transaction manager Database and message broker must support 2PC Impacts reliability Not fashionable 2PC is best avoided
- 21. @crichardson Transaction log tailing How: Readthe database “transaction log” = single source of truth Publish events to message broker LinkedIn databus https://github.com/linkedin/databus Supports Oracle and MySQL Publish as events AWS DynamoDB streams Ordered sequence of creates, updates, deletes made to a DynamoDB table Last 24 hours Subscribe to get changes MongoDB Read the oplog
- 22. Transaction log tailing:benefits and drawbacks Benefits No 2PC No application changes required Guaranteed to be accurate Drawbacks Immature Database specific solutions Low-level DB changes rather business level events = need to reverse engineer domain events
- 23. @crichardson Use database triggers Trackchanges to tables Insert events into an event table Use datastore as a message queue Pull events from event table and write to message broker
- 24. Database triggers: benefits anddrawbacks Benefits No 2PC No application changes required Drawbacks Requires the database to support them Database specific solutions Low-level DB changes rather business level events = need to reverse engineer domain events Error-prone, e.g. missing trigger
- 25. @crichardson Application created events Usedatastore as a message queue Txn #1: Update database: new entity state & event Txn #2: Consume event Txn #3: Mark event as consumed Eventually consistent mechanism (used by eBay) See BASE: An Acid Alternative, http://bit.ly/ebaybase
- 26. Application created events Benefits High-leveldomain events No 2PC Drawbacks Requires changes to the application Only works for SQL and some NoSQL databases Error-prone
- 27. @crichardson Agenda Why event sourcing? Overviewof event sourcing ACID-free design Designing a domain model based on event sourcing Implementing queries in an event sourced application Event sourcing and microservices
- 28. @crichardson Just publish events Application Database MessageBroker update publish X
- 29. @crichardson Event sourcing For eachaggregate (business entity): Identify (state-changing) domain events Define Event classes For example, ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent Order: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
- 30. @crichardson Persists events NOT currentstate Order status …. Event table 101 ACCEPTED Order table X OrderCreated101 901 … OrderApproved101 902 … OrderShipped101 903 …
- 31. @crichardson Replay events torecreate state Order state OrderCreated(…) OrderAccepted(…) OrderShipped(…) Events Periodically snapshot to avoid loading all events
- 32. @crichardson The present isa fold over history currentState = foldl(applyEvent, initialState, events)
- 33.
- 34.
- 35. @crichardson Request handling inan event-sourced application HTTP Handler Event Store pastEvents = findEvents(entityId) Order new() applyEvents(pastEvents) newEvents = processCmd(SomeCmd) saveEvents(newEvents) Microservice A (optimistic locking)
- 36. @crichardson Event Store publishesevents - consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) Aggregate CQRS View update() update() Microservice B send notifications …
- 37. Event store =database + message broker Hybrid database and message broker Implementations: Home-grown/DIY geteventstore.com by Greg Young http://eventuate.io (mine) Event Store Save aggregate events Get aggregate events Subscribe to events
- 38. @crichardson Benefits of eventsourcing Solves data consistency issues in a Microservice/NoSQL-based architecture Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,… Eliminates O/R mapping problem (mostly) Reifies state changes: Built-in, reliable audit log, temporal queries Preserved history More easily implement future requirements
- 39. @crichardson Drawbacks of eventsourcing Weird and unfamiliar Events = a historical record of your bad design decisions Handling duplicate events can be tricky Application must handle eventually consistent data Event store only directly supports PK-based lookup => use Command Query Responsibility Segregation (CQRS) to handle queries
- 40. @crichardson Agenda Why event sourcing? Overviewof event sourcing ACID-free design Designing a domain model based on event sourcing Implementing queries in an event sourced application Event sourcing and microservices
- 41. @crichardson Implementing createOrder() POST /orders classOrderServiceImpl { public Order createOrder() { … Creates Order … } }
- 42. @crichardson Implement requirements andpreserve invariants As a customer I want to place an order So that I get the needed products Given that my available credit is $1500 When I place a $250 order Then the order is created Then my available credit is $1250 Story Scenario Post conditions Pre conditions Given that my available credit is $1500 When I place a $2500 order Then the order is rejected Then my available credit is $1500 Scenario Invariant: sum(open order.total) <= customer.creditLimit
- 43. @crichardson createOrder() updates multiple aggregates UpdatesCustomer aggregate’s available credit Creates Order aggregate Associates Order with Customer
- 44. @crichardson Old-style ACID… BEGIN TRANSACTION …VERIFY CREDIT … … CREATE ORDER… COMMIT TRANSACTION
- 45. @crichardson … becomes eventually consistent(BASE) Updating multiple aggregates multi-step, event-driven flow each step updates one Aggregate Service creates saga to coordinate workflow A state machine Part of the domain, e.g. Order aggregate OR Synthetic aggregate Post-conditions and invariants eventually become true
- 46. @crichardson Creating an order Order Customer Created Creditreserved public Order createOrder() { … Creates Order … } Approved
- 47. @crichardson Need compensating transactions Pre-conditions mightbe false when attempting to update an aggregate Credit check fails cancel order Credit check succeeded but customer cancels order undo credit reservation …
- 48. @crichardson Agenda Why event sourcing? Overviewof event sourcing ACID-free design Designing a domain model based on event sourcing Implementing queries in an event sourced application Event sourcing and microservices
- 49. @crichardson Use the familiarbuilding blocks of DDD Entity Value object Services Repositories Aggregates ⟸ essential
- 50. Aggregate design Graph consistingof a root entity and one or more other entities and value objects Each core business entity = Aggregate: e.g. customer, Account, Order, Product, …. Reference other aggregate roots via primary key Often contains partial copy of other aggregates’ data Order OrderLine Item quantity productId productName productPrice customerId Address street city …
- 51. @crichardson Partition the domainmodel into Aggregates Order OrderLine Item quantity … Address street city … Customer Product name price
- 52. @crichardson Transaction = processingone command by one aggregate No opportunity to update multiple aggregates within a transaction Aggregate granularity is important If an update must be atomic (i.e. no compensating transaction) then it must be handled by a single aggregate e.g. scanning boarding pass at security checkpoint or when entering jetway
- 53.
- 54. Designing domain events Recordstate changes for an aggregate Part of the public API of the domain model ProductAddedToCart id : TimeUUID productId productName productPrice shoppingCartId Required by aggregate Enrichment: Required by by consumers
- 55.
- 56. @crichardson Designing commands Created bya service from incoming request Processed by an aggregate Immutable Contains value objects for Validating request Creating event Auditing user activity
- 57.
- 58.
- 59. @crichardson OO = State+ Behavior creditLimit creditReservations : Map[OrderId, Money] Customer List<Event> processCommand ( Command aCommand) { … } void applyEvent (Event anEvent) { … } State Behavior
- 60. @crichardson Aggregate traits Map Commandto Events Apply event returning updated Aggregate Used by Event Store to reconstitute aggregate
- 61.
- 62. @crichardson Customer - commandprocessing
- 63.
- 64. @crichardson Event Store API Tis a subclass of Aggregate[T] Rx Observable = Future++
- 65. @crichardson Creating an order save()concisely specifies: 1.Creates Customer aggregate 2.Processes command 3.Applies events 4.Persists events
- 66. @crichardson Event handling inCustomers 1.Load Customer aggregate 2.Processes command 3.Applies events 4.Persists events Triggers BeanPostProcessor Durable subscription name
- 67. @crichardson Agenda Why event sourcing? Overviewof event sourcing ACID-free design Designing a domain model based on event sourcing Implementing queries in an event sourced application Event sourcing and microservices
- 68. @crichardson Let’s imagine youwant to display a customer and their recent orders
- 69. @crichardson Need to ‘join’customers and orders BUT Event store only supports primary key lookup
- 70. @crichardson Use Command Query ResponsibilitySegregation (CQRS)
- 71. @crichardson Command Query Responsibility Segregation(CQRS) Command-side Commands Aggregate Event Store Events Query-side Queries (Denormalized) View Events
- 72. @crichardson Query-side design Event Store Updater ViewUpdater Service Events Reader HTTP GET Request View Query Service View Store e.g. MongoDB Neo4J CloudSearch update query
- 73. @crichardson Persisting a customerand order history in MongoDB { "_id" : "0000014f9a45004b-0a00270000000000", "_class" : "net.chrisrichardson…..views.orderhistory.CustomerView", "version" : NumberLong(5), "orders" : { "0000014f9a450063-0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063-0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063-0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063-0a00270000000001", "orderTotal" : { "amount" : "3000" } } }, "name" : "Fred", "creditLimit" : { "amount" : "2000" } } Denormalized = efficient lookup
- 74. @crichardson Persisting customers andorder info using Spring Data for MongoDB...
- 75. @crichardson Persisting customers andorder using Spring Data for MongoDB...
- 76. Other kinds ofviews AWS Cloud Search Text search as-a-Service View updater batches aggregates to index View query service does text search AWS DynamoDB NoSQL as-a-Service On-demand scalable - specify desired read/write capacity Document and key-value data models Useful for denormalized, UI oriented views
- 77. Benefits and drawbacksof CQRS Benefits Necessary in an event-sourced architecture Separation of concerns = simpler command and query models Supports multiple denormalized views Improved scalability and performance Drawbacks Complexity Potential code duplication Replication lag/eventually consistent views
- 78. @crichardson Agenda Why event sourcing? Overviewof event sourcing ACID-free design Designing a domain model based on event sourcing Implementing queries in an event sourced application Event sourcing and microservices
- 79. @crichardson Partitioning into microservices: UseStrategic DDD
- 80. @crichardson Strategic design: identifysub- domains and bounded contexts Online store domain
- 81. @crichardson Strategic design: identifysub- domains and bounded contexts Customer Management Order Management Catalog management …
- 82. @crichardson Bounded context =microservices Customer management domain model Order management domain model Catalog management domain model … domain model Customer view Command side Query side
- 83. @crichardson Event Store HTTP Request HTTPAdapter Event Handler Cmd Cmd Events Events Xyz Adapter Xyz Request microservice Aggregate Service Event Adapter
- 84.
- 85. @crichardson When to usemicroservices? In the beginning: •You don’t need it •It will slow you down Later on: •You need it •Refactoring dependencies is painful
- 86. @crichardson Aggregates + EventSourcing = Modular domain model
- 87. @crichardson Modular domain model Tightlycoupled ACID Loosely coupled aggregates Eventually consistent Customer Order Product Customer Order Product
- 88. @crichardson MonolithicFirst approach Tomcat WAR/EAR Not entirelyfree though - Event Sourcing premium Customer Order Product
- 89. @crichardson But no BigBall of Mud to untangle
- 90. @crichardson Microservices deployment Tomcat WAR/EAR Much higher- microservices premium Customer Tomcat WAR/EAR Order Tomcat WAR/EAR Product
- 91. @crichardson Summary Event sourcing solveskey data consistency issues with: Microservices Partitioned SQL/NoSQL databases Apply strategic DDD to identify microservices Apply tactical DDD to design individual services Use CQRS to implement materialized views for queries
- 92.