Download as PDF, PPTX
![(Almost) self-documenting
{
"type" : "record",
"name" : "User",
"namespace" : "com.example",
"fields" : [
{"name": "username", "type": “string"},
{"name": "phone", "type": [null,"int"], "default" : null},
{"name": "address", "type": "string", "default": “UNKNOWN"},
{"name": "country", "type": "enum", "symbols": ["CA", "US", "OTHER"],
"doc" : "The user's country. Select other if not in list"}
]
}](https://image.slidesharecdn.com/fundamentalsofeventdrivenmicroservices11728489736099-241021085110-2c61dcf9/75/fundamentalsofeventdrivenmicroservices11728489736099-pdf-68-2048.jpg)
Uploaded bySmarak Pani
fundamentalsofeventdrivenmicroservices11728489736099.pdf
The document provides a comprehensive overview of event-driven microservices, focusing on communication structures, service modeling, and event modeling. It discusses the challenges of modern architectures, including data management and scaling with synchronous services, while emphasizing the importance of event streams as a single source of truth. The content is structured in three parts: foundational concepts, event modeling techniques, and practical service modeling examples to guide effective implementation of microservices.
fundamentalsofeventdrivenmicroservices11728489736099.pdf
- 1.
- 2. Table of Contents Part1: Problems, Communication, and Event-Driven basics Part 2: Event Modelling, Schemas, & Bootstrapping Domain Data Part 3: Service Modelling, Using the DCL, & Examples
- 3. System Architectures It’s alltradeoffs
- 4.
- 5.
- 6. (Near-) Real Timeupdates
- 7.
- 8. How do wehandle these issues in the real world?
- 9. First, let’s lookat Communication Structures
- 10. Communication Structures • Business •Implementation • Data
- 11. Business Communication Structure Whatdo we do to achieve our business goals?
- 15. Implementation Communication Structure How dowe optimize the processes that lead to our business goals?
- 17. Data Communication Structures Howdo we get the data we need for our business processes?
- 19. The Implementation Communication Structure Also,The Data Communication Structure!
- 20. Ad-hoc Data CommunicationStructures
- 21. Synchronous Services canbe used to help solve these problems
- 22.
- 23.
- 24.
- 25.
- 26. Where do SynchronousServices work well?
- 27. Many companies succeed withsynchronous services! It’s all about tradeoffs
- 28.
- 29. Events are BusinessFacts
- 30. Example E-commerce • Stores, Merchants,Products, Users, Accounts Shipping • Warehouses, Orders, Shipments, Drivers, Trucks
- 31. Product A has100 units of stock User A has placed an order, ID=1234 Merchant 100’s minimum order amount is $50 Publish the Facts
- 32.
- 33. The Event-Broker storesall events and event streams
- 34.
- 35. Partitions for Scalabilityand Data Locality
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42. Principle Treat domain eventdata as a first-class citizen
- 43. Does this meaneverything should be written to an event-stream? No
- 44. Event Types &Streams 1) Entity 2) Keyed Event 3) Unkeyed Event
- 45. 1) Entity VIN MAKEMODEL COLOUR A1 Ford F150 Tan B2 Toyota Camry Gold Cars ID FIRST LAST 123 Adam Bellemare 444 Guy Incognito People Key Value An entity has a unique key VIN is Key
- 46. Materialize Entities intoEach Service Each event is the current state of a (keyed) entity! Partition 0 Overwrite “9” with “2”
- 47. 2) Keyed Events VININFRACTION AMOUNT DATE DRIVER_ID A1 Speeding $150 2018-10-07 123 A1 Parking $25 2018-11-11 123 B2 Parking $25 2018-11-13 444 Tickets Issued Key Value Multiple events with same VIN
- 48. 2) Keyed Events VININFRACTION AMOUNT DATE DRIVER_ID A1 Speeding $150 2018-10-07 123 A1 Parking $25 2018-11-11 123 B2 Parking $25 2018-11-13 444 Tickets Issued Key Value Multiple events with same VIN “This ticket belongs to VIN ID X”
- 49. Aggregating State fromKeyed Events Keyed by Shape Partition 0 Partition 1 Instance 1 Instance 0 eg: Total Ticket Costs
- 50. 3) Unkeyed Events LICENSEPLATE CAMERA_ID DATETIME IMAGE_URI ZXJ123 123 2020-07-07… s3://… ABC123 234 2020-07-08… hdfs://… ACBZ900 345 2020-07-08… c:/program… Intersection Traffic Camera Value
- 51. 3) Unkeyed Events LICENSEPLATE CAMERA_ID DATETIME IMAGE_URI ZXJ123 123 2020-07-07… s3://… ABC123 234 2020-07-08… hdfs://… ACBZ900 345 2020-07-08… c:/program… Not that Common - Usually have a key! Value May be found in “dumb” data pipelining
- 52. A Simple EnrichmentExample Cars Tickets Ticket $ per Car 1) Materialize 1) Aggregate 2) Join 3) Emit
- 53.
- 54. Part 2: EventModelling, Schemas, & Bootstrapping Domain Data
- 55. “The fundamental problemof communication is that of reproducing at one point, either exactly or approximately, a message selected at another point.” - Claude Shannon, Father of Communication Theory
- 56.
- 57. Modelling Events What businessoccurrence does this event represent?
- 58. Event Modelling Example SALESFACTS FK ITEM_ID FK STORE_ID PRICE PAYMENT ITEM DIMENSIONS PK ITEM_ID NAME BRAND CATEGORY STORE DIMENSIONS PK STORE_ID STORE_NAME ADDRESS CATEGORY Entity Stream Store Stream Key: Store_Id Event Stream Sales Stream Key: N/A Entity Stream Item Stream Key: Item_Id
- 59. Event Modelling Example- Alternative SALES FACTS PK SALES_ID FK ITEM_ID FK STORE_ID PRICE PAYMENT ITEM DIMENSIONS PK ITEM_ID NAME BRAND CATEGORY STORE DIMENSIONS PK STORE_ID STORE_NAME ADDRESS CATEGORY Entity Stream Store Stream Key: Store_Id Entity Stream Item Stream Key: Item_Id Sales Stream Key: Sales_Id Entity Stream
- 60. Event Modelling Example- Denormalized SALES FACTS PK SALES_ID FK ITEM_ID FK STORE_ID PRICE PAYMENT ITEM DIMENSIONS PK ITEM_ID NAME BRAND CATEGORY STORE DIMENSIONS PK STORE_ID STORE_NAME ADDRESS CATEGORY Enriched Sales Stream Key: Sales_Id Merged Entity Stream Join Join
- 61. Unstructured Data A recipefor disaster
- 62.
- 63. Real-life bloopers isScreenOn: “yes”“no” 1 0 true false null productId: null 1234 “null” “<null>” “%3cnull%3e” premium: true false 4040 null “null”
- 64.
- 65. Schemas ID (INTEGER) COLOUR (NULL, STRING)DEFAULT: NULL TIME (DATETIME) 100 Red 2020-08-08… 200 NULL 2020-08-08…
- 66. Producer Responsibility: Create eventsthat match the schema
- 67. Schema Options • ApacheAvro • Google’s Protobuf • JSONSchema (not plain JSON!)
- 68. (Almost) self-documenting { "type" :"record", "name" : "User", "namespace" : "com.example", "fields" : [ {"name": "username", "type": “string"}, {"name": "phone", "type": [null,"int"], "default" : null}, {"name": "address", "type": "string", "default": “UNKNOWN"}, {"name": "country", "type": "enum", "symbols": ["CA", "US", "OTHER"], "doc" : "The user's country. Select other if not in list"} ] }
- 69.
- 70. Consuming & ConvertingEvents
- 71. Backwards Compatible Can converta V2 event to V3 format at Consumer
- 72. Forwards Compatible Can converta V3 event to V2 format at Consumer
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79. Breaking Changes New Streams Newoffsets Migrating consumers Reprocessing
- 80.
- 81.
- 82. Criteria • Performance &Throughput • Indefinite retention • Tiered storage • Security & Access Control • Adoption & Open Source • 3rd-party options and tools
- 83.
- 84. Indefinite Retention It’s justbytes on disk!
- 85. Tiered Storage Put thosebytes in the Cloud!
- 86. Security and AccessControl • Secure / Encrypted connections • Data Encryption (Field Level Encryption!) • Access Control Lists (ACLs) • Enforce single-writer principle • Track stream ownership and subscriptions
- 87. Adoption & OpenSource • Avoid vendor lock-in • Hire talent • Fix own issues • Extended network
- 88. 3rd Party Options& Tools • Schema Registry • Stream Access & Visualization Tooling • Change-Data Capture Tooling • Hosting
- 89. Options Apache Kafka Apache Pulsar AWSKinesis Azure Event Hub Google Pub Sub Limited Retention (2-52 Weeks) Small Community Tiered Storage Tiered Storage (OSS & Private)
- 90. Populating Event Streams forData Communication
- 91. What is itthat we’re willing to do to provide reliable domain data? Principle: Use the event-broker as the single source of truth
- 92.
- 93.
- 94. Forked-Write Anti-Pattern Can’t writeto both data stores atomically!
- 95.
- 96.
- 97.
- 98.
- 99. Forked-Write Anti-Pattern What elsecan we do?
- 100.
- 101.
- 102.
- 103. Transactional Outbox Works! But canbe challenging to set up!
- 104.
- 105.
- 106. Question Won’t these optionsslow down the application?
- 107. Liberating Data fromLegacy Systems
- 108. Connectors & Change-DataCapture
- 109. Connectors & CDCPros Flexible Scalable Single Source of Truth
- 110. Connectors & CDCCons Idiosyncracies Overhead Performance Impact Coupling on Internal Data Model Brittle & Reactive
- 111. It’s not oneor the other! Point-to-Point or Event-Driven?
- 112. Event Streams arepart of the API
- 113.
- 114. Part 3: ServiceModelling, Using Event Streams, & Examples
- 115.
- 116.
- 117. Paying the taxlets your users focus on building the services
- 118. How do youmodel your services? Use Domain-Driven Design (again)
- 119. Modelling Services Build servicesaround business functions
- 120. Consider the ModularMonolith
- 122. Modelling Services It’s anart as much as a science
- 123. Eg Workflows builtfrom Services
- 124. Let’s look ata subset
- 125.
- 126.
- 127. Payment Processing KEY VALUE OrderIdList(ItemId) CustomerInfo KEY VALUE OrderId List(ItemId) CustomerInfo PaymentResults KEY VALUE OrderId PaymentFailureInfo PaymentAPIInfo
- 128. Payment Processing KEY VALUE OrderIdList(ItemId) CustomerInfo KEY VALUE OrderId List(ItemId) CustomerInfo KEY VALUE OrderId PaymentInfo KEY VALUE OrderId PaymentResults
- 129. Service Decides PaymentFailure Try 3 times Failed 3 times
- 130.
- 131.
- 132.
- 133.
- 134. Reduce Complexity withField-Level Encryption
- 135. Reduce Complexity withField-Level Encryption KEY VALUE OrderId List(ItemId) CustomerInfo //Encrypted PaymentInfo (CC, Address, etc) Only Authorized Services can Decrypt the Fields!
- 136. Reduce Complexity withField-Level Encryption KEY VALUE OrderId List(ItemId) CustomerInfo //Encrypted PaymentInfo (CC, Address, etc) KEY VALUE OrderId PaymentFailureInfo PaymentAPIInfo //Encrypted PaymentResults Only Authorized Services can Decrypt the Fields!
- 137.
- 138.
- 139. Schemas KEY VALUE OrderId List(ItemId) CustomerInfo KEYVALUE ItemId Map(Location,Quantity) KEY VALUE OrderId Location FulfimentInfo List(ItemId) CustomerInfo
- 140. Fulfillment Workflow Pt.1 (Automated) High-PriorityMaterialization!
- 141.
- 142.
- 143. Compensation KEY VALUE FulfimentId Location OrderId List(ItemId) CustomerInfo KEYVALUE OrderId State List(ItemId) CustomerInfo
- 144.
- 145.
- 146. Event streams asthe DCL
- 147. Event Streams: TheData Communication Layer
- 148.
- 149. Services Align withBusiness
- 151.