Konsistenz-in-verteilten-Systemen-leichtgemacht-wjax.pdf
0 likes•12 views
The document discusses distributed systems and the trade-offs between consistency and availability, emphasizing the limitations of traditional ACID transactions and the benefits of eventual consistency in modern architectures. It introduces the Calm Theorem, which states that problems can have consistent implementations without coordination if they are monotonic. The document also highlights strategies for handling conflicts in distributed systems and proposes best practices for modeling aggregates to mitigate concurrency anomalies.
Konsistenz-in-verteilten-Systemen-leichtgemacht-wjax.pdf
- 1. © Susanne Braun KEEPING CALM Konsistenz in verteilten Systemen leichtgemacht Susanne Braun 08.11.2022 w-jax
- 2. © Susanne Braun 2 @susannebraun James Hamilton Infrastructure Efficiency, Reliability and Scaling Guru Vice President & Distinguished Engineer at AWS “The first principle of successful scalability is to batter the consistency mechanisms down to a minimum, move them off the critical path, hide them in a rarely visited corner of the system, and then make it as hard as possible for application developers to get permission to use them.”
- 3. © Susanne Braun 3 @susannebraun Distributed Transactions System 4 System 1 Local ACID Transaction System 2 Local ACID Transaction System 3 Local ACID Transaction Local ACID Transaction Global XA Transaction X/Open DTP standard uses 2-Phase-Commit to ensure global atomicity It’s implemented by most relational DBs & some message brokers JTA API for Jakarta EE
- 4. © Susanne Braun 4 @susannebraun Disadvantages of Distributed Transactions n 2PC itself performs rather poor because of the communication overhead n If one system fails, the other systems will be blocked and cannot release locks held in the DB, thereby blocking other transactions n Blocking nature of 2-Phase-Commit (2PC) negatively impacts availability: n 3 systems with an availability of 99.5% each, yield overall availability of n 99.5% x 99.5% x 99.5 % = 98.5% n NoSQL DBs & Modern message brokers such as RabbitMQ and Kafka do not support XA transactions Suited for Modern Software Architecture ?
- 5. © Susanne Braun 5 @susannebraun Quality Attributes impacted by Global Coordination Fault Tolerance Resilience Loose Coupling Availability Scalability Network Partition Tolerance Low Latency Responsiveness Autonomy Global Coordination
- 6. © Susanne Braun 7 @susannebraun Eric Brewer Distributed Systems Researcher Coined the CAP theorem, Contributed to Spanner Prof. emeritus University of California, Berkeley, works now for Google “But we forfeit C and I of ACID for availability, graceful degradation and performance.” ACM Symposium on Principles of Distributed Computing, 2000
- 7. © Susanne Braun 8 @susannebraun ACID vs. BASE ACID BASE ACID Consistency (in the sense of one-copy-consistency) Isolation (in the sense of one-copy-serializability) Pessimistic Synchronization (global locks, synchronous update propagation) Global Commits (2PC, majority consensus, …) Atomicity Consistency Isolation Durability Eventual Consistency (stale data & approximate answers) Availability (top priority) Optimistic Synchronization (no locks, asynchronous update propagation) Independent Local Commits (conflict resolution, reconciliation, …) Atomicity Consistency Isolation Durability Database is in a consistent state & all invariants are being met! This is about Concurrency Control! This is about Convergence!
- 8. © Susanne Braun 9 @susannebraun Douglas Terry Distributed Systems Researcher Coined the term Eventual Consistency in the 90ties Former Prof. University of California, Berkeley, worked for Microsoft, Samsung, AWS “A system providing eventual consistency guarantees that replicas would eventually converge to a mutually consistent state, i.e., to identical contents, if update activity ceased.” Int. Conference on Parallel and Distributed Information Systems, 1994
- 9. © Susanne Braun 10 @susannebraun Douglas Terry Distributed Systems Researcher Coined the term Eventual Consistency in the 90ties Former Prof. University of California, Berkeley, worked for Microsoft, Samsung, AWS Pragmatic Definition A system provides eventual consistency if: (1)each update operation is eventually received by each replica (2)non-commutative update operations are performed in the same order at each replica (3)the outcome of a sequence of update operations is the same at each replica (determinism) Replicated Data Management for Mobile Computing, 2008
- 10. © Susanne Braun 11 @susannebraun Eventual Consistency Remember: You do not get any isolation guarantees like ‘Repeatable Read’ Hard to test Issues emerge randomly in production … are hard to reproduce … are hard to debug Application needs to handle concurrency control: Huge source of human error!
- 11. © Susanne Braun 13 @susannebraun Eventual Consistency Remember: The only guarantee you get: convergence to identical state Events / Operations coming out of order Outdated Data Conflicts Potential Concurrency Anomalies Application needs to handle: Huge source of human error!
- 12. © Susanne Braun 17 @susannebraun Conflict Handling Strategies Avoidance Primary Copy Ignoring Last writer wins Reducing the Chance Reduce granularity of conflicting items, inconsistency windows, etc. Syntactic Timestamps, logical clocks Semantic Domain-specific rules Interesting stuff!
- 13. © Susanne Braun 18 @susannebraun Conflict Handling Strategies Avoidance Primary Copy Ignoring Last writer wins Reducing the Chance Reduce granularity of conflicting items, inconsistency windows, etc. Syntactic Semantic Domain-specific rules Interesting stuff! Timestamps, logical clocks
- 14. © Susanne Braun 19 @susannebraun Pat Helland Database & Distributed Systems Guru Architect of multiple transaction & database systems (e.g. DynamoDB) Worked at Microsoft, Amazon, SalesForce, … Conference on Innovative Data Systems Research, 2009 A C I D 2.0 Associative Commutative Idempotent Distributed (ab)c = a(bc) ab = ba aa = a Distributed Operations
- 15. © Susanne Braun 25 @susannebraun S h o p p i n g C a r d ACID 2.0 By Example – Shopping Cart with State-based Conflict Resolution Shopping Cart － Soap － Lotion Shopping Cart － Lotion － Brush Join Shopping Cart － Soap － Lotion － Brush associativity: 𝑠 ∨ 𝑡 ∨ 𝑢 = 𝑠 ∨ 𝑡 ∨ 𝑢 commutaXvity: 𝑠 ∨ 𝑡 = 𝑡 ∨ 𝑠 idempotence: 𝑠 ∨ 𝑠 = 𝑠 Deleted items might reappear * Werner Vogels, Communications of the ACM, Volume 52, Issue 1, 2009
- 16. © Susanne Braun 27 @susannebraun Can we do better?
- 17. © Susanne Braun 28 @susannebraun S h o p p i n g C a r d Clever Programmer’s Shopping Cart Shopping Cart Bounded Context ShoppingCart id: Identifier lineItems: Set<LineItem> handleAddItemAction(AddItemAction a) Product AddItemAction id: Identifier Customer id: Identifier 1 1 1 * 1 * Bold Font: Aggregate Root Italic Font: Value Object : Cross-Aggregate Reference 1 1
- 18. © Susanne Braun 29 @susannebraun S h o p p i n g C a r d Clever Programmer’s Shopping Cart Commutative
- 19. © Susanne Braun 30 @susannebraun S h o p p i n g C a r d What happens if we allow deletion of items? Shopping Cart Bounded Context ShoppingCart id: Identifier lineItems: Set<LineItem> handleShoppingCartAction(ShoppingCartAction a) Product ShoppingCartAction id: Identifier Customer id: Identifier 1 1 1 * 1 * Bold Font: Aggregate Root Italic Font: Value Object : Cross-Aggregate Reference 1 1 AddItemAction id: Identifier DeleteItemAction id: Identifier
- 20. © Susanne Braun 31 @susannebraun S h o p p i n g C a r d What happens if we allow deletion of items? Non-Commutative
- 21. © Susanne Braun 32 @susannebraun S h o p p i n g C a r d Actions might be processed in different orders at different Replicas… Replica 1 Replica 2 add(Rose) add(Lupine) delete(Rose) add(Lupine) delete(Rose) 1 x Rose Leonardo da Vinci 1 x Rose Leonardo da Vinci 1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids 1x Foxgloves Mix add(Foxgloves) add(Foxgloves) add(Rose) 1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids 1 x Lupine – Russel’s Hybrids 1x Foxgloves Mix 1 x Lupine – Russel’s Hybrids 1x Foxgloves Mix 1 x Rose Leonardo da Vinci
- 22. © Susanne Braun 33 @susannebraun S h o p p i n g C a r d Item Deletion introduces Oder Dependence n Adding items to a set is a monotonic function n processing add-item-actions always increases the result set n Allowing deletion of items destroys this property n State toggles back and forth: item in the set, item not in the set, item in the set, … 0 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 10 F(x) F(x) Processing shopping cart actions becomes non-commutative!
- 23. © Susanne Braun 34 @susannebraun Can we do better?
- 24. © Susanne Braun 35 @susannebraun S h o p p i n g C a r d The cleverest programmers do the following… Shopping Cart Bounded Context ShoppingCart id: Identifier addedItems: Set deletedItems: Set handleShoppingCartAction(ShoppingCartAction a) getLineItems(): Set<LineItem> Product ShoppingCartAction id: Identifier Customer id: Identifier 1 1 1 * 1 * Bold Font: Aggregate Root Italic Font: Value Object : Cross-Aggregate Reference 1 1 AddItemAction id: Identifier DeleteItemAction id: Identifier Monotonic
- 25. © Susanne Braun 36 @susannebraun S h o p p i n g C a r d The cleverest programmers do the following… Monotonic Commutative Derives shopping cart state
- 26. © Susanne Braun 37 @susannebraun I n s i g h t Monotonic Problems n By modelling state differently: n two monotonic insert-only sets n Inserts into these sets commute n Enabled us to process requests (ShoppingCartActions) independent of order * Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72– 81. https://dl.acm.org/doi/10.1145/3369736 Monotonic Problems output depends only on the the content of their input, not in the order in which it arrives!*
- 27. © Susanne Braun 38 @susannebraun F o u n d a t i o n s The CALM Theorem Consistency As Logical Monotonicity ( C A L M ) A problem has a consistent, coordination-free distributed implementation if and only if it is monotonic. Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72– 81. https://dl.acm.org/doi/10.1145/3369736
- 28. © Susanne Braun 39 @susannebraun Joseph Hellerstein Databases & Data-Centric Computing Researcher Coined the CALM theorem, Founder of TriFacta Jim Gray Professor of the University of California, Berkeley “Intuitively, monotonic problems are ”safe” in the face of missing information and can proceed without coordination.” Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72–81.
- 29. © Susanne Braun 42 @susannebraun I n s i g h t From CAP to CALM CAP is a negative result: it captures properties that cannot be achieved in general. CAP only holds if we assume the system in question is required to execute arbitrary programs. CALM is a positive result: it circumscribes the class of problems which can remain available under partition. When the partition heals, monotonic problems converge to identical state without the need for further coordination or conflict resolution. * Joseph M. Hellerstein and Peter Alvaro. 2020. Keeping CALM: when distributed consistency is easy. Commun. ACM 63, 9 (2020), 72– 81. https://dl.acm.org/doi/10.1145/3369736
- 30. © Susanne Braun 43 @susannebraun Proofs of the CALM Theorem
- 31. © Susanne Braun 45 @susannebraun Conflict Handling Strategies Avoidance Primary Copy Ignoring Last writer wins Reducing the Chance Reduce granularity of conflicting items, inconsistency windows, etc. Syntactic Timestamps, logical clocks, identifiers, .. Semantic Domain-specific rules Interesting stuff!
- 32. © Susanne Braun 47 @susannebraun Pat Helland Database & Distributed Systems Guru Architect of multiple transaction & database systems (e.g. DynamoDB) Worked at Microsoft, Amazon, SalesForce, … “Immutability Changes Everything” ACM Queue, Volume 13, Issue 9, 2016
- 33. © Susanne Braun 48 @susannebraun T a x o n o m y 1st Level Classification of Replicated Aggregates Mutable ? Immutable Aggregates* No Concurrent In-Place Updates ? Yes No Derived Aggregates* Multiple Updaters ? Yes Dedicated Aggregates Nontrivial Aggregates Yes No * “Append-Only Computing” – Helland 2015 Immutable Aggregates: • Time series data (machine sensor data, market data, …) • Domain events Derived Aggregates: • Machine generated data (recommendations, …) • Timeline or newsfeed data Dedicated Aggregates: • User generated data (reviews, social media posts, ...) • Dedicated master data (user profiles, account settings) E x a m p l e s
- 34. © Susanne Braun 49 @susannebraun T a x o n o m y 2nd Level Classification of Nontrivial Aggregates Activity Aggregates Collaboration Result Aggregates Reference Aggregates Dedicated Aggregates Derived Aggregates Observed Aggregates Update Frequency in Peak Times Update Simultaneity in Peak Times Concurrency Anomaly Probability low improbable low - Nontrivial Aggregates Reference Aggregates Examples: • Master data (CRM data, resources, products, …) • Values (Valid currencies, product types, gender, …) • Meta data (Tags, descrtiptive data of raw data, ..)
- 35. © Susanne Braun 50 @susannebraun T a x o n o m y 2nd Level Classification of Nontrivial Aggregates Activity Aggregates Collaboration Result Aggregates Reference Aggregates Dedicated Aggregates Derived Aggregates Observed Aggregates Update Frequency in Peak Times Update Simultaneity in Peak Times Concurrency Anomaly Probability low high improbable probable low high - Nontrivial Aggregates Reference Aggregates Examples: • Master data (CRM data, resources, products, …) • Values (Valid currencies, product types, gender, …) • Meta data (Tags, descrtiptive data of raw data, ..) Activity Aggregates Examples: • State data of workflows, business processes, … • Coordination data of joint activities (agricultural field operation, meeting, …) • Task management data, Kanban board data, …
- 36. © Susanne Braun 51 @susannebraun T a x o n o m y 2nd Level Classification of Nontrivial Aggregates Activity Aggregates Collaboration Result Aggregates Reference Aggregates Dedicated Aggregates Derived Aggregates Observed Aggregates Update Frequency in Peak Times Update Simultaneity in Peak Times Concurrency Anomaly Probability low high very high improbable probable highly probable low high very high - Nontrivial Aggregates Reference Aggregates Examples: • Master data (CRM data, resources, products, …) • Values (Valid currencies, product types, gender, …) • Meta data (Tags, descrtiptive data of raw data, ..) Activity Aggregates Examples: • State data of workflows, business processes, … • Coordination data of joint activities (agricultural field operation, meeting, …) • Task management data, Kanban board data, … Collaboration Result Aggregates Examples: • Result data of collaborative knowledge work (CAD model, crop rotation plan, whiteboard diagram, …) • Text data as result of collaborative authorship (manuals, scientific papers, meeting protocols, …)
- 37. © Susanne Braun 52 @susannebraun T a x o n o m y 2nd Level Classification of Nontrivial Aggregates Activity Aggregates Collaboration Result Aggregates Reference Aggregates Dedicated Aggregates Derived Aggregates Immutable Aggregates Update Frequency in Peak Times Update Simultaneity in Peak Times Concurrency Anomaly Probability “Technical Immutability Border” low high very high improbable probable highly probable low high very high - low improbable Nontrivial Aggregates
- 38. © Susanne Braun 55 @susannebraun B e s t P r a c t i c e Make Well-Informed Trade-Off Decisions Concurrency Anomaly Probability Fixing Costs of Data Corruption low high very high very high high very high moderate Reference Aggregates Activity Aggregates Collaboration Result Aggregates Dedicated Aggregates Derived Aggregates Immutable Aggregates low Consequences of Data Corruption critical major critical minor Nontrivial Aggregates “Technical Immutability Border” Trivial Aggregates A Classification of Replicated Data for the Design of Eventually Consistent Domain Models, S. Braun, S. Dessloch, ICSA 2020
- 39. © Susanne Braun 56 @susannebraun Eventual Consistency is standard Estimation - Frequency of Distribution Trad. Enterprise IS (ERP, CRM, Workflow Management) Social Media Apps (Facebook, Twitter) Next Data-Intensive Systems (Smart Farming, Industrie 4.0) 30% 30% 9 % 45% 4% 1% 20% 20% 20% 20% “Technical Immutability Border” Reference Aggregates Activity Aggregates Collaboration Result Aggregates Dedicated Aggregates Derived Aggregates Immutable Aggregates 1 % 50% 20% 30 %
- 40. © Susanne Braun 57 @susannebraun B e s t P r a c t i c e Trivial Aggregates First n Whenever feasible, model aggregates as trivial aggregates Examples of Immutable Aggregates • Domain Events !! • A confirmed order • A released shift plan • A submitted questionnaire • A submitted offer • A placed bid • A survey gone live • …. Examples of Derived Aggregates Derived Monotonic State (CALM Theorem!) • The highest bid (max) of an auction • The state of progressive workflows (state changes increase monotonically) • The number of in-stock items (goods receipts and order confirmations are monotonic sets) Derived Views • Mobile-optimized representations • … Low hanging fruit that is often overlooked!
- 41. © Susanne Braun 58 @susannebraun The Derived Monotonic State Pattern
- 42. © Susanne Braun 59 @susannebraun D e s i g n P a t t e r n C o n t e x t A n t i - P a t t e r n ShoppingCart id: Identifier LineItem productRef: Identifier number: int 0..* 1 Concurrent In-place Updates Nontrivial Activity Aggregate Product productId: Identifier Reference Aggregate DeleteItemAction AddItemAction 0..* 1 0..* 1 Bold Font: Aggregate Root Italic Font: Value Object : Cross-Aggregate Reference
- 43. © Susanne Braun 60 @susannebraun D e s i g n P a t t e r n F o r c e s n Performance and availability of functionality implemented with or based on the aggregate is a crucial business factor. n A conflict resolution scheme that is “good enough” cannot be implemented with reasonable effort.
- 44. © Susanne Braun 61 @susannebraun D e s i g n P a t t e r n S o l u t i o n Nontrivial Activity Aggregate Activity fun deriveCurrentSate(…) Entity Value Object Activity State Entity Value Object Entity Value Object Entity Value Object Domain Event Entity Value Object Less complex Activity Aggregate Derived Aggregate Immutable Aggregates Cross-Aggregate References Factor Out Activity Entity Value Object Entity Event
- 45. © Susanne Braun 62 @susannebraun D e s i g n P a t t e r n C o n t e x t A n t i - P a t t e r n ShoppingCart id: Identifier LineItem productRef: Identifier number: int 0..* 1 Concurrent In-place Updates Nontrivial Activity Aggregate Product productId: Identifier Reference Aggregate DeleteItemAction AddItemAction 0..* 1 0..* 1 Bold Font: Aggregate Root Italic Font: Value Object : Cross-Aggregate Reference
- 46. © Susanne Braun 63 @susannebraun D e s i g n P a t t e r n S o l u t i o n E x a m p l e Bold Font: Aggregate Root Italic Font: Value Object : Cross-Aggregate Reference Immutable Aggregates ShoppingCart id: Identifier Derived Aggregate LineItem productRef: Identifier number: int 1 1..* LineItems cartRef: Identifier Product id: Identifier Reference Aggregates Calculated periodically or on demand AddItemAction id: Identifier cartRef: Identifier productRef: Identifier DeleteItemAction id: Identifier cartRef: Identifier productRef: Identifier getLineItems(): LineItem
- 47. © Susanne Braun 65 @susannebraun D e s i g n P a t t e r n B e n e f i t s n Increased share of trivial aggregates n Size and chance for conflicts of remaining nontrivial aggregates is considerably lower n Complexity of the resulting model is reduced n Aggregates are consisting of fewer domain objects with fewer object associations
- 48. © Susanne Braun 66 @susannebraun The Segregate Aggregate Classes Pattern
- 49. © Susanne Braun 67 @susannebraun D e s i g n P a t t e r n C o n t e x t A n t i - P a t t e r n Nontrivial Activity Aggregate Field Assignment DocRecord 0..* 0..* 1 1 Job id: Identifier status: JobStatusEnum OperationType id: Identifier name: String StaffAssignment MachineAssignment Machine StaffMember 0..1 0..* 0..* 0..* 0..* 1 0..* 1 1 0..* id: Identifier id: Identifier id: Identifier id: Identifier id: Identifier Dedicated to single user Reference data that is rarely updated
- 50. © Susanne Braun 68 @susannebraun D e s i g n P a t t e r n F o r c e s n Performance and availability of functionality implemented with or based on the aggregate is a crucial business factor. n A conflict resolution scheme that is “good enough” cannot be implemented with reasonable effort.
- 51. © Susanne Braun 69 @susannebraun D e s i g n P a t t e r n S o l u t i o n Complex non-trivial Aggregate Dedicated Aggregates Cross-Aggregate References Less complex non- trivial Aggregate Aggregate Entity Value Object Reference Aggregates Entity Value Object Entity Value Object Aggregate Entity Value Object Entity Value Object Entity Value Object Aggregate Entity Value Object Derived Aggregates Entity Value Object Entity Value Object Aggregate Entity Value Object Immutable Aggregates Entity Value Object Entity Value Object Aggregate Entity Value Object Refactor Into Aggregate Entity Value Object Entity Event Entity Value Object Event
- 52. © Susanne Braun 70 @susannebraun D e s i g n P a t t e r n C o n t e x t A n t i - P a t t e r n Nontrivial Activity Aggregate Field Assignment DocRecord 0..* 0..* 1 1 Job id: Identifier status: JobStatusEnum OperationType id: Identifier name: String StaffAssignment MachineAssignment Machine StaffMember 0..1 0..* 0..* 0..* 0..* 1 0..* 1 1 0..* id: Identifier id: Identifier id: Identifier id: Identifier id: Identifier Dedicated to single user Reference data that is rarely updated
- 53. © Susanne Braun 71 @susannebraun D e s i g n P a t t e r n S o l u t i o n E x a m p l e Less complex Activity Aggregate Assignment 0..* 1 Job id: Identifier status: JobStatusEnum operationType: OperationTypeEnum fields: Set<Identifier> docRecords: Set<Identifier> MachineAssignment id: Identifier machineRef: Identifier StaffAssignment id: Identifier staffMemberRef: Identifier Field id: Identifier Machine id: Identifier StaffMember id: Identifier Reference Aggregates DocRecord id: Identifier jobRef: Identifier staffMemberRef: Identifier Dedicated Aggregate
- 54. © Susanne Braun 73 @susannebraun D e s i g n P a t t e r n B e n e f i t s n By segregation of domain objects belonging to different classes large aggregates can be significantly reduced in terms of size, complexity and ramification of object associations n As aggregates become smaller the chances for conflicting updates are reduced n By factoring out immutable, derived and dedicated data into standalone aggregates the advantages of trivial aggregates can be exploited as for these aggregates no conflict resolution schemes are required. n By factoring out reference aggregates low update frequency of reference data can be exploited n As chances for conflicting updates are low simple automatic reconciliation schemes or even manual reconciliation schemes are usually sufficient.
- 55. © Susanne Braun 77 @susannebraun Open Access Design Guidelines n Code Examples n https://github.com/brausu/workshop-eventual-consistency n Follow me on Twitter and GitHub @susannebraun EventuallyConsistentDDD/design-guidelines We’ re on Github! Thanks!
- 56. © Susanne Braun 78 @susannebraun Our latest Study https://arxiv.org/pdf/2108.03758.pdf
- 57. © Susanne Braun 79 @susannebraun Checkout our other publications https://doi.org/10.1109/ICSA-C50368.2020.00014 https://doi.org/10.1145/3447865.3457969
- 58. © Susanne Braun 80 @susannebraun #Thanx SAP Signavio, Berlin Susanne Braun Principal Tech Lead [email protected] @susannebraun EventuallyConsistentDDD/design-guidelines