Eventually-Consistent Data Structures

Editor's Notes

• #2: \n
• #3: \n
• #4: \n
• #5: In an eventually consistent system, you tend to have multiple copies of the same datum, which means that it’s replicated. They also tend to allow loose coordination and things like sloppy quora, since you don’t require expensive multi-phase commit protocols. This also makes them resilient to network partitions. Eventually consistent systems must also include means for state to move forward when staleness is detected. In Dynamo-like systems, this is usually done with read-repair, that is, writing the newer value to stale replicas when reading.\n
• #6: While not as simple to understand as an ACID system, eventual consistency has many practical benefits. When encountering failures, especially network-related ones, the system can more often remain available to reads and writes despite the failures. In the same vein, relying on dynamic participation in operations lends itself to systems with low, consistent latency because only promptly-responding replicas need to be considered.\n
• #7: Of course the tradeoff of those benefits, thanks to the CAP theorem, is that you sacrifice strict consistency. There is no total ordering of events in the system, you have no transactions, you have weak guarantees of delivery. This means it’s incredibly difficult to decide who wins when there are concurrent writes in the system. The solutions to the problem are both non-ideal, but they are generally: first, to throw one version out by applying an arbitrary ordering, usually a timestamp of sorts; second, to keep both values around and let the user decide. These are the approaches of Cassandra, and Riak/Voldemort respectively.\n
• #8: Of course the tradeoff of those benefits, thanks to the CAP theorem, is that you sacrifice strict consistency. There is no total ordering of events in the system, you have no transactions, you have weak guarantees of delivery. This means it’s incredibly difficult to decide who wins when there are concurrent writes in the system. The solutions to the problem are both non-ideal, but they are generally: first, to throw one version out by applying an arbitrary ordering, usually a timestamp of sorts; second, to keep both values around and let the user decide. These are the approaches of Cassandra, and Riak/Voldemort respectively.\n
• #9: So maybe you chose Riak or Voldemort, you get write conflicts (Riak calls them siblings). Now that you’ve got both values, how does your application decide what the real state should be?\n
• #10: One strategy, which I call “semantic resolution”, is to say that your application encodes the domain of the problem and so it can use business rules to resolve the conflict. This is the strategy implemented by the “shopping cart” described in the Amazon Dynamo paper. It merges toward the maximum quantity of each item in the cart; however, it exhibits some problems -- namely that sometimes items that were removed from the cart can reappear! From Amazon’s point of view this is okay because it might encourage the customer to buy more, but it is a bewildering user-experience!\n\nFortunately, there is some interesting recent research about a more rigorous approach to eventual consistency.\n\n\n
• #11: They are sometimes called Conflict-Free Replicated Data Types. This basically means that instead of strictly opaque values, the datastore provides useful abstract data structures. Since we’re in an eventually consistent system, the data structure is replicated to multiple locations, all of which act independently. But by far the most compelling part is that these data structures have the ability to resolve automatically toward a single value, given any number of conflicting values at individual replicas.\n\n\n
• #12: They are sometimes called Conflict-Free Replicated Data Types. This basically means that instead of strictly opaque values, the datastore provides useful abstract data structures. Since we’re in an eventually consistent system, the data structure is replicated to multiple locations, all of which act independently. But by far the most compelling part is that these data structures have the ability to resolve automatically toward a single value, given any number of conflicting values at individual replicas.\n\n\n
• #13: They are sometimes called Conflict-Free Replicated Data Types. This basically means that instead of strictly opaque values, the datastore provides useful abstract data structures. Since we’re in an eventually consistent system, the data structure is replicated to multiple locations, all of which act independently. But by far the most compelling part is that these data structures have the ability to resolve automatically toward a single value, given any number of conflicting values at individual replicas.\n\n\n
• #14: The primary work on this research has been done by two researchers at INRIA and their colleagues in Portugal. Marc Shapiro also gave a great talk on the subject at Microsoft Research called “Strong Eventual Consistency” which you can easily find online.\n\nThe paper above is where I’ve gotten most of the content and diagrams, but I’ve tried to simplify the content so that we can get through it in 40 minutes. If you want the real thing, search for <title>, it’s free to download.\n
• #15: There are two flavors of CRDTs as you might have noticed. They both provide the same conflict-free property, but differ in their implementation strategy.\n\nConvergent types are based on a local modification of state, followed by forwarding the resulting state downstream, where a merge operation is performed at other replicas. The state itself encodes all information needed to converge. They are great for systems with weak message delivery guarantees - for example, a Dynamo-style system. Convergent types can also be resolved in clients, which is helpful for systems that do not provide rich datatypes.\n\nCommutative types, on the other hand, replicate commutative operations rather than state, and tend to rely on systems with reliable broadcast (that assures operations reach all replicas). Operations are generally not required to have a total ordering -- a local causal ordering is sufficient.\n
• #16: This diagram from the paper shows the basic format of a convergent, state based CRDT. Note how the mutation is applied locally, then forwarded downstream as a merge operation. As long as all replicas eventually receive states that include all mutations, they will converge on the same value.\n
• #17: Again, in Commutative types forward operations to other replicas, not the state. Obviously, if an operation is not delivered, or applied out-of-order locally, the states don’t converge. However, again, unlike the convergent type, a reliable broadcast channel is required. As long as functions f() and g() commute, state will converge.\n
• #18: A register is the simplest type of data structure - a memory cell storing an opaque value. It only supports two operations - “assign” and “value” (get and set). Concurrent updates will not commute (who should win?). We’ve seen this problem before.\n
• #19: The two approaches to concurrent resolution are the same ones taken by Cassandra and Riak, respectively. That is, Last-Write-Wins (called an LWW-Register) and Multi-Valued (called MV-Register)-- keeping all divergent values. For resolution, LWW tend to use timestamps with a reasonable guarantee of ordering (which is difficult in practice, but in some systems sufficient). MV on the other hand, requires the more expensive version vector to resolve conflicts and produces the union of all divergent values (but it doesn’t behave like a set!)\n
• #20: Counters are simply integers that are replicated and support the increment and decrement operations. Counters are useful for things like tracking the number of logged-in users, or click-throughs on an advertisement.\n\nThe simplest type of counter is a Commutative or operation-based type, since add and subtract are commutative, any delivery order is sufficient (ignoring over-/under-flow). The state-based counters are more interesting so we’ll look at those.\n
• #21: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #22: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #23: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #24: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #25: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #26: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #27: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #28: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #29: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #30: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #31: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #32: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #33: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #34: A G-Counter only counts up and is basically a version vector (vector clock). Each replica increments its own pair only, the value is computed by summing the count of all replicas. Convergence is achieved by taking the maximum count for each replica. This is basically the Cassandra counters implementation.\n
• #35: PN-Counter - composed of two G-Counters - P for increments and N for decrements. The value is the difference between the values of the two G-Counters. The resolution is the pairwise resolution of the P and N counters.\n
• #36: Sets constitute one of the most basic data structures. Containers, Maps, and Graphs are all based on Sets. There are two operations, add and remove.\n
• #37: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #38: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #39: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #40: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #41: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #42: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #43: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #44: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #45: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #46: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #47: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #48: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #49: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #50: Like a G-Counter, a G-Set only grows in size. That is, it doesn’t allow removal - its merge operation is a simple set-union, returning the maximal grouping without duplicates. Since add commutes with union, a G-Set can also be implemented as a commutative type. However, it’s not an incredibly useful data-type on its own, but it can be part of another data structure.\n
• #51: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #52: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #53: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #54: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #55: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #56: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #57: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #58: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #59: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #60: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #61: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #62: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #63: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #64: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #65: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #66: The second type of Set is a two-phase set, where a removed set member cannot be re-added. It is basically two G-Sets, one for add and one for remove. The removal set is sometimes called a tombstone set. To prevent spurious states (e.g. remove-before-add, making add have no effect), it has a precondition for remove that the local state must already contain the member.\n\nA special case of the 2P-Set is the U-Set. If the system can reasonably guarantee uniqueness, that is, the element will never be added again after removal, then the tombstone set is unnecessary. Uniqueness could be satisfied with a Lamport clock or suitably large RNG space.\n
• #67: Tag each element in A and R with timestamp. Greatest timestamp wins out for each individual element. Could be implemented with Cassandra super-columns.\n\nFigure 12: LWW-element-Set; elements masked by one with a higher timestamp are elided (state-based)\n\n
• #68: Tag each added element uniquely (without exposing them). When removing, remove all seen and forward operation downstream with tags. State-based version would be based on U-Set.\n\n
• #69: You might notice we’re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can’t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I’m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
• #70: You might notice we’re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can’t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I’m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
• #71: You might notice we’re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can’t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I’m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
• #72: You might notice we’re going up in complexity here in terms of the types of data-structures. Graphs are incredibly useful for many problems, but also have a bunch of potential anomalies within them - concurrent add/removes of vertices and edges may not converge - that is, global invariants can’t be guaranteed. For example, in the case of a DAG or linked-list where elements can be removed or added concurrently. Some anomalies may be removed via restricting the semantics, for example, making a graph add-only. I’m not going to go into detail about how Graphs are implemented, but a simple one is the 2P2P graph, based on a pair of 2P-sets, one for vertices and one for edges. In the case where a vertex is removed, the most reliable (and intuitive) solution is to remove all attached edges, thus a 2P-Set paradigm works well for the components of a generic graph.\n\n\n
• #73: \n
• #74: CRDTs tend to create a lot of garbage: tombstones grow and internal structures become unbalanced. In general, garbage collection is extremely difficult to do without synchronization. Luckily, this doesn’t impact correctness, only efficiency and performance.\n
• #75: Client - have to come up with a common representation across languages, allocation of actor IDs is problematic, can only use state-based CRDTs.\nServer - no one implements them yet, really (Cassandra’s counter has some anomalies)\n
• #76: \n