Intro to Databases
2 likes884 views
The document discusses databases, their organization, and the challenges related to distributed systems and eventual consistency. It covers key concepts like ACID properties, CAP theorem, and different database architectures, including insights on Google's F1 database and the evolution of technologies addressing consistency issues. Additionally, it explores various approaches for managing data replication, fault tolerance, and the complexity of transaction handling in distributed environments.
![Model
curl -s http://localhost:8098/types/test/buckets/test/
datatypes/sargun |python -mjson.tool
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}](https://image.slidesharecdn.com/databases2-150827095446-lva1-app6892/85/Intro-to-Databases-120-320.jpg)
![“Primary Key”
curl -s http://localhost:8098/types/test/buckets/test/
datatypes/sargun |python -mjson.tool
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}](https://image.slidesharecdn.com/databases2-150827095446-lva1-app6892/85/Intro-to-Databases-121-320.jpg)
![Causal Context
curl -s http://localhost:8098/types/test/buckets/test/
datatypes/sargun |python -mjson.tool
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}](https://image.slidesharecdn.com/databases2-150827095446-lva1-app6892/85/Intro-to-Databases-122-320.jpg)
![Updated Entries
(during partition)
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq",
"type": "map",
"value": {
"friends_set": [
"Boss"
],
"posts_set": []
}
}
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQtq",
"type": "map",
"value": {
"friends_set": [],
"posts_set": []
}
}](https://image.slidesharecdn.com/databases2-150827095446-lva1-app6892/85/Intro-to-Databases-124-320.jpg)
![Updated Entries
(After Healing)
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQ5q",
"type": "map",
"value": {
"friends_set": [],
"posts_set": [
"Party"
]
}
}
{
"context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQ5q",
"type": "map",
"value": {
"friends_set": [],
"posts_set": [
"Party"
]
}
}](https://image.slidesharecdn.com/databases2-150827095446-lva1-app6892/85/Intro-to-Databases-126-320.jpg)
Intro to Databases
- 2. What is a database? A database is an organized collection of data
- 5. Internet Traffic vs. Penetration 0 25 50 75 100 0 10000 20000 30000 40000 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 IP Traffic (PB/mo) Global Penetration (%)
- 6. Number of Internet Users in 2012
- 7. Average Distance to Every Human
- 8. Extrapolating We have not yet reached Peak “Web” and we won’t see it for some time
- 9. Applications How are they built?
- 12. Scale Out
- 13. Scale Out with Correctness
- 14. What is a Transaction? A Unit of Work
- 17. ACID
- 18. ACID = Atomicity A transaction executes or it does not
- 19. ACID = Consistency Correctness; Require the database to follow set of invariants
- 20. ACID = Isolation Prevent inter-actor visibility during concurrent operations
- 21. ACID = Durability Once you write, it will survive
- 22. Lifecycle of a Transaction
- 23. Vertically Scalability Moore’s Law can take us places
- 24. Biggest AWS Database • vCPUs: 32 • Memory: 244 • Storage: 3TB • IOPs: 30,000 IOPs • Networking: 10 Gigabit • Resiliency: Multi-AZ • SLA: 99.95% • Backend: Postgresql
- 25. $141,052.66/yr
- 26. Scaling Beyond
- 27. Sharding?
- 29. Do we have a natural sharding key?
- 31. Two-phase commit? Three-phase commit? Paxos? Enhanced Three-phase commit? Wat? Egalitarian Paxos?
- 32. Do we really want to run NxM databases?
- 35. Database Failure
- 37. Recovery
- 39. Scaling SSI databases is a hard problem
- 44. No latency win for mutable data
- 45. Must sacrifice recency for latency win
- 47. Multi-master requires at least 1 RTT
- 48. -F1: A Distributed SQL Database That Scales, Google “Because the data is synchronously replicated across multiple datacenters, and because we’ve chosen widely distributed datacenters, the commit latencies are relatively high (50-150 ms).”
- 49. -Kohavi and Longbotham 2007 “Every 100 ms increase in load time of Amazon.com decreased sales by 1%.” (~$120M of losses per 100 ms)
- 50. “Average partition duration ranged from 6 minutes for software-related failures to more than 8.2 hours for hardware-related failures (median 2.7 and 32 minutes; 95th percentile of 19.9 minutes and 3.7 days, respectively).” -The Network is Reliable WANs Fail
- 51. Is there another way?
- 53. -F1: A Distributed SQL Database That Scales, Google “We also have a lot of experience with eventual consistency systems at Google. In all such systems, we find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency and handle data that may be out of date. We think this is an unacceptable burden to place on developers and that consistency problems should be solved at the database level. ”
- 54. CAP Theorem
- 55. “A shared-data system can have at most two of the three following properties: Consistency, Availability, and tolerance to network Partitions.” -Dr. Eric Brewer
- 56. On Consistency • ACID Consistency: Any transaction, or operation will bring the database from one valid state to another • CAP Consistency: All nodes see the same data at the same time (synchrony)
- 57. On Partition Tolerance • The network will be allowed to lose arbitrarily many messages sent from one node to another. • Databases systems, in order to be useful must have communication over the network • Clients count
- 58. There is no such thing as a 100% reliable network: Can’t choose CA http://codahale.com/you-cant-sacrifice-partition-tolerance
- 59. We Can Have Both* (*Just not at the same time)
- 60. PNUTS • Paper released by Yahoo! research in 2008 • Operations: • Read-Any • Read-Critical(Required-Version)* • Read-Latest • Write • Test-and-set-write(Required-Version) * Will fall back to CP operation
- 61. Weak Consistency
- 62. Weak Consistency
- 63. “This is a specific form of weak consistency; the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value.” Definition of “Eventual Consistency” from “Eventually Consistency Revisited” - Werner Vogels
- 65. Eventual Consistency in the LAN
- 67. Good at Building LANs at Scale
- 68. Facebook Fabric
- 69. Microsoft VL2
- 70. Google Jupiter
- 71. Less Interesting
- 72. Eventual Consistency in the WAN
- 74. Write Anywhere Beat the speed of the light
- 75. Build for WAN locality
- 76. Typical Pattern with COTS EC Store
- 77. System Model
- 80. Schema CREATE TABLE test.users ( user_name text PRIMARY KEY, friends set<text>, posts set<text> )
- 81. State *****:test> SELECT * FROM users; user_name | friends | posts -----------+----------+------- sargun | {'BOSS'} | null
- 83. Remove Boss *****:test> UPDATE users SET friends = friends - {'BOSS'} WHERE user_name = 'sargun' ;
- 84. Hidden Failure
- 86. State at DC2 & DC3 *****:test> SELECT * FROM users; user_name | friends | posts -----------+----------+------- sargun | {'BOSS'} | null
- 87. Post Message *****:test> UPDATE users SET posts = posts + {'PARTY'} WHERE user_name = 'sargun' ;
- 88. State at DC2 & DC3 *****:test> SELECT * FROM users; user_name | friends | posts -----------+----------+----------- sargun | {'BOSS'} | {'PARTY'}
- 89. Worse Than Banking Unbounded Financial Loss
- 91. Solution: Wait For Acks
- 92. Very Little Benefit Over CP system Quorum Systems
- 93. RYOW at an Incredible Cost
- 94. Why not just do Paxos*? Single-Decree Paxos Variant such as EPaxos, Cheap Paxos, or Multi-Paxos
- 97. Quorum
- 98. Quorum
- 100. Just Perform Paxos Reconfiguration to Recover from Failure
- 104. Is there an alternative?
- 106. Strong Eventual Consistency “Any set of nodes that have received the same (unordered) set of updates will be in the same state.”
- 107. How do you even use this?
- 108. Vector Clocks
- 109. Vector Clocks • Extension of Lamport Clocks • Used to detect cause and effect in distributed systems • Can determine concurrency of events, and causality violations • Preserves h.b. relationships
- 110. CRDTs
- 111. • CRDTs: • Convergent Replicated Data Types • Commutative Replication Data Types • Enables data structures to be always writeable on both sides of a partition, and replay after healing a partition • Enable distributed computation across monotonic functions • Two Types: • CvRDTs • CmRDTs CRDTs
- 112. CvRDTs • State / value based CRDTs • Minimal state • Don’t require active garbage collection
- 113. Set CvRDT
- 114. CmRDTs • Op / method based CRDTs • Size grows monotonically • Uses version vectors to determine order of operations
- 115. Counter CmRDT
- 116. CRDTs in the Wild • Sets • Observe-remove set • Grow-only sets • Counters • Grow-only counters • PN-Counters • Flags • Maps
- 117. Data structures that are CRDTs • Probabilistic, convergent data structures • Hyper log log • Bloom filter • Co-recursive folding functions • Maximum-counter • Running Average • Operational Transform
- 118. CRDTs • Incredibly powerful primitive • Not only useful for in-database manipulation but client-database interaction • You can compose them, and build your own • Garbage collection is tricky
- 119. Riak In Action
- 120. Model curl -s http://localhost:8098/types/test/buckets/test/ datatypes/sargun |python -mjson.tool { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq", "type": "map", "value": { "friends_set": [ "Boss" ], "posts_set": [] } }
- 121. “Primary Key” curl -s http://localhost:8098/types/test/buckets/test/ datatypes/sargun |python -mjson.tool { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq", "type": "map", "value": { "friends_set": [ "Boss" ], "posts_set": [] } }
- 122. Causal Context curl -s http://localhost:8098/types/test/buckets/test/ datatypes/sargun |python -mjson.tool { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq", "type": "map", "value": { "friends_set": [ "Boss" ], "posts_set": [] } }
- 123. Update curl -XPOST http://localhost:8098/types/test/buckets/ test/datatypes/sargun -H "Content-Type: application/json" -H "X-Riak-Vclock: g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq" -d ' { "update": { "friends_set": { "remove": "Boss" } } }'
- 124. Updated Entries (during partition) { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQpq", "type": "map", "value": { "friends_set": [ "Boss" ], "posts_set": [] } } { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQtq", "type": "map", "value": { "friends_set": [], "posts_set": [] } }
- 125. Updatecurl -XPOST http://localhost:8098/types/test/buckets/ test/datatypes/sargun -H "Content-Type: application/json" -H "X-Riak-Vclock: g2wAAAABaAJtAAAACBjtDYuvG6A4YQtq" -d ' { "update": { "posts_set": { "add": "Party" } } }'
- 126. Updated Entries (After Healing) { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQ5q", "type": "map", "value": { "friends_set": [], "posts_set": [ "Party" ] } } { "context": "g2wAAAABaAJtAAAACBjtDYuvG6A4YQ5q", "type": "map", "value": { "friends_set": [], "posts_set": [ "Party" ] } }
- 128. Future Work: δ-CRDT
- 129. Ship only Deltas
- 131. SEC Enables
- 132. Distributed
- 134. Fault-Tolerant
- 135. Applications
- 136. Eventual Consistency (CAP) Without Consistency (ACID) Gives EC a Bad Name
- 137. Invariant Operation AP / CP Specify unique ID Any CP Generate unique ID Any AP > INCREMENT AP > DECREMENT CP < INCREMENT CP < DECREMENT AP Secondary Index Any AP Materialized View Any AP AUTO_INCREMEN T INSERT CP Linearizability CAS CP Operations Requiring Weak Consistency vs. Strong Consistency
- 138. BASE not ACID •Basically Available: There will be a response per request (failure, or success) •Soft State: Any two reads against the system may yield different data (when measured against time) •Eventually Consistent: The system will eventually become consistent when all failures have healed, and time goes to infinity
- 139. Brand New Technology Still being invented
- 140. Technology Timeline • 1996 - Log structured merge tree • 2000 - CAP Theorem • 2007 - Amazon Dynamo Paper • 2011 - INRIA CRDT Technical Report • 2014 - Riak DT map: a composable, convergent replicated dictionary
- 141. Further Reading • Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS • PNUTS: Yahoo!’s Hosted Data Serving Platform • F1: A Distributed SQL Database That Scales • Spanner: Google's Globally-Distributed Database • The Network is Reliable: An informal survey of real-world communications failures • A comprehensive study of Convergent and CommutativeReplicated Data Types • Riak DT Map: A Composable, Convergent Replicated Dictionary
- 142. Get in Touch • If you’re interested in cheating the speed of light • Come use our software • If you’re interested in solving today’s computer science problems • Come work for us • If you’d like to learn more about distributed systems at scale • Maybe you have a better idea
- 143. Sargun Dhillon @Sargun [email protected] The Case for Eventual Consistency