A New Transactional Model - Keith Bostic
2 likes332 views
This document discusses MongoDB's advancements in transactional capabilities, particularly focusing on the introduction of ACID transactions in version 4.0 and its journey through previous versions. It highlights the challenges faced in implementing these features and the ongoing efforts to improve consistency, replication, and performance in a distributed environment. Key technological components such as storage engines, logical timestamps, and multi-document transactions are detailed to illustrate the evolution of MongoDB's architecture.
A New Transactional Model - Keith Bostic
- 4. Relational vs document data models
- 5. Goals: performance and complexity
- 6. Goals: new applications, programmer models
- 7. Goals: checkboxes and vendor lock-in
- 9. But not our only audience
- 10. “ACID transactions are a key capability for business critical transactional systems, specifically around commerce processing. No other database has both the power of NoSQL and cross collection ACID transaction support. This combination will make it easy for developers to write mission critical applications leveraging the power of MongoDB.” – Dharmesh Panchmatia, Director of E-commerce, Cisco Systems
- 11. ACID
- 17. MongoDB World talk! Transactions and Durability: Putting the “D” in ACID – Sue LoVerso, Senior Engineer, MongoDB
- 18. The Path
- 19. Multi-year, all-hands company effort • The storage layer,
- 20. Multi-year, all-hands company effort • The storage layer, • Sharding architecture,
- 21. Multi-year, all-hands company effort • The storage layer, • Sharding architecture, • Introducing a global logical clock,
- 22. Multi-year, all-hands company effort • The storage layer, • Sharding architecture, • Introducing a global logical clock, • New replication consensus protocol,
- 23. Multi-year, all-hands company effort • The storage layer, • Sharding architecture, • Introducing a global logical clock, • New replication consensus protocol, • Metadata management, … and that’s just part of the list!
- 24. Multi-year, all-hands company effort • The storage layer, • Sharding architecture, • Introducing a global logical clock, • New replication consensus protocol, • Metadata management, … and that’s just part of the list! • Not forgetting the driver teams, documentation and education.
- 25. MongoDB 2.6: mmapv1 with ACID 2.6 P S S S
- 26. Detour: the storage engine The storage engine is part of the MongoDB database server. Think of it as “owning” the data. networking, sharding, replication, drivers analytics, middleware, query optimizer storage engine You are here.
- 27. Detour: the storage engine Single Node
- 28. MongoDB’s pluggable storage architecture • MMAPv1 • RocksDB (Facebook) • TokuMX (TokuTek) • WiredTiger
- 29. MongoDB 3.0: a new storage engine F Keys & Values E Keys & Values G Keys & Values Write-heavy workloads Document level locking Compaction Encryption Compression In-Memory 2.6 3.0 B-D Keys E-I Keys J-N Keys
- 30. MongoDB 3.0: transactional features 2.6 3.0
- 32. MongoDB 3.4 2.6 3.0 3.43.2
- 33. MongoDB 3.6: transactional features P S S S logical sessions global clock (Lamport) … leads to causal consistency 2.6 3.0 3.63.43.2
- 34. MongoDB 3.6: enhanced consistency P S S S consistent secondary reads retryable writes read concern majority 2.6 3.0 3.63.43.2
- 35. MongoDB University video Implementation of Cluster- Wide Causal Consistency in MongoDB – Misha Tyulenev, Engineer, MongoDB
- 36. MongoDB 4.0: multi-document transactions P S S S 2.6 3.0 3.63.43.2 4.0
- 37. MongoDB 4.0: prepared transactions Commit? Yes Yes Commit! Phase One Phase Two 2.6 3.0 3.63.43.2 4.0
- 38. MongoDB 4.X P S S S PS S 2.6 3.0 3.63.43.2 4.0 4.X
- 39. MongoDB World talk! What’s Next? The Path to Sharded Transactions – Andy Schwerin, VP, Engineering, MongoDB
- 40. Why is this work hard?
- 41. Reason #1: single-node vs. group ordering WiredTigerMongoDB
- 42. Replication example ... 37 38 39 ... 37 38 39 ... 37 38 39 ... 37 38 39 Primary Oplog Secondary reads
- 43. “People assume that time is a strict progression of cause to effect, but actually, from a non-linear, non- subjective point of view, it is more like a big ball of wibbily-wobbly timey-wimey … stuff.” – Doctor Who
- 44. Multi-threaded secondary oplog application ... 37 38 39 40 4241 Primary oplog order Secondary apply order 40 39 41 38 OpLog
- 45. Reason #2: increasing entanglement Getting the right answer requires communication. C PA
- 46. Timestamps
- 47. MongoDB’s timestamp ... 40 4138 39 42 ACID for a set of systems durability based on replication
- 48. WiredTiger’s transaction ID ... 40 4138 39 42 ACID for a single-node durability based on checkpoints and journaling
- 49. MongoDB timestamp overrides WiredTiger ID 64-bit value fast comparisons strictly increasing MSB hex string API
- 50. Timestamp stored with each MVCC update FRUIT mango apple banana TXN ID TIMESTAMP
- 51. Multi-threaded secondary oplog application ... 37 38 39 40 4241 Primary oplog order Secondary timestamp order ... 37 38 39 40 4241 Set the timestamp at update or commit OpLog
- 52. Overlapping reads with oplog application Secondary timestamp order ... 37 38 39 40 4241 Timestamp #1 Timestamp #2 3.6 4.0 Blocking Parallel
- 53. Queries can be “as of” a timestamp set at transaction begin largest LTE value
- 54. We need an “oldest” timestamp
- 55. Durability can be “as of” a timestamp
- 56. Replication rollback for a single server ... 37 38 39 40 4241 Primary Oplog ... 37 38 39 117 119118 Corrected (Secondary) Oplog Replacements
- 57. We need a “stable” timestamp
- 58. Sydney & New York