Distributed Systems In One Lesson
15 likes2,926 views
The document provides an overview of distributed systems concepts including distributed storage, transactions, computation, and coordination. It discusses various approaches to distributed storage including replication, sharding, consistent hashing, and distributed file systems. It also covers distributed transactions and the challenges of maintaining ACID properties in a distributed environment. The document introduces concepts of distributed computation frameworks like MapReduce and Spark and how they move computation towards data rather than moving large amounts of data. It discusses distributed coordination challenges and approaches like NTP for time synchronization, gossip protocols, and Paxos for reliable distributed logging.
![MapReduce
Mapper
Reducer
(k ,v) [(k ,v), (k ,v), (k ,v), …]
Shuffle(k , [v, v, v, …])
[(k ,v), (k ,v), (k ,v), …]](https://image.slidesharecdn.com/distributedsystems-150224073002-conversion-gate01/85/Distributed-Systems-In-One-Lesson-43-320.jpg)
![MapReduce
Once:[1]
upon:[1]
midnight:[1]
dreary:[1]while:[1]
pondered:[1]suddenly:[1] there:[1]came:[1]a:[1,1]
gently:[1]
rapping:[1,1]
tapping:[1,1]
at:[1]
my:[1]chamber:[1]
door:[1]
Reduce](https://image.slidesharecdn.com/distributedsystems-150224073002-conversion-gate01/85/Distributed-Systems-In-One-Lesson-49-320.jpg)
Distributed Systems In One Lesson
- 2. What are ?Distributed Systems Any system too large to fit on one computer.
- 3. What are • Storage • Transac6ons • Computa6on • Coordina6on ?Distributed Systems
- 5. Distributed Storage • Read replica6on • Sharding • Consistent hashing • Distributed filesystems
- 6. Distributed Storage • MongoDB • Cassandra • HDFS
- 7. When Life is Good
- 8. When Life is Good
- 9. When Life is Good Tim:Americano
- 10. When Life is Good Americano Tim?
- 11. When Life is Good
- 12. When Life is Busier
- 13. When Life is Busier Tim:Triple Skinny
- 14. When Life is Busier Tim:Triple SkinnyTim:Triple Skinny
- 15. When Life is Busier Tim:Triple Skinny Tim:Triple Skinny
- 16. When Life is Busier Tim?
- 17. ReplicaCon Problems • Complexity • Consistency
- 18. When Life is REALLY busy Aaron-‐ Faye
- 19. When Life is REALLY busy Aaron-‐ Faye Faye-‐ Nancy
- 20. When Life is REALLY busy Aaron-‐ Faye Faye-‐ Nancy Nancy-‐ Zed
- 21. Sharding Problems • More Complexity • Broken data model • Limited data access paEerns
- 29. CAP Theorem C PA
- 30. CAP Theorem • Consistency • Availability • Par66on Tolerance
- 31. CAP Theorem • Shared wri6ng project • Coffee shop closes • Synchronizing over the phone • BaEery dies • Status report!
- 32. CAP Theorem C PA
- 35. ACID TransacCons • Atomic • Consistent • Isolated • Durable • One barista • Mul6ple baristas
- 36. Ordering Coffee • Receive order • Process payment • Enqueue order • Make coffee • Deliver drink
- 37. Ordering Coffee • Receive order • Process payment • Enqueue order • Make coffee • Deliver drink {Different Actors
- 38. Ordering Coffee • Why split up order processing? • What can fail? • What are the consequences of failure? • How do we repair failure?
- 39. Ordering Coffee • How can we design a coffee shop with atomic transac6ons? • How does that limit the business? • Why give up atomicity?
- 41. Distributed ComputaCon • ScaEer/Gather • MapReduce • Hadoop • Spark
- 42. MapReduce • All computa6on in two func6ons: Map and Reduce • Keep data (mostly) where it is • Move compute to data
- 43. MapReduce Mapper Reducer (k ,v) [(k ,v), (k ,v), (k ,v), …] Shuffle(k , [v, v, v, …]) [(k ,v), (k ,v), (k ,v), …]
- 44. MapReduce Once upon a midnight dreary, while I pondered, weak and weary, Over many a quaint and curious volume of forgotten lore, While I nodded, nearly napping, suddenly there came a tapping, As of some one gently rapping, rapping at my chamber door. "'Tis some visitor," I muttered, "tapping at my chamber door- Only this, and nothing more." Ah, distinctly I remember it was in the bleak December, And each separate dying ember wrought its ghost upon the floor. Eagerly I wished the morrow;- vainly I had sought to borrow poems/raven.txt:
- 45. MapReduce It shall clasp a sainted maiden whom the angels name Lenore- Clasp a rare and radiant maiden whom the angels name Lenore." Quoth the Raven, "Nevermore." "Be that word our sign in parting, bird or fiend," I shrieked, upstarting- "Get thee back into the tempest and the Night's Plutonian shore! Leave no black plume as a token of that lie thy soul hath spoken! Leave my loneliness unbroken!- quit the bust above my door! Take thy beak from out my heart, and take thy form from off my door!" Quoth the Raven, "Nevermore." And the Raven, never flitting, still is sitting, still is sitting On the pallid bust of Pallas just above my chamber door; And his eyes have all the seeming of a demon's that is dreaming, And the lamp-light o'er him streaming throws his shadow on the floor; And my soul from out that shadow that lies floating on the floor Shall be lifted- nevermore! poems/raven.txt:
- 51. Real-‐World MapReduce • Hadoop • Cloudera, Hortonworks, MapR • Nobody write map, reduce func6ons • Hive (SQL-‐like interface) • Integra6on with BI front-‐ends
- 52. Hadoop • MapReduce API • Job Tracker, Task Tracker • Distributed Filesystem (HDFS) • Enormous ecosystem
- 53. Spark
- 54. Spark • ScaEer/gather paradigm (similar to MapReduce) • More general data model (RDDs) • More general programming model (transform/ac6on) • Storage agnos6c
- 61. Executor Task Task Cache Executor Task Task CacheRDD RDD RDD RDD RDD RDD Spark Worker Node
- 62. What’s an RDD?
- 63. What’s an RDD? This is
- 64. What’s an RDD? • Bigger than a computer • Read from an input source • Output of a pure func6on • Immutable • Typed • Ordered • Lazily evaluated • Par66oned • Collec6on of things
- 66. 5444 5676 0686 8389: List(xn-1, xn-2, xn-3) 4532 4569 7030 1191: List(xn-4, xn-5, xn-6) 5444 5607 6517 6027: List(xn-7, xn-8, xn-9) 4532 4577 0122 2189: Hash these Distributed, how?
- 68. Distributed CoordinaCon • NTP • Epidemic Protocols • Paxos
- 69. NTP • Distributed 6me synchroniza6on • A network of 6ered clocks • Synchroniza6on algorithm • O`en accurate to ±10ms
- 71. Paxos • Distributed log protocol • Assumes distributed, unreliable nodes • Performs leader elec6on
- 74. Review • Storage • Transac6ons • Computa6on • Coordina6on