QCon NYC: Distributed systems in practice, in theory
5 likes1,495 views
This document discusses the author's experience reading computer science research papers as an industry practitioner in distributed systems. It covers several topics from papers, including staged event-driven architectures, the use of leases for distributed locking and heartbeats, and designing systems to allow some inaccuracy for improved performance and resilience. The author advocates using insights from academic research to inform design decisions and mitigate technical risks when building large-scale, robust systems in industry.
![[Trade off] Inaccuracy for Performance](https://image.slidesharecdn.com/nyc-distributedsystemsinpracticeintheory-160614214934/85/QCon-NYC-Distributed-systems-in-practice-in-theory-76-320.jpg)
![[Trade off] Inaccuracy for Resilience](https://image.slidesharecdn.com/nyc-distributedsystemsinpracticeintheory-160614214934/85/QCon-NYC-Distributed-systems-in-practice-in-theory-80-320.jpg)
![[In production]
Inaccuracy for Performance & Resilience](https://image.slidesharecdn.com/nyc-distributedsystemsinpracticeintheory-160614214934/85/QCon-NYC-Distributed-systems-in-practice-in-theory-97-320.jpg)
![[Designing with]
Inaccuracy for Performance & Resilience](https://image.slidesharecdn.com/nyc-distributedsystemsinpracticeintheory-160614214934/85/QCon-NYC-Distributed-systems-in-practice-in-theory-101-320.jpg)
![[Designing with]
Inaccuracy for Performance & Resilience
simplified implementation
focus on observability
applicable to
some problem
domains](https://image.slidesharecdn.com/nyc-distributedsystemsinpracticeintheory-160614214934/85/QCon-NYC-Distributed-systems-in-practice-in-theory-102-320.jpg)
![[Designing with]
Inaccuracy for Performance & Resilience
fuzz testing
generative testing
simplified implementation
fault injection testing
focus on observability
applicable to
some problem
domains](https://image.slidesharecdn.com/nyc-distributedsystemsinpracticeintheory-160614214934/85/QCon-NYC-Distributed-systems-in-practice-in-theory-103-320.jpg)
QCon NYC: Distributed systems in practice, in theory
- 1. Aysylu Greenberg June 14, 2016 Distributed Systems in Practice, in Theory
- 2. How I got into reading papers as a practitioner in industry
- 7. Operating systems research Concurrency Concurrency primitives: mutex & semaphore
- 8. Operating systems research Concurrency Concurrency primitives: mutex & semaphore Processes execute at different speeds
- 9. Time in distributed systems https://www.flickr.com/photos/national_archives_of_norway/6263353228
- 10. Time in distributed systems
- 11. Time in distributed systems Pipelining
- 12. 1980
- 13. 1980
- 14. Internet 1980
- 20. Reconsider large systems Shared infrastructure ...
- 21. CS Research is Timeless Inform decisions Mitigate technical risk
- 23. Papers We Love NYC
- 24. Papers We Love SF
- 26. Today ● Staged Event-Driven Architecture
- 27. Today ● Staged Event-Driven Architecture ● Leases
- 28. Today ● Staged Event-Driven Architecture ● Leases ● Inaccurate Computations
- 30. Hardware to Data Pipelines
- 31. Hardware to Data Pipelines https://en.wikipedia.org/wiki/Graphics_pipeline
- 33. Staged Event Driven Architecture
- 34. Staged Event Driven Architecture + -
- 35. Single-machine pipeline generalizes to distributed pipelines Staged Event Driven Architecture
- 45. Leases as Heart Beat in Distributed Systems 1989
- 48. Leases ● Distributed locking ● Lease term tradeoffs ○ short
- 49. Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long
- 50. Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election TTL (in etcd)
- 51. Leases ● Distributed locking ● Lease term tradeoffs ○ short vs long ● Use of leases in modern applications ○ Leader election TTL (in etcd) ○ Liveness detection
- 53. Leases in Build System: Success Scenario
- 56. Build my project Build System OK Waiting for the results
- 57. Build my project Build System OK Waiting for the results Build is in progress
- 58. Build my project Build System OK Waiting for the results Build is in progress Waiting for the results
- 59. Build my project Build System OK Waiting for the results Build is in progress Waiting for the results Build is finished
- 60. Leases in Build System: Failure Scenario
- 61. Leases in Build System
- 62. Leases in Build System
- 63. Leases in Build System
- 64. Leases in Build System
- 65. Leases in Build System
- 66. Leases in Build System
- 67. Using etcd leases for heartbeat $ curl http://server.com/v2/keys/foo -XPUT -d value=bar -d ttl=300
- 68. { "action": "set", "node": { "createdIndex": 2, "expiration":"2016-06-14T16:15:00", "key": "/foo", "modifiedIndex": 2, "ttl": 300, "value": "bar" } }
- 69. Using etcd leases for heartbeat $ curl http://server.com/v2/keys/foo -XPUT -d value=bar -d ttl=300 … 3 minutes later...
- 70. Using etcd leases for heartbeat $ curl http://server.com/v2/keys/foo -XPUT -d value=bar -d ttl=300 $ curl http://server.com/v2/keys/foo?prevValue=bar -XPUT -d ttl=300 -d refresh=true -d prevExist=true
- 71. { "action": "update", "node": { "createdIndex": 2, "expiration":"2016-06-14T16:18:00", "key": "/foo", "modifiedIndex": 3, "ttl": 300, "value": "bar" } "prevNode": {...} }
- 72. { "action": "update", "node": { "createdIndex": 2, "expiration":"2016-06-14T16:18:00", "key": "/foo", "modifiedIndex": 3, "ttl": 300, "value": "bar" } "prevNode": {...} } "prevNode": { "createdIndex": 2, "expiration":"2016-06-14T16:15:00", "key": "/foo", "modifiedIndex": 2, "ttl": 120, "value": "bar" }
- 73. Leases for heartbeat: How long should the lease term be?
- 74. Inaccurate Computations & Serving Search Results
- 75. From Accurate to "Good Enough"
- 76. [Trade off] Inaccuracy for Performance
- 80. [Trade off] Inaccuracy for Resilience
- 84. Inaccuracy for Resilience 1. Task decomposition
- 86. Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness
- 88. Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness 3. Criticality Testing
- 94. Inaccuracy for Resilience 1. Task decomposition 2. Baseline for correctness 3. Criticality Testing 4. Distortion and timing models
- 95. Distortion Model
- 96. Timing Model
- 97. [In production] Inaccuracy for Performance & Resilience
- 98. Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010
- 101. [Designing with] Inaccuracy for Performance & Resilience
- 102. [Designing with] Inaccuracy for Performance & Resilience simplified implementation focus on observability applicable to some problem domains
- 103. [Designing with] Inaccuracy for Performance & Resilience fuzz testing generative testing simplified implementation fault injection testing focus on observability applicable to some problem domains
- 104. References ● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All" ● M. Rinard "Probabilistic Accuracy Bounds for Fault- Tolerant Computations that Discard Tasks" ● F. Corbato, M. Daggett, R. Daley "An Experimental Time- Sharing System" ● E. Dijkstra "Cooperating Sequential Processes" ● L. Lamport "Time, Clocks, and the Ordering of Events in a Distributed System" ● http://blinkdb.org/
- 105. References ● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy Method to Support Highly-Available Distributed Systems" ● L. Lamport "The Part-Time Parliament" ● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well- Conditioned, Scalable Internet Services" ● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency" ● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data"
- 106. Gratitude Ines Sombra David Greenberg Karan Parikh Matt Welsh Erran Berger
- 107. Robust & scalable pipelines
- 108. Robust & scalable pipelines Leases for sharing & heartbeat
- 109. Robust & scalable pipelines Leases for sharing & heartbeat Inaccuracy for resilience & performance
- 110. Robust & scalable pipelines Leases for sharing & heartbeat Inaccuracy for resilience & performance CS research is timeless: use it to mitigate risk
- 111. Aysylu Greenberg June 14, 2016 Distributed Systems in Practice, in Theory @aysylu22