Erlang - Because s**t Happens by Mahesh Paolini-Subramanya
Download as PPTX, PDF•1 like•2,251 views
The document discusses the importance of fault tolerance in systems, particularly through the lens of Erlang's capabilities. It emphasizes key principles such as concurrency, fault detection, error encapsulation, code upgrades, and stable storage to improve reliability in software development. Additionally, it highlights the challenges software projects face, including high failure rates and the significance of effective monitoring and management strategies.
Erlang - Because s**t Happens by Mahesh Paolini-Subramanya
- 1. { Erlang : Because S**t happens Mahesh Paolini-Subramanya (@dieswaytoofast) V.P. Ubiquiti Networks
- 2. AGILITY
- 3. My Vacation
- 4. (Actually, the day before)
- 7. The Horror! The Horror!
- 8. Why are my calls failing?
- 9. You better call me back!
- 10. I’m still p***ed off!
- 11. And you’re stupid Apps don’t work!
- 12. The Horror! The Horror!
- 14. 1000 year floods
- 16. Fault Tolerance
- 17. Concurrency The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 18. Concurrency Fault detection The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 19. Concurrency Fault detection Fault identification The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 20. Concurrency Fault detection Fault Identification Error Encapsulation The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 21. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 22. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 23. erlang…
- 24. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 25. Concurrency Oriented Concurrency Hell My Blue Heaven My Blue Heaven
- 26. Concurrency Oriented Concurrency Hell My Blue Heaven Deep Problems My Blue Heaven Deep Problems
- 27. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 28. Fault Detection
- 29. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 30. Stack Traces?
- 31. Immutable Variables X = 1.
- 32. Immutable Variables X = 1. X = 2. Huh?
- 33. Immutable Variables X = 1. X = 2. X = X + 1. Huh?
- 35. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 36. Let It Crash
- 37. BEAM!
- 38. Faster to create JVM is not necessarily your friend!
- 39. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 40. Code Upgrade
- 41. Live! Hot SwappingCode Upgrade
- 42. Concurrency Fault detection Fault Identification Error Encapsulation Code upgrade Stable Storage The Big Six From http://www.erlang.org/download/armstrong_thesis_2003.pdf
- 43. The Intangibles
- 44. 4x – 10x less code
- 45. Code Size
- 46. Faster to create 4x – 10x less code
- 47. Faster to create Easier to reason about 4x – 10x less code
- 48. Faster to create Easier to reason about Fewer bugs 4x – 10x less code
- 49. Faster to create Easier to reason about Fewer bugs Speedy refactoring 4x – 10x less code
- 50. The Shell is our friend
- 51. Live Debugging
- 52. Predictability
- 53. Performance
- 55. Fault Tolerance - Systems
- 57. Romney 2012
- 58. Fault Tolerance - Systems
- 59. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems
- 60. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems
- 61. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems LOOSECOUPLING
- 62. Loose Coupling?
- 63. Breeds Trust Loose Coupling
- 64. Loose Coupling
- 65. Breeds Trust Devote more brainpower to specific areas Loose Coupling
- 66. Loose Coupling
- 67. Breeds Trust Devote more brainpower to specific areas No. of bugs/line is constant Loose Coupling
- 68. Performance
- 69. 60 - 90% of all SW projects fail 10 – 25% of all SW projects get abandoned Fault Tolerance
- 71. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems M ONITORING
- 72. Monitoring?
- 74. Dashboards Out of band systems Monitoring?
- 75. Supervision
- 76. Dashboards Out of band systems Polyglot safety Monitoring?
- 77. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems
- 78. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems POLYGLOT PERSISTENCE
- 79. Concurrency Error encapsulation Fault detection Fault identification Code upgrade Stable Storage The Big Six - Systems EVERYW HERE!!!
- 80. No battle plan survives contact with the enemy
- 81. Not just about Systems Fault Tolerance
- 82. Fault Tolerance
- 83. People Vendors Fault Tolerance
- 84. People Vendors Fraud Fault Tolerance
- 85. The BusinessBeware the Black Swan
- 86. Is It Safe?
- 87. erlang…
- 90. Queues
- 91. Queues
- 92. Queues
- 93. Queues
- 94. Can you recover quickly? Buffer-bloat doesn’t matter, right? Once up, can you deal with the backlog? Back-pressure isn’t an issue, right? Queues
- 95. Can you recover quickly? Buffer-bloat doesn’t matter, right? Once up, can you deal with the backlog? Back-pressure isn’t an issue, right? Queues NOPE
- 96. Programmable
- 97. Behavioral
- 98. Self Managed
- 100. Tail Drop
- 102. RED
- 103. RED
- 104. Newark Airport
- 105. FRED
- 106. RED-PD
- 107. WRED
- 108. RED – Many many more SRED RRED ARED (and Blue!) CHOKe
- 109. Special Mention RED in a different Light
- 110. SERIOUSLY! RED in a different Light CoDel and fq_codel
Editor's Notes
- #2: An overall approach to Preparedness
- #3: This is a story about unexpectedness.The only constant is change
- #4: Our story starts on a happy Saturday in february
- #5: Its still Friday
- #6: Just part of one cluster failed, but a threshold had been passed
- #7: No worries, we’ll just bounce that one cluster, it’ll all be good
- #8: Total System Meltdown
- #9: All the calls keep retrying, causing memory utilization to go through the roof
- #10: Voicemail conversion was going on independent of everything else, causing CPU utilization to spike
- #11: Eventually, the cache timed out, and tried to reload stuff from the disk.
- #12: And then everyone tries the Apps, and the Twitters and the facebooks and the everythings.
- #13: Total System Meltdown
- #14: What about testing? Didn’t you check loads? Specs? Capabilities?
- #15: There is only so much planning you can do. At some point, the 1000 year flood hits
- #16: The point being, Shit will happen.The question is, when Shit happens, can you clean up?
- #17: There is a formal definition of Fault Tolerance
- #18: The Six Essential Characteristics of a Fault Tolerant System
- #19: The Six Essential Characteristics of a Fault Tolerant System
- #20: The Six Essential Characteristics of a Fault Tolerant System
- #21: The Six Essential Characteristics of a Fault Tolerant System
- #22: The Six Essential Characteristics of a Fault Tolerant System
- #23: The Six Essential Characteristics of a Fault Tolerant System
- #24: The Six Essential Characteristics of a Fault Tolerant System
- #25: The Six Essential Characteristics of a Fault Tolerant System
- #26: ‘Distributed’ problems mean you spend a huge chunk of your time dealing with theadminstrivia of distribution.With erlang you get that for free!Processes, Messages, Immutability, “Writing Concurrent Programs in Java”
- #27: Ok, not really true. You still have to deal with ‘deep problems’ (hard core parallelization issues, etc.)But you’d have to deal with that anyhow!
- #28: The Six Essential Characteristics of a Fault Tolerant System
- #29: The Six Essential Characteristics of a Fault Tolerant System
- #30: The Six Essential Characteristics of a Fault Tolerant System
- #31: Testing is infinitely easier. Trivial to simulate (its all messages!)Thank you immutability!
- #34: Garbage Collection, Referential Integrity, Testing!!!
- #35: The Six Essential Characteristics of a Fault Tolerant System
- #36: The Six Essential Characteristics of a Fault Tolerant System
- #37: Let it Crash
- #38: BEAM --> insanely reliable. will last till the heat death of the universe if you leave it alone
- #39: JVM is not necessarily your friend.Running on the JVM is not necessarily good - do you trust all the other java code? i don't. trust _me_, i've been there
- #40: The Six Essential Characteristics of a Fault Tolerant System
- #41: Let it Crash
- #43: Mnesia, ETS, gen_servers, etc.
- #44: Testing is infinitely easier. Trivial to simulate (its all messages!)Thank you immutability!
- #45: Testing is infinitely easier. Trivial to simulate (its all messages!)Thank you immutability!
- #46: The bigger they are, the harder they fall
- #51: Just connect to a remote node and trace to figure out what is going on
- #52: Why wait? Just log on to a node
- #53: Soft real-time. Brief discussion of instrumentation and ‘reductions’
- #54: i/o (and message passing. basically the same thing) is _wicked_ fast. Not just IPC, but network, web (cowboy) websockets, etc.
- #55: The Buddha nature of erlang
- #56: This is pretty much what we’re talking about right?Systems – Development/Production and Internal/External
- #57: Its not just us
- #58: Its not just us
- #59: Lets talk about systems
- #60: The Six Essential Characteristics of a Fault Tolerant System
- #61: Loose Coupling, of course, gives us all these benefits
- #62: Loose Coupling, of course, gives us all these benefitsLoosely couple systems can operate concurrently. Well D-UHErrors can be contained/constrained
- #63: Keep components/modules/systems ‘loosely coupled’Connect via specs/apis/busesDo this by default, even when you don’t need to!
- #65: Builds trust Trust in the stupidity of people, trust that things will fail, trust that you will be affected
- #67: The amount of brainpower we have is limited.Reduce complexity by being able to focus on specific / limited areas
- #68: There are many studies (some not so controversial) that show the number of bugs/line is constantFocus on smaller areas gives you fewer things to tackle
- #69: Isn’t Performance an issue w/ Loose Coupling?
- #70: remember the bit about failure? well, why optimize if you're going to fail anyhow? yeah yeah, you might fail because you don't perform, but that is rarely the problem
- #71: yes, that mine craft plugin you built might gt a million signupsit won'tseriously – it doesn't register statistically
- #74: DashboardsOtherwise, how do you know whats going on?
- #75: Out of band access Don’t rely on the system to always tell you whats happening
- #76: Corresponds to how we think, and helps deal with edge-cases much *much* better!
- #77: Be PolyglotEverything fails – even erlang. (noooo)
- #79: Why Polyglot?Because you want to limit your failure modes (increasing diversity can actually reduce systemic risk)
- #80: Macro Effects Matter! Systems span divisionsFinance, Customer Support, Sales, HR, etc.
- #81: Helmuth vonMoltke
- #83: People fall ill
- #84: Vendors Fail(Amazon)
- #85: Fraud: You wonder why your CFO is in Brazil…
- #86: Tail Risk (Things that can never happen)This deserves its own section(financial crisis)
- #87: Ask yourself this. Over and over again…
- #88: The Six Essential Characteristics of a Fault Tolerant System
- #89: Yeah, yeah. Understandable lies. But the bottlenecks are pretty far down the road (and much further than you would have gotten before!)
- #90: Tail RiskThis deserves its own section(financial crisis)
- #91: How fast are you?How quickly can you come back up? Can you store enough state to survive?
- #92: Is BufferBloat a problem?
- #93: Once you are up, can you draw down the queue fast enough?Or at all, for that matter?
- #94: Is backpressure going to be a problem?
- #95: If the answer is “Yes”, then the talk is over, because it just works.
- #96: What if the answer is “No”? (Now we have a story)
- #97: ProgrammableIf you’re lucky, you’re infrastructure will automagically support ramping
- #98: Fake it. People respond subconsciously to these, and actually waitYou can even get away with dropping the request(This assumes that you can recover in time)
- #99: This happens inside the airport too!Passengers self-select the best gates to enter(intelligent routing)
- #100: The question is, what do you do when you can’t come up in time? 3 gallon bucket, 5 gallons of water…
- #101: Just start dropping when queue fills upThis is pretty bad – global synchronization becomes a problemPlanes don’t take off till they get clearance from the other end
- #102: Slow Start, AQM, RED, CoDEL, …Why don’t we learn from networks?They certainly don’t learn from us, why do we ignore them?
- #103: RED / SRED(RED in a different light – toilet bowl)
- #104: RED / SRED(RED in a different light – toilet bowl)
- #105: The 3rd priority airport always gets the shaft
- #106: F(low) REDRED on a per-flow basis (the entire route map)Kinda the default. Discard second request)
- #107: RED – P(referential) D(rop)Does RED only for High BW flows (high traffic routes)(Throttle spammy clients. Or features.)
- #108: W(eighted) REDDifferent discard probabilities for different flows (translatlantic routes)(Major clients vs small ones0
- #109: S(tabilized) RED – estimate flows and probabilitiesR(obust) RED – Protect against low-rate DoS (with filters) (even unintentional DoS)A(daptive) RED – Modify prob based on queue CHO(ose and) K(eep) or CHO(ose and) K(ill) - open for < min; drop tail for > maxelse, compare packet to random packet. if same flow, drop it w/ prob.
- #110: Fixed two bugs in REDMade it feedback based (self-tuning)Toilet diagram caused problems
- #111: Van Jacobson strikes backUse Queue length as metric (bursts can fill up queue)Drop probabilistically
- #112: Yeah, yeah. Understandable lies. But the bottlenecks are pretty far down the road (and much further than you would have gotten before!)