Failure Happens Interop Nyc
0 likes•986 views
This document discusses system failures and outlines reasons why failures are inevitable. It notes that a single point of failure can cause widespread outages and that recovery from failures can take hours or more to resolve issues across services. Specific examples of failures include a datacenter outage that lasted 45 minutes and affected services recovery times ranging from hours to over 12 hours. The document emphasizes that failures will happen and providers are also susceptible to failures, so anyone claiming failures will not occur is either foolish, lying, or both.
Failure Happens Interop Nyc
- 1. Failure Happens F***, the F***ing thing is F***king F***ed* *Official WebOps term from Artur Bergman Jesse Robbins [email protected]
- 2. This will be on the test: FAILURE HAPPENS!
- 5. 25% 75%
- 6. 25% 75% Paranoid
- 7. 25% Pyromaniac 75% Paranoid
- 9. Good Book!
- 10. “multiple and unexpected interactions of failures are inevitable” -‐Charles Perrow
- 11. Failure Happens
- 13. define: Nines (roughly) 99% 5256 min (3.5 days)
- 14. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours )
- 15. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min
- 16. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min
- 17. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min 99.9999% 30 Seconds
- 18. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min 99.9999% 30 Seconds 99.99999% 3 Seconds
- 24. ;''-1(<"=/-)"3.1>0?-'"@'-': !"#$$%"&'(')*)"+,-.,-/01,( +/.01210*"345467"89: #
- 26. #googlefail
- 27. YOU
- 28. Continuous Power... isn’t
- 29. 365 Main SF
- 30. 365 364.96 Main SF
- 33. Failure happens A single datacenter is the problem • Since they all fail at some point Recovery procedures after failure • Power was gone ~45 minutes • Most services took hours to come back • Some unnamed ones more than 12 hours
- 34. Truck 1, Rackspace 0
- 36. Geography is a Single Point of Failure
- 40. +2304,$5%67"#,-8$1 !"#$%#&'()(#*&+,&!"#$%&!'()* #%-#%*%,.&'(/,.#+%*&0+.1&-#%2+3&(/."4%*&(2&".&)%"*.&5678 !"#$%&''( +#,$-#$,%./-$0,1 )*
- 41. Taser weilding robbers C I Hosts' Chicago facility robbed twice! (the other two times were merely "break-ins where things were stolen")
- 46. Failure Happens. Anyone promising otherwise is either foolish or lying (or both).
- 47. Go Here! Jesse Robbins [email protected]