Chaos Engineering: Injecting Failure for Building Resilience in Systems
0 likes357 views
This document discusses chaos engineering and building resilient systems. It defines chaos engineering as experimenting in production to reveal weaknesses and build confidence in resilience. Some key principles of chaos engineering are discussed, such as having steady state periods between experiments and formulating hypotheses before experiments. Game days are mentioned where engineers take on roles like master of disaster to experiment with failures. The goal of chaos engineering is to design systems that can withstand failures through practices like circuit breaking and observability.
Chaos Engineering: Injecting Failure for Building Resilience in Systems
- 1. Chaos Engineering Injecting failure for building resilience in systems
- 2. Nice to meet you YURY NIÑO Software Engineer and Chaos Engineer Advocate. Loves building software applications, solving resilience issues and teaching. Passionate about reading, writing and cycling.
- 3. Agenda ● Resilience vs Reliability ● Why the world needs Resilience and Reliability? ● Chaos Engineering ● Principles of Chaos ● Chaos in Practice ● Game Days
- 4. How many of you Have encountered a crash of your systems on production?
- 5. A recognition for ... This talk is dedicated to the #SystemAdministrators well caffeinated, who get woken up in the middle of the night when “things go bump”. #EngineeringTeam #DigitalFactory @jnhernandz @
- 6. What is a Resilient System?
- 7. A resilient system can maintain an acceptable level of service in the face of failure. A resilient system can weather the storm such a large scale natural disaster or a controlled chaos engineering. Tammy Bütow Principal SRE at Gremlin
- 9. A distributed system on production needs to be resilient in order to be reliable and this is precisely a target that we Software Engineers, Systems Engineers, Site Reliability Engineers and Chaos Engineers always aim. Mine :)
- 10. Why the world needs Resilient Systems?
- 11. Because ... We are surrounded by distributed systems. When we read the news in our cellphones, send an email or buy our lunch ... We do not tolerate that they fail!
- 13. February 28th, 2017 will be remembered ● Simple Storage Service (S3) went down in US-EAST. ● Outage lasted about 4 hrs. ● > 100.000 websites across the world were impacted.
- 14. Me :(
- 15. The World is Chaotic! ● Distributed systems contains moving parts. ● Many things can go wrong. ○ Hard disks can fail. ○ The network can go down. ○ Customer traffic can overload.
- 16. How many of you know What is Chaos Engineering?
- 17. Chaos Engineering It is the discipline of experimenting in production on a distributed system in order to reveal their weakness and to build confidence in their resilience capability. https://principlesofchaos.org/
- 18. Chaos Engineering It is deliberately inducing stress or fault into software and/or hardware as a way of learning/verifying things about systems. https://www.gremlin.com
- 19. Chaos Engineering is about ● Simulating the failure of a datacenter. ● Injecting latency between services. ● Randomly causing exceptions. ● Changing time travel. ● Emulating I/O errors. http://principlesofchaos.org/
- 20. 2008 Chaos Engineering began at Netflix 2010 Chaos Monkey was launched 2018 A lot of resources for Chaos Engineering. 2014 Role of Chaos Engineer was created. History of Chaos Engineering Kolton Andrus
- 22. Principles of Chaos https://principlesofchaos.org/
- 23. 1. Steady Stead
- 27. 4. Run the Experiment Application Name Finer Observability DataDog Hypothesis Circuit Breaker works Environment My Home Results Duration 5 - 10 seconds Load 1 request Actions
- 28. 4. Run the Experiment Application Name Finer Observability DataDog Hypothesis Circuit Breaker works Facing latencies > 5 seconds between dashboard_api and smart_api to open the circuit. Environment My Home Results Duration 20 milliseconds Load 1 request Issue #4356 Configure the proper hystrix parameters according the results. Implement a fallback. Actions
- 29. Game Days
- 30. Game Day: Roles Master of Disaster First on-call Team https://www.pinterest.es/pin/824299538021645731/
- 31. Game Days can Transform our Teams Even though Game Days are not real! they make Engineers gain confidence.
- 32. Since we, Engineers are experiencing the failure as part of our job, we should start designing for failure. Me :) The best time to learn about fire is when you’re on fire. —Jen Hammond, New Relic engineering manager
- 33. How to begin ... https://chaosengineering.slack.com https://github.com/dastergon/awesome-chaos -engineering https://www.infoq.com/chaos-engineering @yurynino