Resiliency in Distributed Systems
Download as PPTX, PDF3 likes763 views
The document discusses the importance of resiliency in distributed systems, outlining key concepts, faults, and failures that can affect system performance. It provides a production readiness checklist and resilience testing methods such as the Simian Army for ensuring system robustness. Key strategies include monitoring, rate limiting, and using fault tolerance patterns to enhance system recovery capabilities.
Resiliency in Distributed Systems
- 2. Agenda • Basics of Resilience and Distributed Systems • Why we need Resilience ? • How do we achieve Resilience ? • Faults vs Failures • Measures in designing Resilient systems
- 3. Distributed Systems Networked Components communicate and coordinate their actions by passing messages
- 4. Same codebase different modules (monolith)
- 5. Different Codebase Same Database
- 6. Different Codebase Its own Databases
- 7. Different Codebase (with duplication or redundancy) Its own Databases
- 8. Resiliency The capacity to recover from difficulties
- 9. Why do we care about Resilience ?
- 10. How do we build a Resilient System ?
- 11. Fault vs Failure
- 12. Fault Incorrect Internal state in your system
- 13. Faults • Go routine leaks • File Descriptor Leaks (not closing response body, file, .... ) • Memory Leaks • Thread Pool Bottleneck • Network issues • Contract Issues • Throughput increase / attacks
- 14. Failure Inability of the system to do its intended Job
- 16. Fault Tolerance
- 19. Production Readiness Checklist ● Logging ● System Monitoring ● Application Monitoring ● Custom metrics (Business specific) ● Circuit Breakers ● Rate-limit critical flows ● Load Testing ● Performance Testing ● Runbooks ● Timeouts/Retries ● Code quality metrics ● Recovery Handler (In GO)
- 20. Problem Demo
- 22. Circuit Breakers
- 24. Code Demo
- 25. Fallbacks
- 26. Redundancies
- 27. Resiliency Testing - Simian Army
- 28. Clusters of our network duplicated
- 29. Simian Army • Chaos Monkey • Latency Monkey • Chaos Gorilla
- 31. Monitoring • Application • System • Customer Business Metrics • Request Monitoring
- 36. Feedbacks / Reach out Github: https://github.com/dineshkumar-cse Mail: [email protected] Github: https://github.com/rshetty mail: [email protected]