SRE Conference 2022 - How to Build a Healthy On-Call Culture
2 likes•1,045 views
The document outlines best practices for building a healthy on-call culture, emphasizing the importance of monitoring services to maintain operational health and incident management. It covers various monitoring types, the responsibilities of on-call engineers, and the incident handling process, including root cause analysis. Additionally, it introduces tools and systems for effective communication and notification during incidents to ensure a blameless culture.
![How to Prepare RCA?
● Incident Timeline
○ 2022/04/29 16:00 Receive Alert from Prometheus
○ 2022/04/29 16:03 Start Incident Response
○ …
● FIndings/Root Cause
○ The HTTP Status Code Return 400
○ Finding TLS Certificate Expired
● Follow-up/Corrective Action
○ Monitoring All TLS Certificate [Ticket: ID-1234][Owner:
smalltown][Status: On-Going]](https://image.slidesharecdn.com/smalltownsreconferencehowtobuildahealthyon-callculture-220429083835/85/SRE-Conference-2022-How-to-Build-a-Healthy-On-Call-Culture-33-320.jpg)
SRE Conference 2022 - How to Build a Healthy On-Call Culture
- 1. How to Build a Healthy On-Call Culture
- 2. Hello! I am smalltown MaiCoin Group Lead SRE Taipei HashiCorp UG Organizer AWS UG Taiwan Staff
- 3. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 4. Join MaiCoin First Week (End of 2017) ~$ ssh [email protected] ~$ df -ah ~$ rm /var/log/nginx/* ~$ service nginx restart
- 5. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 6. Why Need Monitoring? ● Service W/O Monitoring Just Like a Blind ○ Not Know Service Usual Health Status ○ No Notice When Something Bad Will/Has Happen(ed) ○ Not Investigate Issue After Incident ● No Monitoring -> No Measure -> No Quality
- 7. Common Monitoring Types ● End-To-End ○ Functionality ○ Performance ● Error Tracking ● Networking ○ Latency ○ Connectivity ● Infrastructure ○ CPU ○ Memory ○ Disk ○ …
- 8. External Monitoring ● DNS ● Certificate ● Geolocation (Global Ping) ○ Routing ○ Submarine Cable ● Content Delivery Network ● Web Application Firewall ● …
- 9. Monitoring As Code - Internal YAML Git Repository ArgoCD Prometheus
- 10. Monitoring As Code - External Terraform HCL Git Repository …
- 11. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 14. Organization Architecture - Traditional DC Development Operations
- 15. Organization Architecture - DevOps SRE SE EM SRE = Site Reliability Engineer EM = Engineering Manager SE = Software Engineer
- 16. Our Organization Architecture SRE SE EM SRE SE EM EM
- 17. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 18. On Call Engineer Responsibility ● Routine Operation Job ● Handle Incident ● Runbook Refine/Writing ● Weekly On Call Report
- 19. On Call Model - Two Tracks SE SE EM SRE SRE EM Escalation Path Infrastructure Application
- 20. Notification Problem Voice Call Notification User Experience Communication Between Teams Arrange On Call Engineer Schedule
- 22. Notification System - Rotation ● On Call Model ● Rotation Frequency ● Fail to Answer a Page
- 23. Notification System - Communication ● Slack User Group Break the Communication Cap ● Check Document -> Check Calendar -> Tag Group Name
- 24. Notification System - Effectiveness ● Contacts Phone Book -> Slack Command W/ User Group/Name ● Short Phone Call Notice -> Phone Call W/ Validation ● Notification Visibility -> Detail Status in Slack
- 25. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 26. Aware Incident Before Customer
- 27. Incident Handle Terminology Role ● Incident Commander ● Tech Lead ● Communication Lead ● Engineering Manager Incident Level ● S3 ● S2 ● S1 ● S0
- 28. Incident Handle Process Initiate Incident Define Incident Assign Roles Start Maintenance Initiate Notify Investigate & Fixing Verification Maintenance End Recovery Notify Restrospective Enhancements
- 29. Incident Visibility - Internal
- 30. Incident Visibility - External ● Service Health Page ● Mobile App Push Notification ● Facebook, Telegram, Twitter… Customer/Vendor Communication Channel
- 31. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 32. What is Root Cause Analysis (RCA)? ● A Systematic Process for Identifying “Root Causes” of Problems or Events and an Approach for Responding to Them ○ What Happened ○ How it Happened ○ Why it Happened…so ○ Actions for Preventing Reoccurrence are Developed
- 33. How to Prepare RCA? ● Incident Timeline ○ 2022/04/29 16:00 Receive Alert from Prometheus ○ 2022/04/29 16:03 Start Incident Response ○ … ● FIndings/Root Cause ○ The HTTP Status Code Return 400 ○ Finding TLS Certificate Expired ● Follow-up/Corrective Action ○ Monitoring All TLS Certificate [Ticket: ID-1234][Owner: smalltown][Status: On-Going]
- 35. Organization Type On Call Incident Response Root Cause Analysis Monitoring
- 36. THANKS! ANY QUESTIONS? You can find me at my office: ● MicroService Engineer ● Backend Engineer ● Frontend Engineer ● ...