Standard Bank: Agile, DevOps, Engineering Transformation and the Role of AppDynamics - AppSphere16
2 likes943 views
The document discusses the challenges and solutions of implementing DevOps practices at Standard Bank, highlighting the need for improved collaboration and innovative engineering practices due to previous outages. It emphasizes the deployment of AppDynamics as a vital tool to enhance performance monitoring, reduce outage times, and improve customer experiences. Key takeaways include the importance of automation, proper tool deployment, and enabling development and operations teams to work effectively within a complex banking environment.
Standard Bank: Agile, DevOps, Engineering Transformation and the Role of AppDynamics - AppSphere16
- 1. Agile, DevOps, Engineering transformation and the role of AppDynamics. Andrew Kretschmer Head of Multi Channel Development CoE | Standard Bank
- 2. 1,221 Branches 8,800 ATMs Operations in 20 Countries Est. 154 years ago 14,9 million Retail Customers Over 44 000 Staff A Broad & Complex Footprint
- 3. Vision & Strategy Our vision is to be the leading financial services organisation in, for and across Africa, delivering exceptional client experiences and superior value.
- 4. “DevOps is Impossible in a Big Bank” OHI (Climate & Culture) – 3rd quartile !!
- 5. Create an environment where people can collaborate and innovate Engineering Culture
- 9. Digital Strategy Unified Customer Experience Mobile, Web, USSD, IVR, ATM, POS & Branch
- 10. Some History The Challenges 26 February 2015 Mobile Banking app was down for the whole day!
- 11. The Status Quo If you had an outage then… 1. Invited to the crisis room 2. No access to production 3. No access to the tools being used 4. Tools turned off because it impacts production 5. Yet you need to be able to tell people what is wrong 6. Changes being backed out 7. No clear root cause !!
- 12. The Fallout The Challenges • 27 February 2015 Meeting Group CIO to explain what happened • No RCA • Dev’s treated as Second Class Citizens when it comes to prod! • We need to do something, doing nothing and leaving things as they are were not an option…
- 13. The Journey To a Solution Discovery • We tried to use existing tools… • Looked at alternatives • Convince the right people
- 14. The Journey To a Solution 7 Key Requirements 1. Always running in production 2. Easy to deploy and use 3. “I do not want to call the vendor!” 4. DevOps enabler 5. For everybody to use 6. Code drill-down capability 7. Auto discovery
- 15. How can we remove impediments and deliver quality software to our customers?
- 16. • Rest of Africa Nigeria – Internet Banking • Performance issues • Login takes 30 Seconds • Slow responses in services layer • Developers spend 3 – 4 hours per day looking for/in logs • It wasn't me… • Login issue traced to core banking system • Week later patch received - Logins now take less than 3 sec • Developers build features instead of hunting logs and bugs
- 17. • South Africa USSD • Project that has been running for 2 years • Complex stack – so many layers… • Performance issues everywhere • AppDynamics deployed in a couple of days • Pinpoint where the performance issues and errors are • Finally in production 26 Oct 2016
- 18. You can prevent outages and reduce the time to fix production issues…
- 19. •MTBF – We can prevent bad customer experience South Africa – Mobile Banking • Alert was triggered that the error rate was higher than usual • Investigation found one node in the cluster not working • 25% of our customers were experiencing timeouts
- 20. •MTBF – We can prevent bad customer experience South Africa – Mobile Banking • Identified that it was only one node • Isolated node by stopping traffic • Customer Experience returned to normal
- 21. •MTTR – We can solve problems… quicker • April 2016 outage is reported on Internet Banking • 5 minutes into the outage we could pinpoint the problem • Issue in our Adaptive Risk system • Issue isolated and service was restored South Africa – Internet Banking • No need for a crisis meeting (How boring…) • Responsible team dealt the issue • Service was restored quickly
- 22. •MTTR – We can solve problems… quicker • Alert triggered due to slow responses for transaction logging • Not impacting customer experience yet… • Connection pools started to fill up • Customers experiencing slow performance South Africa – Mobile Banking • Pinpoint that the issue was on a message queue • We could pro-actively fix the issues and restart the broker • Response times improved and service returned to normal
- 23. Enable feature teams to do dev and ops
- 24. • Visibility on errors and exceptions • Metrics on how code is performing • I did not know the code was doing that… in production • Pin point where the issues are • Better quality code going into production (Engineering practices) • Alerts to when things go wrong • Have visibility without having to logon to the server • We can even monitor certificate expiry Dev and Ops in a feature team is possible
- 25. Dev and Ops in a feature team is possible
- 26. Learnings from Standard Bank TIPS
- 27. Automate Automate as much you can • Recipes to install Agents (App & Machine) • Recipes to install and configure the controller • Repeatable & easy to upgrade • Agents are always there
- 28. The right people AppDynamics in the right hands… • Allow devs and ops team to deploy the agents • Do not centralise the deployment of the agents • Devs and ops teams knows their applications best
- 29. Ready for the storm? Once people see AppDynamics they want it • Make sure you can manage the demand • In tough economic conditions funding is not easy • Convince the right people
- 30. DevOps & AppDynamics in a Complex Banking Environment. Questions? “They call it Africa, we call it home.”
- 31. Please give us your feedback—Session B20858 • Complete the online survey you'll receive via email later today or via text at: Text this number: 878787 Text this word: APPSPHERE • Every time you submit a session survey, your name will be entered in a random drawing. We're giving away Amazon Echos to 5 lucky winners! • Thank you for your input APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 31 Win!
- 32. Thank you