Application resilience: challenges and good practice
- 1. STRICTLY CONFIDENTIAL Application Resilience: Challenges & Good Practice City Business Club - ISITC Europe | July 2019 Dr Aled Sage [email protected]
- 2. © Cloudsoft Corporation 2019 About Aled Sage VP Engineering @ Cloudsoft PhD and 20 years experience with distributed systems Expertise in Cloud, Devops and Automation Across different verticals, including financial services @aledsage [email protected] linkedin.com/in/aledsage github.com/aledsage© Cloudsoft Corporation 2019 2
- 3. © Cloudsoft Corporation 2019 1. What is resiliency? 2. Changing landscape 3. Case study 4. Problems & solutions Agenda © Cloudsoft Corporation 2019 3
- 4. © Cloudsoft Corporation 2019 • Business applications • Service level objectives (SLOs) Service level agreements (SLAs) • Aim is not 100% availability • “Availability” is not binary • Design for failure • Proactive and reactive What is Resiliency Google Site Reliability Engineers (SRE) “Striving for Imperfection: Using an error budget to move fast without compromising high reliability” CRM, payroll, analysis, trading systems, settlement systems, … Proactive: High availability (HA) Avoid problems Reactive: Know when something is wrong Know how to respond Repair Disaster recovery (DR)© Cloudsoft Corporation 2019 4
- 5. © Cloudsoft Corporation 2019 Business Impact © Cloudsoft Corporation 2019 5 Gartner Why Business Leaders Don’t Care About the Cost of Downtime 9 April 2019
- 6. © Cloudsoft Corporation 2019 • Private and public cloud • Kubernetes and Openshift (using containers) • Bare-metal and mainframes • Business wants agility • Security and reliability still vital • All while driving down costs Changing Landscape • Metro Bank, Monzo, Starling • Facebook? Amazon? © Cloudsoft Corporation 2019 6
- 7. © Cloudsoft Corporation 2019 Comparison with Disruptors Incumbents Disruptors Hybrid environment Everything is cloud-native Chaos engineering? “No way!” Kill servers in production Wall Street bank: decrease VM provisioning to 9 hours New container in seconds; New Virtual Machines in minute(s) 3-6 month releases Small changes several times a day Improve the basics: ● Improve resiliency and agility of existing apps ● Fit with old-world and the future © Cloudsoft Corporation 2019 100s of applications Many lines of business Small product set Legacy high-value apps No legacy apps 7
- 8. © Cloudsoft Corporation 2019 APPLICATION-CENTRIC APPROACH ✓ Empower and involve application architects and developers ✓ Codify best practices and use of the bank’s strategic tools ✓ Consistent way to automate resiliency Self-healing applications! Your application can automatically detect and fix problems Case-study: Tier-1 Investment Bank ACHIEVEMENTS UNLOCKED ✓ Application-level Resiliency at scale ✓ Meets business & regulatory needs ✓ Across multiple evolving platforms © Cloudsoft Corporation 2019 8
- 9. © Cloudsoft Corporation 2019 Challenges © Cloudsoft Corporation 2019 9 Wide variation in techniques and quality; bespoke processes. Involves many systems. Fiefdoms & politics. Resiliency is a cross-cutting concern. Manual procedures and tribal knowledge. Responses to degradation or failure not automated. Manual Bespoke Operations fragmented Rarely tested Sufficient to meet auditors’ requirements. But little practice or variation in scenarios; manual testing is expensive.
- 10. © Cloudsoft Corporation 2019 Problems and Solutions Problems Solutions Tribal knowledge, rarely tested Gamedays Processes, environments and applications are complicated and ever-changing Post mortems & continual improvement High availability and DR needs understanding of application, but infrastructure-focused Application-centric approach Bespoke / inconsistent approaches Empower the application architects; Codify and share best practices. © Cloudsoft Corporation 2019 10
- 11. © Cloudsoft Corporation 2019 Problems and Solutions
- 12. © Cloudsoft Corporation 2019 Problems and Solutions Problems Solutions Manual runbooks Automated recovery Involves many systems Encourage APIs and automation; “glue code” to codify their use; focus on common application architectures. Runbooks and CMDB out-of-date “Living model” of the application; Deployment and in-life management codified in this model. © Cloudsoft Corporation 2019 12
- 13. © Cloudsoft Corporation 2019 • Restart the web-servers (e.g. on an out-of-memory) • Repave servers or clusters (e.g. when a restart fails, or an update is required) • Auto-scale based on latency, load or time-of-day • Manage DR within a region and/or across regions Automated Recovery © Cloudsoft Corporation 2019 13
- 14. © Cloudsoft Corporation 2019 Self Managing Systems: Autonomics © Cloudsoft Corporation 2019 14 Self configuring • Automated configuration of components and systems follows high-level policies. Rest of system adjusts automatically and seamlessly Self healing • System automatically detects, diagnoses, and repairs software and hardware problems Self optimizing • Components and systems continually seek opportunities to improve their own performance and efficiency Self protecting • System automatically defends against malicious attacks or cascading failures. It uses early warning to anticipate and prevent system wide failures
- 15. © Cloudsoft Corporation 2019 Cloudsoft AMP © Cloudsoft Corporation 2019 15 AMP streamlines development, operations and governance for any application on any cloud Rapidly Scalable Autonomic policy-based management Cloud Enabling Liberate applications from infrastructure Powerful Tools Consistent in-life management tooling and features Improve Visibility Unified management view of applications
- 16. © Cloudsoft Corporation 2019 Suggested Next Steps © Cloudsoft Corporation 2019 This week ❏ Get the Gartner report: “Why Business Leaders Don’t Care About the Cost of Downtime” free of charge ❏ Talk to person(s) in charge of application resiliency ❏ Example post mortem from the last major incident; how was this knowledge shared and used? ❏ How are application authors involved in planning for and implementing resiliency? ❏ Is there consistent monitoring and recovery strategies (focusing 80:20 rule of existing apps)? ❏ How often is it tested? Are there gamedays? Next 90 days ❏ Identify example app(s) ❏ Get a review from an expert of an example workload (“well-architected review”) ❏ Organise gamedays ❏ Introduce post mortems Next 6 months ❏ Consider automation software at the application level for ‘low-hanging fruit’ ❏ Involve and empower your application authors 16 get report for free cloudsoft.io/report
- 17. 17 e: [email protected] w: cloudsoft.io Thank You! Any Questions?