Implementing a Reliable, Auto-Healing Scalable Platform at VMware
- 1. THE CLOUD CONNECTIVITY COMPANY GopalaKrishna Sandur Kustigi Rajshekhar Deodurg Date: 16th July 2020 THE CLOUD CONNECTIVITY COMPANY 1 Implementing a Reliable, Auto-Healing Scalable Platform at VMware
- 2. THE CLOUD CONNECTIVITY COMPANY 2 Robust API Gateway is Critical for IT Enterprise Integration VMware IT identified implementing an API gateway in a reliable, auto-healing scalable platform as a priority We chose multiple Kubernetes platforms to enable auto-failover and auto-scalable capabilities in a dual data center Centralized log management and analytics provide deep operational visibility and enable faster troubleshooting Customized monitoring scripts execute auto-failover between the data centers Combined use of several open-source and VMware tools equips VMware IT to perform chaos engineering
- 3. THE CLOUD CONNECTIVITY COMPANY 3 It Starts with Architecture CDN Global Traffic manager Data center-1 Active Kubernetes Cluster Cassandra Pods Kong Pods Application Pods Load Balancer Proxy Server Proxy Server Load Balancer Service Kubernetes Cluster Cassandra Pods Kong Pods Application Pods Load Balancer Proxy Server Proxy Server Load Balancer Service Data center-2 Standby Firewall Firewall Proxy Server Proxy Server Proxy Server
- 4. THE CLOUD CONNECTIVITY COMPANY 4 Architecture Continued… Active/Standby dual data center setup Auto-failover upon failure detection in primary data center Auto-failover implemented with custom monitoring script at Layer-1 Integrated with monitoring and log aggregation tools
- 5. THE CLOUD CONNECTIVITY COMPANY 5 Metrics and Dashboard Monitoring Enables Early Detection and Notification Across Application Stack VMware Tanzu Observability by Wavefront tool monitors Kubernetes cluster and pods Tanzu Observability collects data from multiple services and sources across the entire application stack, providing real-time resource utilization of the Kubernetes platform and at the KONG application Pod level Tanzu Observability also provides alert feature setup for different failure scenarios such as pod restarts, resource limit exceeded and more
- 6. THE CLOUD CONNECTIVITY COMPANY 6 Tanzu Observability Dashboard
- 7. THE CLOUD CONNECTIVITY COMPANY 7 Centralized Log Management and Analytics We integrated KONG API gateway log and Kubernetes platform logs with vRealize Log lnsight to achieve centralized log management and analytics during a production issue vRealize Log lnsight helps us manage logs with its actionable dashboards Log pattern monitoring was enabled with vRealize Log Insight to provide deep operational visibility and faster troubleshooting across different layers
- 8. THE CLOUD CONNECTIVITY COMPANY vRealize™ Log Insight Dashboard
- 9. THE CLOUD CONNECTIVITY COMPANY Chaos Engineering Enabled Make-Break-Remake Chaos is the art of breaking things purposefully but in a controlled manner We used a combination of open-source, in-house tools and VMware products to create two types of chaos Self-induced Network Failures CPU Load, Memory Load, Kill Process Fault, Disk Space Fault etc.. Network delay on container network overlays and the virtual ethernet port, packet drop Self-Induced Infrastructure failures
- 10. THE CLOUD CONNECTIVITY COMPANY Chaos Dashboard
- 11. THE CLOUD CONNECTIVITY COMPANY 11 Synthetic API Monitoring Results To measure the availability of the Kong System we use a custom monitoring tool which continuously invokes Tier 1 API’s at regular intervals. With these in-house tools for monitoring, analytics, alerting in place and multi-data center architecture, we have achieved high availability.
- 12. THE CLOUD CONNECTIVITY COMPANY 12 Q&A