Livnat Peer & Arthur Berezin, Red Hat - Neutron High Availability - OpenStack Israel 2015
1 likeâ˘1,056 views
This document discusses techniques for achieving high availability in OpenStack Neutron. It describes enabling technologies like Pacemaker and HAProxy that provide redundancy. It also outlines several built-in Neutron mechanisms for high availability, including keeping DHCP and metadata agents highly available, and using Virtual Router Redundancy Protocol (VRRP) for L3 agent HA. Distributed Virtual Routing (DVR) is also discussed as a way to improve availability by distributing routing functionality across compute nodes.
Livnat Peer & Arthur Berezin, Red Hat - Neutron High Availability - OpenStack Israel 2015
- 1. @Livnat_Peer Sr. Engineering Manager, Red Hat @ArthurBerezin Sr. Technical Product Manager, Red Hat Neutron High Availability OpenStack Israel Tel-Aviv June 2015
- 2. Agenda HA Enabling Technologies Pacemaker and HAProxy Neutron Built-in Mechanisms DHCP Agent HA L3 Agent with Virtual Router Redundancy Protocol(VRRP) Distributed Virtual Routing(DVR)
- 3. cc: Morio2015 Source: https://www.wikiwand.com/en/Scuderia_Ferrari
- 5. High Availability â Minimize Downtime By Avoiding SPOF â Service redundancy â Active-Active When possible â Stateless services â Built-in HA mechanisms â Active-Passive for others â Scale out Architecture Add nodes as you go
- 7. HA Enabling Technologies Pacemaker, HAProxy
- 8. â Cluster Resource Manager â Uses Corosync for cluster communication â Monitor and Control Resources: â Floating Virtual IP Address (VIP) â SystemD/LSB/OCF Services â Cloned Services(Active/Active) â STONITH - Fencing with Power Management â Important for ensuring data consistency Pacemaker
- 9. â Virtual IP(VIP) â SystemD Cloned Resource â STONITH Fencing Pacemaker OpenStack Service Node 2 - 192.168.1.2Node 1 - 192.168.1.1 pcsd pcsd Cloned STONITH STONITH Service Service Service Virtual IP 10.0.0.1
- 10. HAProxy Load Balancer Load Balancing and Proxy for HTTP/TCP â Mature and popular with web applications â Health Checking â Load Distribution
- 11. â Load Distribution â Round Robin, â Stick-Table â API Isolation â Failure Detection Node 1 Node 2 Node 3 HAProxy Load Balancer Service Service HAProxy
- 12. Avoiding SPOFs A day in a Highly Available Service Life
- 13. Neutron-Server Controller Give Me Horizon Web UI NOW!
- 14. Neutron-Server Controller Give Me Horizon Web UI NOW! Single Point Of Failure
- 15. Neutron-Server Controller 1 Neutron-Server Controller 2 Neutron-Server Controller 3 Give Me Horizon Web UI NOW! HAProxy Controller 1
- 16. Neutron-Server Controller 1 Neutron-Server Controller 2 Neutron-Server Controller 3 Give Me Horizon Web UI NOW! HAProxy Controller 1 Single Point Of Failure Each Could Fail
- 17. Neutron-Server Controller 1 Neutron-Server Controller 2 Neutron-Server Controller 3 Give Me Horizon Web UI NOW! HAProxy Controller 1 Single Point Of Failure Pacemaker Cloned Horizon Service
- 18. Neutron-Server Controller 1 Neutron-Server Controller 2 Neutron-Server Controller 3 Give Me Horizon Web UI NOW! HAProxy Controller 1 HAProxy Controller 3 HAProxy Controller 2 Pacemaker Cloned Horizon Service Pacemaker Cloned HAProxy Service
- 19. Pacemaker Cloned HAProxy Service Neutron-Server Controller 1 Neutron-Server Controller 2 Neutron-Server Controller 3 HAProxy Controller 1 HAProxy Controller 3 HAProxy Controller 2 Give Me Horizon Web UI NOW! Horizon VIP Pacemaker Cloned Horizon Service
- 21. â External mechanisms â Neutron built-in mechanisms â Reference implementation vs. vendors code My HA Solution
- 22. Architecture - Assuming Centralized Network Node Compute Node Controller Node Network Node Neutron server MySQL server Neutron server Neutron serverRabbitmq server Neutron server OVS agent OVS OVS Agent keepalived Neutron serverOVS DHCP agentDHCP Agent Neutron serverMetadata Agent Metadata Proxy dnsmasq Internet External Network API Network Management Network Data Network L3 Agent
- 23. DHCP Agent â IP address allocation is done by the Neutron server â dnsmasq is used as a distribution mechanism of predefined allocations â The DHCP protocol allows multiple DHCP servers to co-exist while serving the same pool â Configuration in Neutron neutron.conf : dhcp_agents_per_network = X OVS Agent Neutron serverOVS DHCP agent Neutron serverMetadata Agent Metadata Proxy dnsmasq L3 Agent keepalived DHCP Agent
- 24. â Dynamic process creation: dnsmasq, keepalived, metadata proxy etc. â ProcessMonitor check processes liveliness periodically â Optional actions: â Respawn process â Exit agent â Notify (not available yet) â Default configuration check_child_processes_action = respawn check_child_processes_period = 0 Process Monitoring OVS Agent Neutron serverOVS DHCP agentDHCP Agent Neutron serverMetadata Agent Metadata Proxy dnsmasq L3 Agent keepalived
- 25. Metadata Agent OVS What Else? DHCP Agent Metadata Proxy dnsmasq L3 Agent keepalived OVS Agent OVS Metadata Agent
- 26. What Else? Metadata Agent OVS DHCP Agent Metadata Proxy dnsmasq L3 Agent keepalived OVS Agent OVS Metadata Agent
- 27. What Else? Metadata Agent OVS DHCP Agent Metadata Proxy dnsmasq L3 Agent keepalived OVS Agent OVS Metadata Agent
- 28. VRRP (Virtual Router Redundancy Protocol) â Providing HA of the networkâs default gateway â Configuring default gateway as VIP + Virtual MAC â Gratuitous ARP after failover Sync Net
- 29. L3 HA Implementing VRRP â Using keepalived which internally implements VRRP â Creating a per tenant HA network, used for VRRP sync messages â When HA router is created it is scheduled on multiple network nodes (Configurable) â New in Kilo â Report which network node is hosting the master instance â On the work â L3 HA + l2pop â External interface tracking â L3 HA+DVR
- 30. Traffic Flow 3-tier Application Host 1 WWW VM Host 2 App VM Host 3 DB VM Network Node Virtual Router
- 31. DVR â Distributed Virtual Router â DVR is moving most of the routing to the compute node â Isolating the failure domain of the network node â Optimizing the network flow â Traffic types â East â West (Within the tenant, different networks) â North â South with floating IP (VM to/from external network) â North â South without floating IP (Based on SNAT) Direct between compute nodes Through network node
- 32. Architecture - Assuming DVR Compute Node Controller Node Network Node Neutron server MySQL server Neutron server Neutron serverRabbitmq server Internet External Network API Network Management Network Data Network Network Node OVS Agent keepalived Neutron serverOVS DHCP agentDHCP Agent Neutron serverMetadata Agent Metadata Proxy dnsmasq L3 Agent Neutron server OVS agent OVS
- 33. Architecture - Assuming DVR Compute Node Controller Node Network Node Neutron server MySQL server Neutron server Neutron serverRabbitmq server Internet External Network API Network Management Network Data Network Network Node OVS Agent keepalived Neutron serverOVS DHCP agentDHCP Agent Neutron serverMetadata Agent Metadata Proxy dnsmasq L3 Agent Neutron server OVS agent OVS
- 34. Architecture - Assuming DVR Compute Node Controller Node Network Node Neutron server MySQL server Neutron server Neutron serverRabbitmq server Neutron server OVS agent OVS Internet External Network API Network Management Network Data Network Compute Node Neutron server OVS agent Neutron serverOVS L3 agent Neutron serverMetadata agent Metadata Proxy Network Node OVS Agent keepalived Neutron serverOVS DHCP agentDHCP Agent Neutron serverMetadata Agent Metadata Proxy dnsmasq L3 Agent
- 35. Summary â No one stop shop â Maximize the use of built-in solutions â They are vendor neutral â Highly maintained â Widely documented â Understand what you need, use the appropriate tools â DVR vs VRRP â What size is your deployment, maybe A/P is good enough... â The more complicated the solution is the more likely it is to have bugs
- 36. Thank You
- 37. Resources â http://assafmuller.com â http://specs.openstack.org/openstack/neutron-specs/specs/kilo/agent-child-processes-statu s.html â https://github.com/beekhof/osp-ha-deploy/blob/master/ha-openstack.md â https://docs.google.com/document/d/1jCmraZGirmXq5V1MtRqhjdZCbUfiwBhRkUjDXGt5QUQ /edit â https://docs.google.com/document/d/1jCmraZGirmXq5V1MtRqhjdZCbUfiwBhRkUjDXGt5QUQ /edit â https://www.youtube.com/watch?v=00j1x-T1vhA