Dreamhost deploying dreamcompute at scale
2 likesā¢826 views
The presentation discusses the collaboration between Cumulus Networks and Akanda at DreamHost to enhance cloud services through open networking and efficient scalability. It highlights the evolution of network architecture from Generation 1 to Generation 3, focusing on the implementation of VXLAN for layer 2 virtualization over layer 3 IP fabrics and the improvements in orchestration and network performance. Key benefits include simplified operations, increased scalability, and integration of open-source solutions, enabling DreamHost to manage over 1,000 customers and thousands of virtual machines effectively.
Dreamhost deploying dreamcompute at scale
- 1. Cumulus and Akanda at DreamHost Driving Scale, Efficiency, and Cost Reduction Presenters: Jonathan LaCour (DreamHost), Nolan Leake (Cumulus Networks) & Mark McClain (Akanda)
- 2. Introduction
- 3. āŖ Founded in 1997 āŖ Managed, mass-market web hosting āŖ ~400,000 customers āŖ Why Cloud? āŖ The rise of AWS āŖ The world needs a viable, open alternative ā¢ Ceph and OpenStack lead the way!
- 4. ā¢ Public cloud compute service ā¢ Built on OpenStack and Ceph ā¢ Core networking requirements ā¢ L2 tenant isolation ā¢ IPv6 ā¢ 10G+ everywhere
- 5. Network: Gen 1 āŖ Physical: White Box switches running Cumulus Linux āŖ L2 isolation: virtualized with Nicira NVP āŖ L3+ āŖ Nicira lacks L3 āŖ Software routing vendors donāt understand cloud āŖ Astara is born!
- 6. āŖ Nicira / VMWare adds L3 āŖ Time for a bake-off! āŖ Astara wins the battle, but gets some enhancements āŖ Move from OpenBSD and PF to Linux and iptables āŖ Significant optimizations to orchestration platform āŖ Gen 2 allows us to scale to 1,000+ customers, thousands of VMs Network: Gen 2
- 7. DreamCompute Network: Generation 3 āŖ VMWare NSX problems āŖ Scale: maxes out around 1,250 tenants āŖ Performance: OVS is slow and unstable āŖ Magic: difficult to debug and operate āŖ Gen 3 is built on open āŖ Physical: Cumulus Linux āŖ L2 isolation: hardware accelerated VXLAN in switch and hypervisor āŖ L3+: Astara Network: Gen 3
- 8. DreamCompute Network: Generation 3 āŖ Simple, open architecture āŖ Operational ease āŖ Proven technology: VXLAN, iptables, Linux networking stack āŖ Astara simplifies Neutron deployment āŖ Performance and scale āŖ Hardware accelerated VXLAN pervasive on switches / NICs āŖ VXLAN tunnels scale up massively āŖ Astara model of virtual network appliances scales easily Gen 3 Benefits
- 9. ā Created to fill in gaps in Neutron ā L3-L7 Service Orchestration for OpenStack ā Dynamic Routing ā IPv6 ā Simplified Operations ā Using standard APIs ā Astara Project ā Open Source ā OpenStack Foundation top-level project
- 10. Reference Neutron Message QueueNeutron Server L2 Agent L3 Agent DHCP Agent Adv ServicesDatabase
- 11. Astara + OpenStack Neutron Message QueueNeutron Server L2 Agent Astara Database
- 12. Astara + OpenStack Neutron Astara Physical Network (L2) Nova Neutron Open: OVS/LinuxBridge Proprietary Astara OTT Platform (L2 Agnostic) Astara Network Services: Routing/LB/FW/VPN OpenStack APIs
- 13. Physical Network
- 14. Traditional L2-centric Design Falls Short āŖ Bottleneck! āŖ Core/Agg limits scale āŖ Dead Agg switch is a Big Deal āŖ Complex, Proprietary āŖ MLAG/vPC/Stacking āŖ HSRP/GLBP/NSRP āŖ Alphabet soup āŖ Complex Failure Modes āŖ Loops āŖ MAC flapping āŖ Large blast radius āŖ Scalability āŖ Limited total network size āŖ Limited number of VLANs
- 15. A Better Way L2 L3
- 16. IP Fabric: CLOS/Fat-tree āŖ No Bottleneck! āŖ Full bandwidth across racks āŖ Crucial for network virtualization āŖ Simple, Open āŖ IP āŖ BGP āŖ Fine-grained failures āŖ BGP runs the Internet āŖ Scales up to any size āŖ Just add more layers!
- 17. Open Networking: Bare-Metal Ecosystem ONIE (Open Network Install Environment)
- 18. Automation and Monitoring āŖ Only way to effectively manage large numbers of switches! āŖ Choice of Automation Tools āŖ DreamHost was already using Chef āŖ But you can use any tool that works on Linux! āŖ Choice of Monitoring Tools āŖ DreamHost was already using collectd+Graphite āŖ SNMP still there for legacy monitoring systems āŖ Other Options āŖ Elastic Search/LogStash/Kibana āŖ Sensu āŖ Even good old MRTG!
- 19. DreamCompute Gen 3 Details
- 20. VXLAN: L2 Virtualization over L3 IP Fabrics āŖ UDP tunnels between vswitches āŖ Guest L2 traffic is safely encapsulated in L3 packets on the physical network āŖ No L2 required in the physical network āŖ What about BUM Packets: Broadcast, Multicast, Unknown Unicast? āŖ āOfficialā RFC7348 answer: Multicast āŖ Multicast is complex and scales poorly: disabled on most networks āŖ Replicator āŖ Cumulus authored, Open Source daemon: https://github.com/CumulusNetworks/vxfld āŖ Replicates BUM packets to multiple unicast receivers āŖ Can run on Linux switches, or Linux servers/hypervisor āŖ Hardware-accelerated when run on Cumulus Linux
- 21. VXLAN: HW VTEP āŖ VTEP: āVXLAN Tunnel Encapsulation/Decapsulation Pointā āŖ Thing that encapsulates virtual network L2 traffic in L3 UDP packets for physical transport āŖ Neutron-managed software VTEPs on hypervisors āŖ Encapsulations/Decapsulates packets for VMs āŖ Cumulus-managed hardware VTEP to connect to non-virtual networks āŖ Encapsulates/Decapsulates packets from VMs to routers, appliances, etc āŖ 100% in hardware, line rate.
- 22. Questions?
- 23. Extras