FreeBSD VPC Introduction
3 likes2,357 views
The document provides an overview of the FreeBSD/VPC virtual private cloud solution. Key points include: - VPC uses the bhyve hypervisor for virtualization with good CPU and memory isolation between guests. - Network isolation between guests is challenging with existing FreeBSD networking approaches like tap/bridge/vxlan due to performance issues. - A new VPC subsystem is proposed to provide dedicated virtual network interfaces (vmnic, vpcp) for guests with improved performance. - The VPC solution uses VXLAN encapsulation and unique VXLAN network identifiers (VNIs) to provide overlay network isolation between guests on different hosts in a multi-host deployment.
![vpc_id_t
int
vpc_open(const vpc_id_t *vpc_id,
vpc_handle_type_t obj_type,
vpc_flags_t flags);
16 bytes, UUID-like:
type ID struct {
TimeLow uint32
TimeMid uint16
TimeHi uint16
ClockSeqHi uint8
ObjType ObjType
Node [6]byte // Default MAC address vpc(4)
}](https://image.slidesharecdn.com/freebsdvpcintroduction-180315215710/85/FreeBSD-VPC-Introduction-34-320.jpg)
![VTEP: VXLAN Tunnel End Point
• vpcsw(4) traps broadcast packets
• If the packet is:
• a broadcast packet and
• either an IPv4 ARP or IPv6 ND packet
Packet is added to a knote (part of kqueue(2)) and passed to all
kqueue(2) subscribers filtering for EVFILT_VPC filters
• Packet payload is stored in user-allocated buffer pointed to by ext[0]
• Userland utility parses packet and performs lookup
• Overlay and underlay forwarding information written back to the kernel
via vpc_ctl(2)
• vpcsw(4) caches forwarding information for the src/dst MAC tuple](https://image.slidesharecdn.com/freebsdvpcintroduction-180315215710/85/FreeBSD-VPC-Introduction-65-320.jpg)
FreeBSD VPC Introduction
- 1. FreeBSD/VPC Introduction to Virtual Private Cloud
- 3. Compute Isolation Status • bhyve(4) is a stable, performant hypervisor • ZFS zvol passthrough to guests works well TIP: use multiple zvols per guest and stripe IO across devices using lvcreate(8) • Good CPU isolation - CPU pinned for low-density workloads • Perfect memory isolation (guest memory is wired)
- 4. Compute Isolation em0 Guest 1 Customer A Guest 2 Customer B
- 5. Network Isolation Status • Network isolation is not core to bhyve(4) today • Use of VNET(9) for manipulating FIBS for tap(4) interfaces is possible, but limited and not performant
- 6. Guest Workloads em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B bridge0 tap51 tap52
- 7. Incomplete Solution • bhyve(4) guests run customer workloads • Cloud providers need a single FIB for the underlay network • Guests run in isolated overlay networks • How do you map guests to their respective overlay network?
- 8. Multi-Host Network Isolation em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B ???
- 9. Possible Solution • Plug a tap(4) device into a guest • Plug tap(4) device into a bridge(4) • Plug the physical or cloned interface into the underlay bridge(4)
- 10. if_bridge(4) Approach em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B tap51 bridge0 tap50 tap52 bridge2bridge1
- 11. Possible Solution++ • Plug a tap(4) device into a guest • Plug tap(4) device into a bridge(4) • Plug a vxlan(4) interface into a per-subnet bridge(4) • Plug the vxlan(4) into an underlay bridge(4) instance • Plug the physical or cloned interface into the underlay bridge(4)
- 12. Fuster Cluck Isolation em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B bridge1 tap51 tap52tap50 bridge0 vxlan1vxlan0 bridge2
- 13. Sad Performance • Performance was "uninteresting" • 1-2Gbps?
- 14. Problems with tap(4)/bridge(4)/vxlan(4)/VNET(9) • tap(4) is slow • bridge(4) is slow • vxlan(4) sends received packets through ip_input() twice (i.e. "sub-optimal") • VNET(9) virtualizes underlay networks, not overlay networks • How do you ARP between VMs in the overlay network? • How do you perform vxlan(4) encap?
- 15. uvxbridge(8) POC uvxbridge(8) POC provides a high PPS interface and guest VXLAN encapsulation using netmap(9)/ ptnetmap(4): https://github.com/joyent/uvxbridge
- 16. Guest Workloads em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B tap51 tap52tap50 uvxbridge
- 17. Reasonable Performance • 21Gbps guest-to-guest on the same host • 15Gbps guest-to-guest across the wire • Supports AES PSK encryption for cross-DC traffic • Incomplete, but a successful POC
- 18. New Approach • Build a network isolation subsystem aligned with the capabilities of network hardware • Reduce the number of times a packet is copied • Reduce context switches • Build a bottom-up abstraction rooted in the capabilities of hardware, not a hardware implementation defined in terms of an administrative policy
- 19. FreeBSD/VPC em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vmnic1 ethlink0 vmnic0 vmnic2 vpcsw1vpcsw0
- 20. FreeBSD/VPC em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vmnic1 ethlink0 vmnic0 vmnic2 vpcsw1 vpcsw0 vpcp0 vpcp1 vpcp3 vpcp4 vpcp5
- 21. FreeBSD/VPC Multi-Host em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 ???
- 22. VXLAN to the Rescue • Encapsulates all IP packets as UDP • Adds a preamble to IP packet • Tags packets and with a VXLAN ID, known as a VNI • VXLAN is similar to VLAN tagging, but embeds tagging in the IP header, not in the L2 frame
- 23. VXLAN Encapsulated Ethernet Packet Physical Ethernet Frame 1500B OuterFrameChecksum OuterEthernetHeader OuterIPHeader OuterUDPHeader VXLANHeader InnerSourceMAC(SMAC) InnerDestMAC(DMAC) 802.1QHeader Payload EtherType
- 24. FreeBSD/VPC Multi-Host em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 VNI 123 VNI 987 VNI 123 VNI 987 VXLAN Packets em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2
- 25. vpc(4) Interfaces • vpcsw(4) - switches packets - one switch per customer per host, multiple subnets supported in the same switch • vmnic(4) - dedicated guest NIC, looks like a virtio network device to guests • vpcp(4) - plugs vmnic(4) ports into vpcsw(4) switches • vpci(4) - Non-bhyve(4) interface, usable in jails(2) • ethlink(4) - Performs unencapsulated packet forwarding, wraps a cloned or physical ethernet interface • vpclink(4) - Performs VXLAN encapsulation
- 26. Interface Design
- 27. ioctl(2) • Initially used ioctl(2) • Unable to completely secure via capsicum(4) - able to specify flags, but not scope the target device • ioctl(2) is the kernel equivalent of an HTTP PUT, the interface for everything without a design "Hi ifconfig(8), I'm looking at you."
- 28. One Device Per Object? • New device in /dev for every new interface: /dev/vpcsw0 /dev/vpcsw1 /dev/vpcp0 • Query via reads to individual devices in/dev • Control devices via writes • Security via devd(8)? • Using VFS ACLs to secure network primitives?
- 29. One Device Per Object? • New device in /dev for every new interface: /dev/vpcsw0 /dev/vpcsw1 /dev/vpcp0 • Query via reads to individual devices in/dev • Control devices via writes • Security via devd(8)? • Using VFS ACLs to secure network primitives? Da Vinci mashed up with ...
- 30. One Device Per Object? • New device in /dev for every new interface: /dev/vpcsw0 /dev/vpcsw1 /dev/vpcp0 • Query via reads to individual devices in/dev • Control devices via writes • Security via devd(8)? • Using VFS ACLs to secure network primitives? Da Vinci mashed up with Jackson Pollock.
- 31. One Device Per Object? • New device in /dev for every new interface: /dev/vpcsw0 /dev/vpcsw1 /dev/vpcp0 • Query via reads to individual devices in/dev • Control devices via writes • Security via devd(8)? • Using VFS ACLs to secure network primitives? Da Vinci mashed up with Jackson Pollock. Kernel API design hate crime.
- 32. New System Calls • vpc_open(2) - Creates a new VPC descriptor • vpc_ctl(2) - Manipulates VPC descriptors • Capsicum-like, intended for privilege separation • Intended for idempotent tooling • Makes aggressive use of UUIDs as operator handles to be compatible with Triton
- 33. vpc_open(2) int vpc_open(const vpc_id_t *vpc_id, vpc_type_t obj_type, vpc_flags_t flags); • Creates a new "VPC descriptor" • Similar to open(2) • Manipulate descriptor via vcp_ctl(2) • Responds to close(2) • Priv-sep native security semantics • Version aware • System Call 580
- 34. vpc_id_t int vpc_open(const vpc_id_t *vpc_id, vpc_handle_type_t obj_type, vpc_flags_t flags); 16 bytes, UUID-like: type ID struct { TimeLow uint32 TimeMid uint16 TimeHi uint16 ClockSeqHi uint8 ObjType ObjType Node [6]byte // Default MAC address vpc(4) }
- 35. vpc_id_t int vpc_open(const vpc_id_t *vpc_id, vpc_handle_type_t obj_type, vpc_flags_t flags); • All VPC objects have a MAC address • Reused the node component of the vpc_id to set the MAC address
- 36. vpc_handle_type_t int vpc_open(const vpc_id_t *vpc_id, vpc_type_t obj_type, vpc_flags_t flags); typedef struct { uint64_t vht_version:4; uint64_t vht_pad1:4; uint64_t vht_obj_type:8; uint64_t vht_pad2:48; } vpc_handle_type_t;
- 37. vpc_obj_type int vpc_open(const vpc_id_t *vpc_id, vpc_type_t obj_type, vpc_flags_t flags); enum vpc_obj_type { VPC_OBJ_INVALID = 0, VPC_OBJ_SWITCH = 1, VPC_OBJ_PORT = 2, VPC_OBJ_ROUTER = 3, VPC_OBJ_NAT = 4, VPC_OBJ_VPCLINK = 5, VPC_OBJ_VMNIC = 6, VPC_OBJ_MGMT = 7, VPC_OBJ_ETHLINK = 8, VPC_OBJ_META = 9, VPC_OBJ_TYPE_ANY = 10, VPC_OBJ_TYPE_MAX = 10, };
- 38. vpc_id_t int vpc_open(const vpc_id_t *vpc_id, vpc_handle_type_t obj_type, vpc_flags_t flags); • 16 Versions • 255 Object Types • 4080 Object Types ought to be enough for anybody.
- 39. vpc_flags_t int vpc_open(const vpc_id_t *vpc_id, vpc_type_t obj_type, vpc_flags_t flags); #define VPC_F_CREATE (1ULL << 0) #define VPC_F_OPEN (1ULL << 1) #define VPC_F_READ (1ULL << 2) #define VPC_F_WRITE (1ULL << 3) NOTE: going to revisit flags to enable a bit field for capabilities per VPC Object Type
- 40. vpc_ctl(2) int vpc_ctl(int vpcd, vpc_op_t op, size_t keylen, const void *key, size_t *vallen, void *buf); • Only interface for manipulating a VPC object • Available operations different per VPC Object • Inspired by ioctl(2), capsicum(4), and HTTP • System Call 581
- 41. vpc_ctl(2) enum vpc_vpcsw_op_type { VPC_VPCSW_INVALID = 0, VPC_VPCSW_PORT_ADD = 1, VPC_VPCSW_PORT_DEL = 2, VPC_VPCSW_PORT_UPLINK_SET = 3, VPC_VPCSW_PORT_UPLINK_GET = 4, VPC_VPCSW_STATE_GET = 5, VPC_VPCSW_STATE_SET = 6, VPC_VPCSW_RESET = 7, VPC_VPCSW_RESPONSE_NDV4 = 8, VPC_VPCSW_RESPONSE_NDV6 = 9, VPC_VPCSW_RESPONSE_DHCPV4 = 10, VPC_VPCSW_RESPONSE_DHCPV6 = 11, VPC_VPCSW_OP_TYPE_MAX = 11, };
- 42. vpc_ctl(2) enum vpc_vpcp_op_type { VPC_VPCP_INVALID = 0, VPC_VPCP_CONNECT = 1, VPC_VPCP_DISCONNECT = 2, VPC_VPCP_VNI_GET = 3, VPC_VPCP_VNI_SET = 4, VPC_VPCP_VTAG_GET = 5, VPC_VPCP_VTAG_SET = 6, VPC_VPCP_UNUSED7 = 7, VPC_VPCP_UNUSED8 = 8, VPC_VPCP_PEER_ID_GET = 9, VPC_VPCP_MAX = 9, };
- 43. vpc_ctl(2) enum vpc_vmnic_op_type { VPC_VMNIC_INVALID = 0, VPC_VMNIC_NQUEUES_GET = 1, VPC_VMNIC_NQUEUES_SET = 2, VPC_VMNIC_UNUSED3 = 3, VPC_VMNIC_UNUSED4 = 4, VPC_VMNIC_UNUSED5 = 5, VPC_VMNIC_UNUSED6 = 6, VPC_VMNIC_ATTACH = 7, VPC_VMNIC_MSIX = 8, VPC_VMNIC_FREEZE = 9, VPC_VMNIC_UNFREEZE = 10, VPC_VMNIC_OP_TYPE_MAX = 10, };
- 44. What's it all mean? • Network interfaces are first-class objects • Network interfaces are pluggable • Configurations can be arbitrarily complex • All vpc(4) interfaces are iflib(9) and can be tcpdump(8)'ed • Performance is nice
- 45. Tooling
- 46. Tooling • Able to be cross-compiled • Developer-friendly • Stable ABI • Idempotent Operations • No a priori knowledge of the target OS required (i.e. no headers required to cross-compile tooling)
- 47. Tooling • Able to be cross-compiled • Developer-friendly • Stable ABI • Idempotent Operations • No a priori knowledge of the target OS required (i.e. no headers required to cross-compile tooling)
- 49. ELI5: vpc(4) edition em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 VNI 123 VNI 987 VNI 123 VNI 987 VXLAN Packets em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2
- 50. ELI5: vpc(4) Assumptions • Guest is running Ubuntu or CentOS Linux • Multi-Queue TX/RX in Host • Multiple network queues available in the guest • Underlay hosts can pass traffic • Overlay hosts are in the same subnet
- 51. ELI5: vpc(4) edition em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vmnic1 ethlink0 vmnic0 vmnic2 vpcsw1 vpcsw0 vpcp0 vpcp1 vpcp3 vpcp4 vpcp5
- 52. ELI5: vpc(4) edition em0 Guest 3 Customer B Guest 2 Customer B vmnic1 ethlink0 vmnic2 vpcsw1 vpcp3 vpcp4 vpcp5
- 53. ELI5: vpcsw(4) Setup VNI=123 VLAN=456 VPCSW0_ID=da64c3f3-095d-91e5-df01-5aabcfc52468 vpc switch create --switch-id=${VPCSW0_ID} --vni=${VNI} --vlan=${VLAN}
- 54. ELI5: vpc(4) edition em0 Guest 3 Customer B Guest 2 Customer B vpcsw1
- 55. ELI5: vmnic(4) Setup VMNIC0_ID=07f95a11-6788-2ae7-c306-ba95cff1db38 VPCP0_ID=fd436f9c-1f77-11e8-8002-0cc47a6c7d1e vpc vmnic create --vmnic-id=${VMNIC0_ID} vpc switch port add --switch-id=${VPCSW0_ID} --port-id=${VPCP0_ID} vpc switch port connect --port-id=${VPCP0_ID} --interface-id=${VMNIC0_ID}
- 56. ELI5: vpc(4) edition em0 Guest 3 Customer B Guest 2 Customer B vmnic1 vpcsw1 vpcp3
- 57. ELI5: vmnic(4) Setup VMNIC1_ID=a774ba3a-1f77-11e8-8006-0cc47a6c7d1e VPCP1_ID=0ebf50e1-1f79-11e8-8002-0cc47a6c7d1e vpc vmnic create --vmnic-id=${VMNIC1_ID} vpc switch port add --switch-id=${VPCSW0_ID} --port-id=${VPCP1_ID} vpc switch port connect --port-id=${VPCP1_ID} --interface-id=${VMNIC1_ID}
- 58. ELI5: vpc(4) edition em0 Guest 3 Customer B Guest 2 Customer B vmnic1 vmnic2 vpcsw1 vpcp3 vpcp4
- 59. ELI5: ethlink(4) Setup ETHLINK0_ID=5c4acd32-1b8d-11e8-b408-0cc47a6c7d1e UPLINK_PORT_ID=ea58b648-203b-a707-cdf6-7a552c8d5295 UPLINK_IF=em0 vpc switch port add --switch-id=${VPCSW0_ID} --uplink --port-id=${UPLINK_PORT_ID} --l2-name=${UPLINK_IF} --ethlink-id=${ETHLINK0_ID}
- 60. ELI5: vpc(4) edition em0 Guest 3 Customer B Guest 2 Customer B vmnic1 ethlink0 vmnic2 vpcsw1 vpcp3 vpcp4 vpcp5
- 61. ELI5: vpc(4) + VXLAN
- 62. ELI5: vpc(4) edition em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 VNI 123 VLAN 456 VNI 987 VNI 123 VTAG 456 VNI 987 VXLAN Packets em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 10.65.5.161 10.65.5.162
- 63. ELI5: vpclink(4) Setup • How do you do ARP or IPv6 Neighbor Discovery? • Broadcast? • Multicast?
- 64. ELI5: vpc(4) edition em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 VNI 123 VLAN 456 VNI 987 VNI 123 VTAG 456 VNI 987 VXLAN Packets em0 Guest 1 Customer A Guest 3 Customer B Guest 2 Customer B vpclink0 vmnic0 vpcsw1vpcsw0 vmnic1 vmnic2 10.65.5.161 10.65.5.162 ???
- 65. VTEP: VXLAN Tunnel End Point • vpcsw(4) traps broadcast packets • If the packet is: • a broadcast packet and • either an IPv4 ARP or IPv6 ND packet Packet is added to a knote (part of kqueue(2)) and passed to all kqueue(2) subscribers filtering for EVFILT_VPC filters • Packet payload is stored in user-allocated buffer pointed to by ext[0] • Userland utility parses packet and performs lookup • Overlay and underlay forwarding information written back to the kernel via vpc_ctl(2) • vpcsw(4) caches forwarding information for the src/dst MAC tuple
- 66. Ongoing Work • Firewalling - integrated at vpcp(4) • Routing • NAT • Userland Control Plane (including setup and teardown of bhyve(4) guests via something not a shell script)
- 67. vpc(8) % doas vpc vm create % doas vpc vm run • bhyve(4) VM creation and launch tool • Unifies network isolation with compute isolation
- 68. Desktop Software + vpc(4) • vpc(4) + NAT == Desktop Software Hotness • Ability to sandbox VMs with no prior knowledge of the host's networking • Uses: % vagrant up % packer build No more dependency on pf(4) or dnsmasq(8)
- 69. Code • Kernel: https://github.com/joyent/freebsd/tree/projects/VPC • Kernel Libraries: https://github.com/joyent/freebsd/tree/projects/VPC/ libexec/go/src/go.freebsd.org/sys/vpc • Userland tooling: https://github.com/sean-/vpc Future home: https://github.com/joyent-/vpc
- 70. Questions?