Accelerating Virtual Machine Access with the Storage Performance Development Kit (SPDK) – Vhost Deep Dive
- 1. Anu H Rao Storage Software Product line Manager Datacenter Group, Intel® Corp
- 2. Notices & Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Check with your system manufacturer or retailer or learn more at intel.com. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/benchmarks . Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/benchmarks . Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. © 2017 Intel Corporation.
- 4. Latency(µs) Technology claims are based on comparisons of latency, density and write cycling metrics amongst memory technologies recorded on published specifications of in-market memory products against internal Intel specifications. 0 25 50 75 100 125 150 175 200 10,000 HDD +SAS/SATA SSDNAND +SAS/SATA SSDNAND +NVMe™ SSDoptane™ +NVMe™ kerneldriveroverhead1-8% kerneldriverOverhead<0.01% kerneldriveroverhead30%-50% Drive Latency Controller Latency Driver Latency The Challenge: Media Latency
- 5. Storage Performance Development Kit 5 Scalable and Efficient Software Ingredients • User space, lockless, polled-mode components • Up to millions of IOPS per core • Designed to extract maximum performance from non-volatile media Storage Reference Architecture • Optimized for latest generation CPUs and SSDs • Open source composable building blocks (BSD licensed) • Available via SPDK.io • Follow @SPDKProject on twitter for latest events and activities
- 6. Benefits of using SPDK SPDK more performance from CPUs, non- volatile media, and networking 10X MORE IOPS/coreUp to for NVMe-oF* vs. Linux kernel for NVMe vs. Linux kernel8X MORE IOPS/coreUp to Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance 50% BETTERUp to for RocksDB workloadTail Latency 3X BETTERUp to for Virtualized StorageIOPS/core & Latency Up to10.8 Million IOPS with Intel® Xeon Scalable Family and 24 Intel® Optane™ SSD DC P4800X 1.5X BETTERUp to for NVMEoF vs Kernel for Optane SSDLatency
- 7. SPDK Community http://SPDK.IO Real Time Chat w/ Development Community Backlog and Ideas for Things to Do Main Web Presence Email Discussions Weekly Calls Multiple Annual Meetups Code Reviews & Repo Continuous Integration
- 9. Architecture Drivers Storage Services Storage Protocols iSCSI Target NVMe-oF* Target SCSINVMe NVMe Devices Blobstore NVMe-oF* Initiator Intel® QuickData Technology Driver Block Device Abstraction (bdev) Ceph RBD Linux AIO Logical Volumes 3rd Party NVMe NVMe* PCIe Driver Released New release 18.01 1H‘18 vhost-blk Target BlobFS Integration RocksDB Ceph Core Application Framework GPT PMDK blk virtio scsi QEMU QoS Linux nbd RDMA virtio blk vhost-scsi Target
- 11. Virtio • Paravirtualized driver specification • Common mechanisms and layouts for device discovery, I/O queues, etc. • virtio device types include: • virtio-net • virtio-blk • virtio-scsi • virtio-gpu • virtio-rng • virtio-cryptoHypervisor (i.e. QEMU/KVM) Guest VM (Linux*, Windows*, FreeBSD*, etc.) virtio front-end drivers virtio back-end drivers device emulation virtqueuevirtqueuevirtqueue 11
- 12. QEMU Kernel Guest VM Guest kernel Application virtqueue I/O Processing AIO QEMUVirtIOSCSI 12 1. Add IO to virtqueue 2. IO processed by QEMU 3. IO issued to kernel 4. Kernel pins memory 5. Device executes IO 6. Guest completion interrupt
- 14. vhost target (kernel or userspace) Vhost • Separate process for I/O processing • vhost protocol for communicating guest VM parameters • memory • number of virtqueues • virtqueue locations Hypervisor (i.e. QEMU/KVM) Guest VM (Linux*, Windows*, FreeBSD*, etc.) virtio front-end drivers device emulation virtio back-end drivers virtqueuevirtqueuevirtqueue vhostvhost
- 15. QEMU Kernel Guest VM Guest kernel Application virtqueue vhost-kernel AIO KernelVHOST 15 1. Add IO to virtqueue 2. Write virtio doorbell 3. Wake vhost kernel 4. Kernel pins memory 5. Device executes IO 6. Guest completion interrupt kvm
- 17. 17 Host Memory QEMU Guest VM virtio-scsi Shared Guest VM Memory SPDK vhost vhost DPDK vhost virtio-scsi virtqueuevirtqueuevirtqueue eventfd UNIX domain socket SPDKVHOSTArchitecture
- 18. QEMU Kernel Guest VM Guest kernel Application virtqueue SPDK Vhost vhost i/o SPDKVHOST 18 1. Add IO to virtqueue 2. Poll virtqueue 3. Device executes IO 4. Guest completion interrupt kvm
- 19. Architecture Drivers Storage Services Storage Protocols iSCSI Target NVMe-oF* Target SCSI vhost-scsi Target NVMe NVMe Devices Blobstore NVMe-oF* Initiator Intel® QuickData Technology Driver Block Device Abstraction (bdev) Ceph RBD Linux AIO Logical Volumes 3rd Party NVMe NVMe* PCIe Driver Released New release 18.01 1H‘18 vhost-blk Target BlobFS Integration RocksDB Ceph Core Application Framework GPT PMDK blk virtio scsi QEMU QoS Linux nbd RDMA virtio blk
- 20. Sharing SSDs in userspace Typically not 1:1 VM to local attached NVMe SSD otherwise just use PCI direct assignment What about SR-IOV? SR-IOV SSDs not prevalent yet precludes features such as snapshots What about LVM? LVM depends on Linux kernel block layer and storage drivers (i.e. nvme) SPDK wants to use userspace polled mode drivers SPDK Blobstore and Logical Volumes!
- 22. SPDK vhost Performance 0 10 20 30 40 50 Linux QEMU SPDK QD=1 Latency (in us) System Configuration: 2S Intel® Xeon® Platinum 8180: 28C, E5-2699v3: 18C, 2.5GHz (HT off), Intel® Turbo Boost Technology enabled, 12x16GB DDR4 2133 MT/s, 1 DIMM per channel, Ubuntu* Server 16.04.2 LTS, 4.11 kernel, 23x Intel® P4800x Optane SSD – 375GB, 1 SPDK lvolstore or LVM lvgroup per SSD, SPDK commit ID c5d8b108f22ab, 46 VMs (CentOS 3.10, 1vCPU, 2GB DRAM, 100GB logical volume), vhost dedicated to 10 cores As measured by: fio 2.10.1 – Direct=Yes, 4KB random read I/O, Ramp Time=30s, Run Time=180s, Norandommap=1, I/O Engine = libaio, Numjobs=1 Legend: Linux: Kernel vhost-scsi QEMU: virtio-blk dataplane SPDK: Userspace vhost-scsi SPDK up to 3x better efficiency and latency
- 23. 48 VMs: vhost-scsi performance (SPDK vs. Kernel) Intel Xeon Platinum 8180 Processor, 24x Intel P4800x 375GB 2 partitions per VM, 10 vhost I/O processing cores 1 11 2.86 2.77 3.4 9.23 8.98 9.49 0 1 2 3 4 5 6 7 8 9 10 4K 100% Read 4K 100% Write 4K 70%Read30%Write IOPSinMillions vhost-kernel vhost-spdk 3.2x 2.7x3.2x • Aggregate IOPS across all 48x VMs reported. All VMs on separate cores than vhost-scsi cores. • 10 vhost-scsi cores for I/O processing • SPDK vhost-scsi up to 3.2x better with 4K 100% Random read I/Os • Used cgroups to restrict kernel vhost-scsi processes to 10 cores System Configuration:Intel Xeon Platinum 8180 @ 2.5GHz. 56 physical cores 6x 16GB, 2667 DDR4, 6 memory Channels, SSD: Intel P4800x 375GB x24 drives, Bios: HT disabled, p-states enabled, turbo enabled, Ubuntu 16.04.1 LTS, 4.11.0 x86_64 kernel, 48 VMs, number of partition: 2, VM config : 1core 1GB memory, VM OS: fedora 25, blk-mq enabled, Software packages: Qemu-2.9, libvirt-3.0.0, spdk (3bfecec994), IO distribution: 10 vhost-cores for SPDK / Kernel. Rest 46 cores for QEMU using cgroups, FIO-2.1.10 with SPDK plugin, io depth=1, 8, 32 numjobs=1, direct=1, block size 4k
- 24. VM Density: Rate Limiting 20K IOPS per VM Intel Xeon Platinum 8180 Processor, 24x Intel P4800x 375GB 10 vhost-scsi cores 1 11 0 10 20 30 40 50 60 70 80 90 100 0 200000 400000 600000 800000 1000000 1200000 1400000 1600000 1800000 24 48 96 %CPUUtilization (lowerisbetter) IOPS (higherisbetter) No. of VMs Kernel IOPS SPDK IOPS Kernel CPU Util. SPDK CPU Util. • % CPU utilized shown from VM side • Each VM was running queue depth=1, 4KB random read workload • Hyper threading enabled to allow 112 cores. • Each VM rate limited to 20K IOPS using cgroups • SPDK able to scale to 96 VMs, supporting 20K per VM. Kernel scale till 48 VMs. Beyond 48 VMs, 10 vhost- cores seem bottleneck System Configuration:Intel Xeon Platinum 8180 @ 2.5GHz. 56 physical cores 6x 16GB, 2667 DDR4, 6 memory Channels, SSD: Intel P4800x 375GB x24 drives, Bios: HT disabled, p-states enabled, turbo enabled, Ubuntu 16.04.1 LTS, 4.11.0 x86_64 kernel, 48 VMs, number of partition: 2, VM config : 1core 1GB memory, VM OS: fedora 25, blk-mq enabled, Software packages: Qemu-2.9, libvirt-3.0.0, spdk (3bfecec994), IO distribution: 10 vhost-cores for SPDK / Kernel. Rest 46 cores for QEMU using cgroups, FIO-2.1.10 with SPDK plugin, io depth=1, 8, 32 numjobs=1, direct=1, block size 4k
- 26. SPDK Vhost BDAL Logical Vol NVMe Driver BDAL NVMe Bdev VM Intel® SSD for Datacenter VMEPHEMERALSTORAGE • Increased efficiency yields greater VM density 26 BDAL Logical Vol VM
- 27. Intel® SSD for Datacenter SPDKSPDK Vhost NVMe-oF Initiator BDAL NVMe-oF BD VM NVMe-oF Target VMRemoteStorage • Enable disaggregation and migration of VMs using remote storage 27
- 28. Ceph Cluster SPDK Intel® SSD for Datacenter Ceph RBD Driver BDAL Ceph Bdev SPDK Vhost VM VMCEphStorage • Potential for innovation in data services • Cache • Deduplication 28
- 29. For More information on SPDK • Visit SPDK.io for tutorials and links to github, maillist, IRC channel and other resources • Follow @SPDKProject on twitter for latest events, blogs and other SPDK community information and activities
- 31. Basic Architecture Configure vhost-scsi controller JSON RPC creates SPDK constructs for vhost device and backing storage creates controller-specific vhost domain socket Logical Core 0 Logical Core 1 vhost-scsi ctrlr NVMe SSD scsi dev scsi lun bdev nvme /spdk/vhost.0
- 32. Basic Architecture Launch VM QEMU connects to domain socket SPDK Assigns logical core Starts vhost dev poller Allocates NVMe queue pair Starts NVMe poller Logical Core 0 Logical Core 1 vhost-scsi ctrlr NVMe SSD scsi dev scsi lun bdev nvme /spdk/vhost.0 vhost-scsi poller VQVQVQ QP bdev-nvme poller VM
- 33. Basic Architecture Repeat for additional VMs pollers spread across available cores Logical Core 0 Logical Core 1 vhost-scsi ctrlr NVMe SSD scsi dev scsi lun bdev nvme /spdk/vhost.0 vhost-scsi poller VQVQVQ QP bdev-nvme poller vhost-scsi poller bdev-nvme poller vhost-scsi poller bdev-nvme poller vhost-scsi poller bdev-nvme poller VM
- 35. Blobstore Design – Design Goals • Minimalistic for targeted storage use cases like Logical Volumes and RocksDB • Deliver only the basics to enable another class of application • Design for fast storage media
- 36. Blobstore Design – High Level Application interacts with chunks of data called blobs Mutable array of pages of data, accessible via ID Asynchronous No blocking, queuing or waiting Fully parallel No locks in IO path Atomic metadata operations Depends on SSD atomicity (i.e. NVMe) 1+ 4KB metadata pages per blob
- 37. Logical Volumes Blobstore plus: UUID xattr for lvolstore, lvols Friendly names – lvol name unique within lvolstore – lvolstore name unique within application Future – snapshots (requires blobstore support) NVMe SSD bdev bdev nvme lvol blobstore lvolstore ... bdev lvol
- 38. Asynchronous Polling Poller execution Reactor on each core Iterates through pollers round-robin vhost-scsi poller – poll for new I/O requests – submit to NVMe SSD bdev-nvme poller – poll for I/O completions – complete to guest VM Logical Core 0 Logical Core 1 vhost-scsi ctrlr NVMe SSD scsi dev scsi lun bdev nvme /spdk/vhost.0 vhost-scsi poller VQVQVQ QP bdev-nvme poller vhost-scsi poller bdev-nvme poller vhost-scsi poller bdev-nvme poller vhost-scsi poller bdev-nvme poller VM