Kvmopt osforce
- 3. Agenda • CPU • context switch • cache • Memory • IO • Storage • Network
- 4. Context Switch - Intel VT-x Virtualization Technology ring 0 kernel mode ring 3 User mode
- 5. Context Switch - Intel VT-x Virtualization Technology ring 0 kernel mode VMM ring 0 kernel mode VMM ring 3 User mode ring 3 User mode VM ring 0 Kernel mode VM ring 3 User mode
- 6. Context Switch - Intel VT-x Virtualization Technology ring 0 kernel mode VMM ring 0 kernel mode VMM ring 3 User mode ring 3 User mode VM ring 0 Kernel mode VM ring 3 User mode 设置：宿主机BIOS中 启，目前默认 启
- 7. Cache - Node Binding
- 8. Cache - Node Binding • 将qemu进程绑定到特定的CPU node或core上 ——避免L2/L3 Cache miss
- 9. Cache - Node Binding • 将qemu进程绑定到特定的CPU node或core上 ——避免L2/L3 Cache miss • Node binding v.s core binding
- 10. Cache - Node Binding • 将qemu进程绑定到特定的CPU node或core上 ——避免L2/L3 Cache miss • Node binding v.s core binding • 设置：
- 11. Cache - Node Binding • 将qemu进程绑定到特定的CPU node或core上 ——避免L2/L3 Cache miss • Node binding v.s core binding • 设置： • taskset
- 13. Agenda • CPU • Memory • Addressing • Space • IO • Storage • Network
- 14. Addressing - EPT (SLAT) • Extended page tables/second level address translation Vaddr VMM (Page table) Paddr
- 15. Addressing - EPT (SLAT) • Extended page tables/second level address translation Vaddr VMM (Page table) VM Vaddr VM Page table VM Paddr/VMM Vaddr VMM Page table Paddr VMM Paddr
- 16. Addressing - EPT (SLAT) • Extended page tables/second level address translation Vaddr VMM (Page table) VM Vaddr VM Page table VM Paddr/VMM Vaddr VMM Page table Paddr VMM Paddr 设置：宿主机BIOS中 启，目前默认 启
- 18. Addressing - HugePage • 少page table尺寸，降低查找缓存(TLB)的cache-miss，加速VM内存地址 转换
- 19. Addressing - HugePage • 少page table尺寸，降低查找缓存(TLB)的cache-miss，加速VM内存地址 转换 • 默认Page size: 4KB/Hugepage size: 2M
- 20. Addressing - HugePage • 少page table尺寸，降低查找缓存(TLB)的cache-miss，加速VM内存地址 转换 • • 默认Page size: 4KB/Hugepage size: 2M Transparent hugepage： kernel进程khugepaged周期性扫描内存，自动将 地址连续可合并的普通4KB page合并为2MB Hugepage
- 21. Addressing - HugePage • 少page table尺寸，降低查找缓存(TLB)的cache-miss，加速VM内存地址 转换 • • 默认Page size: 4KB/Hugepage size: 2M Transparent hugepage： kernel进程khugepaged周期性扫描内存，自动将 地址连续可合并的普通4KB page合并为2MB Hugepage • 设置：
- 22. Addressing - HugePage • 少page table尺寸，降低查找缓存(TLB)的cache-miss，加速VM内存地址 转换 • • 默认Page size: 4KB/Hugepage size: 2M Transparent hugepage： kernel进程khugepaged周期性扫描内存，自动将 地址连续可合并的普通4KB page合并为2MB Hugepage • 设置： • sysctl -w sys.kernel.mm.transparent_hugepage.enabled=always
- 23. Addressing - HugePage • 少page table尺寸，降低查找缓存(TLB)的cache-miss，加速VM内存地址 转换 • • 默认Page size: 4KB/Hugepage size: 2M Transparent hugepage： kernel进程khugepaged周期性扫描内存，自动将 地址连续可合并的普通4KB page合并为2MB Hugepage • 设置： • • sysctl -w sys.kernel.mm.transparent_hugepage.enabled=always sysctl -w sys.kernel.mm.transparent_hugepage.defrag=always
- 25. Space - KSM • Kernel same-page merging • kernel进程ksmd周期性扫描内存，将内容相同的 page合并， 少物理内存使用量
- 26. Agenda • CPU • Memory • IO • Storage • Network
- 27. Full virtualization v.s. paravirtualization Guest VM Guest VM Drivers Traps Virtual Hardware Interface Virtual Hardware Qemu Qemu kernel kernel Hardware Hardware
- 29. Agenda • CPU • Memory • IO • Storage • Network
- 31. virtio-blk • 基于virtio框架的虚拟PCI磁盘设备 • /dev/vdx -drive file=win_xp.img,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5
- 33. virtio-SCSI • 基于virtio框架的虚拟SCSI磁盘设备 • /dev/sdx -drive file=win_xp.img,if=none,id=drive_0,cache=none,aio=native -device virtio-scsi-pci,drive=drive_0,bus=pci.0,addr=0x5
- 34. 缓存模式 Guest User Space Guest FS page cache Brk Driver writeback cache Qemu Host FS page cache Kernel
- 35. 缓存模式(cont) writeback page cache cache 写同步 说明 (flush) directsync NO NO N/A 无优化 writethrough YES NO YES 依靠Host操作系统优化IO性能 闭Host page cache. none/off NO YES N/A 优化写性能，并保证安全性 writeback YES YES YES 优化读写性能，可能丢失数据 unsafe YES YES NO 优化读写行 干，不保证数据安全 -drive file=win_xp.img,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5
- 37. Native AIO • Native aio: kernel AIO • threaded aio: user space AIO emulated by posix thread workers
- 38. Native AIO • Native aio: kernel AIO • threaded aio: user space AIO emulated by posix thread workers -drive file=win_xp.img,if=none,id=drive_0,cache=none,aio=native -device virtio-blk-pci,drive=drive_0,bus=pci.0,addr=0x5
- 39. 块设备IO调度器 cfq per-process IO queue per-device deadline IO queue 较好公平性 较低aggregate throughput 较好实时性，较好aggregate throughput 不 公平，容易出现VM starvation
- 40. 块设备IO调度器 cfq per-process IO queue per-device deadline IO queue 较好公平性 较低aggregate throughput 较好实时性，较好aggregate throughput 不 公平，容易出现VM starvation sysctl -w sys.block.sdb.queue.scheduler=cfq
- 41. Agenda • CPU • Memory • IO • Storage • Network
- 43. virtio-net • 基于virtio框架的虚拟以太网设备 -netdev type=tap,id=pub226,ifname=pub226,vhost=on,script=up.sh,downscript=down.sh -device virtio-net-pci,netdev=pub226,mac=00:02:dc:04:59:36,bus=pci.0,addr=0xf
- 44. • vhost_net 内核进程vhost_xxx负责tap设备和guest virtio queue之间的数据交换， 少qemu通过用户态和tap设备交换数据的system call和内存拷贝 qemu pid=5489 guest VM virtqueue vnic tap0 vSwitch nic kernel
- 45. • vhost_net 内核进程vhost_xxx负责tap设备和guest virtio queue之间的数据交换， 少qemu通过用户态和tap设备交换数据的system call和内存拷贝 qemu pid=5489 qemu pid=5489 guest VM virtqueue guest VM vnic virtqueue tap0 tap0 vSwitch vnic vSwitch nic kernel nic vhost_5489 kernel
- 46. • vhost_net 内核进程vhost_xxx负责tap设备和guest virtio queue之间的数据交换， 少qemu通过用户态和tap设备交换数据的system call和内存拷贝 qemu pid=5489 qemu pid=5489 guest VM virtqueue guest VM vnic virtqueue tap0 tap0 vSwitch vnic vSwitch nic kernel nic vhost_5489 kernel -netdev type=tap,id=pub226,ifname=pub226,vhost=on,script=up.sh,downscript=down.sh -device virtio-net-pci,netdev=pub226,mac=00:02:dc:04:59:36,bus=pci.0,addr=0xf
- 48. 其他优化选项 • CPU: scheduler • Memory: NUMA • Storage: PCI-passthrough • Network: SR-IOV, PCI-passthrough • 提升硬件指标