SlideShare a Scribd company logo
Implementing SR-IOV Failover for
Windows Guests During Migration
Annie Li - Principal Software Engineer, Oracle
Yan Vugenfirer - CEO, Daynix
http://github.com/virtio-win/
Agenda
• Virtio-win drivers
• Windows guest terminology
• The problem
• Different solutions
• Failover solution with virtio-net on Windows
guest
• Upstream: https://github.com/virtio-win/kvm-guest-
drivers-windows/
• Drivers for the major virtio devices:
• virtio-net
• virtio-blk, virtio-scsi
• virtio-balloon, virtio-serial, virtio-vsock, virtio-input,
virtio-rng
• Panic, fw-cfg
• INF files (pci-serial, sm-bus on Q35)
Drivers for Windows
• WDF drivers for the “simple” devices
• Miniport architecture for network and
storage
• NDIS
• Storport
• Scsiport
VirtIO Drivers for Windows
• Supported OS
• Windows XP, Vista, 7, 8, 8.1, 10 (up to
recent builds)
• Widows Server 2003, 2008, 2008R2,
2012, 2012R2, 2016, 2019
VirtIO Drivers for Windows
• Send PRs - https://github.com/virtio-
win/kvm-guest-drivers-windows/
pulls
• Code changes should pass WHQL
• We are running WHQL CI on
upstream (HCK-CI)
How to Contribute
Contributors
• And others
NDIS (Network Driver Interface Specification) Architecture
NotifyObject
Network	
configuration	
subsystem
HW	device HW	device
Protocol	drivers	
Protocol	
Driver
Miniport	Drivers
Network	Device
Miniport	driver
Miniport	edge
Protocol	edge
Intermediate	
drivers
	Protocol	driver
Intermediate	driver
	Protocol	driver
Miniport	driver
NDIS
User	space
• NDIS miniport driver
• Basic driver package:
• INF file – installation description
• SYS file – driver binary
• PDB file – symbols for debugging
• CAT file - package digital signature
Virtio-net (NetKVM) Driver for Windows
The Problem – Live Migration and SR-IOV
Overview of SR-IOV Migration Solutions
• Previous efforts and vendor specific HW
solutions
• Hyper-V and Windows
• Linux and VirtIO
Previous Efforts
• KVM Forum 2015 Live Migration with SR-IOV
Pass-through - Weidong Han, Huawei
• KVM Forum 2018 - Live Migration Support for
GPU with SR-IOV - Zheng Xiao, Alibaba
Cloud; Jerry Jiang & Ken Xue, AMD
• KVM Forum 2020 (parallel session) - Device
Keepalive State for Local Live Migration and
VMM Fast Restart - Jason Zeng, Intel
Overview of Software Solutions
• Windows NIC Teaming
• Windows MUX Intermediate driver
• Hyper-V Solution
Windows NIC Teaming
• Similar to bond in Linux
• Provides failover capability
• Configured through GUI or Powershell
Cmdlets in user space
Windows MUX Intermediate Driver
• Kernel space solution with various models
• One-to-two model for SR-IOV live
migration
Windows MUX Intermediate Driver
TCPIP
VF	miniport	
driver
					
NotifyObject
Open	Adapter1 Open	Adapter2
Network	
configuration	
subsystem
VirtIO	network	
device
Virtual	Function	
network	device
Protocol	driver	
Protocol	
Driver
Miniport	Driver
Network	Device
Virtual	Adapter
NetKVM
Miniport	edge
Protocol	edge
One-to-Two	MUX	
Intermediate	
Driver
Internal	Binding
Other	Protocols
ndisndis
ndis
ndis
ndis
• Netkvm	is	Open	Source	driver	code	for	VirtIO	network	 • VF	miniport	driver	is	provided	by	vendor
NDIS(Kernel	Space)
User	space
Network Binding of NIC Teaming or MUX
Hyper-V VM Network
• Network virtual service client(NetVSC )
• Synthetic data path
• SR-IOV data path
• Two Installation files(INF)
• Share same driver binary
Hyper-V SR-IOV VF Failover
TCPIP Other	
Protocols
VF	Miniport	
Driver
					Hyper-V	NetVSC	Driver
Hyper-V	NetVSC	
Miniport	Driver
Hyper-V	NetVSC	
Protocol	Driver
Miniport	Driver 1:1	
binding
Hyper-V	Virtual	
Network	Device
Virtual	Function	
Device
Virtual	Network	
Device
Protocol	Driver	
Protocol	
Driver
ndisvf
ndisvf
• No	Bond/Teaming	
• No	NotifyObject	
• No	new	Virtual	Adapter
NDIS(Kernel	space)
Network Binding in Hyper-V
Comparison Summary
• MUX Driver model:
• Complicated, New virtual adapter,
Restore offload, NotifyObject.
• Hyper-V model:
• Simplified, Appropriate for Hyper-V
• 2-netdev model in KVM
Windows Network of VirtIO and VF
TCPIP Other	Protocols
VF	Miniport	
Driver
NetKVM	
Miniport	Driver
VirtIO	Network	
Device
Virtual	Function	
Device
Protocol	Driver
Miniport	Driver
Virtual	Network	
Device
ndis
ndis
NDIS(Kernel	sapce)
TCPIP Other	Protocols
VF	Miniport	
Driver
					VirtIO	NetKVM	Driver
NotifyObject
Network	
configuration	
subsystem
Virtual	Function	
Device
Protocol	Driver
Miniport	Driver
Virtual	Network	
Device
1:1	
binding ndis
ndis
ndis
ndis
VirtIO SR-IOV VF Failover
VirtIO	Network	
Device
VirtIO	Protocol	
Driver
NetKVM	Miniport	
Driver
User	Space
NDIS	in	Kernel	Space
• Netkvm	is	Open	Source	driver	code	for	VirtIO	network	 • VF	miniport	driver	is	provided	by	vendor
Protocol Driver in 2-netdev Model
• Behaves like a bridge
• VF adapter is coupled to VirtIO adapter
with the same MAC address
• Handling TX/RX network data
• Object identifiers(OIDs) are wrapped and
forwarded, offloads are propagated
Network Binding of VirtIO SR-IOV
Known Issues
• DHCP issue
• Only happens in specific scenario
• Statistics is missing
• NetKVM driver needs to keep the statistics of packets sent to
or received from VF driver
• Old Windows system
• Windows Server 2003 and Windows XP
• Need to add VF PNP to Notification Object code or to the
registry
• Possible race during boot?
VirtIO and SR-IOV Failover
• VirtIO specification specifics
• Feature bit called
VIRTIO_NET_F_STANDBY. It is
appropriate for 3-netdev model in Linux,
but not for Windows 2-netdev model.
Installation
• Before
• INF for Miniport
• After
• INF for Miniport
• INF for Protocol
driver definition
and Notify Object
• Not part of the guest tools installer: https://github.com/virtio-win/virtio-win-guest-tools-installer
WHQL Certification
• Before
• Certification of
miniport
• Automatic review
• After
• Two steps
certification
• Automatic review for
miniport
• Manual review of
protocol driver and
the final package
Performance
From	VM	to	Remote	Host(NetPerf,	32	vcpus,	32	TCP	Streams,	MTU=9000)
0
25
50
75
100
Default	Default 16k	32k 16k	128k 32k	64k 32k		256k 64k	128k 128k	128k 256k	256k
VF 2-netdev NetKVM
• x	axis:	“TX/RX	Buffer	size”	and	“TX/RX	Socket	Buffer	size” • y	axis:	Network	throughput	from	0	to	100G
Performance
From	Remote	Host	to	VM(NetPerf,	32	vcpus,	32	TCP	Streams,	MTU=9000)
0
25
50
75
100
Default	Default 16k	32k 16k	128k 32k	64k 32k		256k 64k	128k 128k	128k 256k	256k
VF 2-netdev NetKVM
• x	axis:	“TX/RX	Buffer	size”	and	“TX/RX	Socket	Buffer	size” • y	axis:	Network	throughput	from	0	to	100G
Performance
From	VM	to	VM	on	Remote	Host	(NetPerf,	32	vcpus,	32	TCP	Streams,	MTU=9000)
0
25
50
75
100
Default	Default 16k	32k 16k	128k 32k	64k 32k		256k 64k	128k 128k	128k 256k	256k
VF 2-netdev NetKVM
• x	axis:	“TX/RX	Buffer	size”	and	“TX/RX	Socket	Buffer	size” • y	axis:	Network	throughput	from	0	to	100G
annie.li@oracle.com yan@daynix.com
Links – source code
• virtio-win drivers source code:
• https://github.com/virtio-win/kvm-guest-drivers-
windows
• MiniPort and Protocol driver:
• https://github.com/virtio-win/kvm-guest-drivers-
windows/tree/master/NetKVM
• Notification object:
• https://github.com/virtio-win/kvm-guest-drivers-
windows/tree/master/NetKVM/NotifyObject
Links – download binary drivers
• https://docs.fedoraproject.org/en-US/quick-docs/
creating-windows-virtual-machines-using-virtio-
drivers/index.html
Links – related presentations
• KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han,
Huawei
• https://www.youtube.com/watch?v=vnwEnzVp9Zo
• https://www.linux-kvm.org/images/9/9a/03x07-Juniper-Weidong_Han-
LiveMigrationWithSR-IOVPass-through.pdf
• KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao,
Alibaba Cloud; Jerry Jiang & Ken Xue, AMD
• https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live-
Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao-
Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf
• KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live
Migration and VMM Fast Restart - Jason Zeng, Intel
• https://sched.co/eE3W
Implementing SR-IOv failover for Windows guests during live migration

More Related Content

What's hot (20)

PDF
XPDS16: The OpenXT Project in 2016 - Christopher Clark, BAE Systems
The Linux Foundation
 
PDF
LFNW2014 Advanced Security Features of Xen Project Hypervisor
The Linux Foundation
 
PDF
XPDS16: Hypervisor-based Security: Vicarious Learning via Introspektioneerin...
The Linux Foundation
 
PDF
XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - ...
The Linux Foundation
 
PDF
XPDS16: A Paravirtualized Interface for Socket Syscalls - Dimitri Stiliadis, ...
The Linux Foundation
 
PDF
XPDS16: libvirt and Tools: What's New and What's Next - James Fehlig, SUSE
The Linux Foundation
 
PDF
Virtunoid: Breaking out of KVM
Nelson Elhage
 
PDF
XPDS16: CPUID handling for guests - Andrew Cooper, Citrix
The Linux Foundation
 
PDF
XPDS16: Xen Scalability Analysis - Weidong Han, Zhichao Huang & Wei Yang, Huawei
The Linux Foundation
 
PDF
XPDS16: AMD's virtualization memory encryption technology - Brijesh Singh, A...
The Linux Foundation
 
PDF
Evolution of ota_update_in_the_io_t_world
Stefano Babic
 
PDF
XPDS16: Xen Development Update
The Linux Foundation
 
PDF
Kernel Recipes 2015: Kernel packet capture technologies
Anne Nicolas
 
PPTX
Zumasys Citrix Top 10 Tips and Tricks
Frank A. Petillo, Sr.
 
PDF
kdump: usage and_internals
LinuxCon ContainerCon CloudOpen China
 
PDF
Sw update elce2017
Stefano Babic
 
PDF
XPDS16: Hypervisor Enforced Data Loss Prevention - Neil Sikka, A1LOGIC
The Linux Foundation
 
PDF
Rootlinux17: An introduction to Xen Project Virtualisation
The Linux Foundation
 
PDF
XPDS14 - Zero-Footprint Guest Memory Introspection from Xen - Mihai Dontu, Bi...
The Linux Foundation
 
PDF
XPDS16: Xen Live Patching - Updating Xen Without Rebooting - Konrad Wilk, Ora...
The Linux Foundation
 
XPDS16: The OpenXT Project in 2016 - Christopher Clark, BAE Systems
The Linux Foundation
 
LFNW2014 Advanced Security Features of Xen Project Hypervisor
The Linux Foundation
 
XPDS16: Hypervisor-based Security: Vicarious Learning via Introspektioneerin...
The Linux Foundation
 
XPDS16: Xenbedded: Xen-based client virtualization for phones and tablets - ...
The Linux Foundation
 
XPDS16: A Paravirtualized Interface for Socket Syscalls - Dimitri Stiliadis, ...
The Linux Foundation
 
XPDS16: libvirt and Tools: What's New and What's Next - James Fehlig, SUSE
The Linux Foundation
 
Virtunoid: Breaking out of KVM
Nelson Elhage
 
XPDS16: CPUID handling for guests - Andrew Cooper, Citrix
The Linux Foundation
 
XPDS16: Xen Scalability Analysis - Weidong Han, Zhichao Huang & Wei Yang, Huawei
The Linux Foundation
 
XPDS16: AMD's virtualization memory encryption technology - Brijesh Singh, A...
The Linux Foundation
 
Evolution of ota_update_in_the_io_t_world
Stefano Babic
 
XPDS16: Xen Development Update
The Linux Foundation
 
Kernel Recipes 2015: Kernel packet capture technologies
Anne Nicolas
 
Zumasys Citrix Top 10 Tips and Tricks
Frank A. Petillo, Sr.
 
kdump: usage and_internals
LinuxCon ContainerCon CloudOpen China
 
Sw update elce2017
Stefano Babic
 
XPDS16: Hypervisor Enforced Data Loss Prevention - Neil Sikka, A1LOGIC
The Linux Foundation
 
Rootlinux17: An introduction to Xen Project Virtualisation
The Linux Foundation
 
XPDS14 - Zero-Footprint Guest Memory Introspection from Xen - Mihai Dontu, Bi...
The Linux Foundation
 
XPDS16: Xen Live Patching - Updating Xen Without Rebooting - Konrad Wilk, Ora...
The Linux Foundation
 

Similar to Implementing SR-IOv failover for Windows guests during live migration (20)

PDF
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
Jim St. Leger
 
PPTX
Windows server 8 and hyper v
Susantha Silva
 
PDF
PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
iXsystems
 
PDF
RMLL / LSM 2009
Franck_Villaume
 
PDF
Rmll Virtualization As Is Tool 20090707 V1.0
guest72e8c1
 
PDF
Unikernels: Rise of the Library Hypervisor
Anil Madhavapeddy
 
PDF
Deploying vn fs with kubernetes pods and vms
LibbySchulze1
 
PDF
Deploying vn fs with kubernetes pods and vms
LibbySchulze1
 
PDF
Unikernels: the rise of the library hypervisor in MirageOS
Docker, Inc.
 
PDF
Windows server 8 hyper v networking (aidan finn)
hypervnu
 
PDF
Linux on Hyper-V
Tudor Damian
 
PPTX
Windows Server 8 Hyper V Networking
Aidan Finn
 
PDF
Project ACRN hypervisor introduction
Project ACRN
 
PDF
OpenStack - JobShop @Iași, 2016
Alexandru Coman
 
PDF
OpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
eurobsdcon
 
PPTX
Easily emulating full systems on amazon fpg as
RISC-V International
 
PDF
Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...
Odinot Stanislas
 
PPTX
6WINDGate™ - Powering the New-Generation of IPsec Gateways
6WIND
 
PDF
NFV в сетях операторов связи
TERMILAB. Интернет - лаборатория
 
PPTX
5. IO virtualization
Hwanju Kim
 
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
Jim St. Leger
 
Windows server 8 and hyper v
Susantha Silva
 
PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)
iXsystems
 
RMLL / LSM 2009
Franck_Villaume
 
Rmll Virtualization As Is Tool 20090707 V1.0
guest72e8c1
 
Unikernels: Rise of the Library Hypervisor
Anil Madhavapeddy
 
Deploying vn fs with kubernetes pods and vms
LibbySchulze1
 
Deploying vn fs with kubernetes pods and vms
LibbySchulze1
 
Unikernels: the rise of the library hypervisor in MirageOS
Docker, Inc.
 
Windows server 8 hyper v networking (aidan finn)
hypervnu
 
Linux on Hyper-V
Tudor Damian
 
Windows Server 8 Hyper V Networking
Aidan Finn
 
Project ACRN hypervisor introduction
Project ACRN
 
OpenStack - JobShop @Iași, 2016
Alexandru Coman
 
OpenStack and OpenContrail for FreeBSD platform by Michał Dubiel
eurobsdcon
 
Easily emulating full systems on amazon fpg as
RISC-V International
 
Bare-metal, Docker Containers, and Virtualization: The Growing Choices for Cl...
Odinot Stanislas
 
6WINDGate™ - Powering the New-Generation of IPsec Gateways
6WIND
 
NFV в сетях операторов связи
TERMILAB. Интернет - лаборатория
 
5. IO virtualization
Hwanju Kim
 
Ad

More from Yan Vugenfirer (7)

PDF
Receive side scaling (RSS) with eBPF in QEMU and virtio-net
Yan Vugenfirer
 
PDF
Qemu device prototyping
Yan Vugenfirer
 
PDF
Windows network teaming
Yan Vugenfirer
 
PDF
Rebuild presentation during Docker's Birthday party
Yan Vugenfirer
 
PDF
Contributing to open source using Git
Yan Vugenfirer
 
PDF
Introduction to Git
Yan Vugenfirer
 
PDF
UsbDk at a Glance 
Yan Vugenfirer
 
Receive side scaling (RSS) with eBPF in QEMU and virtio-net
Yan Vugenfirer
 
Qemu device prototyping
Yan Vugenfirer
 
Windows network teaming
Yan Vugenfirer
 
Rebuild presentation during Docker's Birthday party
Yan Vugenfirer
 
Contributing to open source using Git
Yan Vugenfirer
 
Introduction to Git
Yan Vugenfirer
 
UsbDk at a Glance 
Yan Vugenfirer
 
Ad

Recently uploaded (20)

PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
Blockchain Transactions Explained For Everyone
CIFDAQ
 
PDF
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
PDF
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
July Patch Tuesday
Ivanti
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
Python basic programing language for automation
DanialHabibi2
 
PDF
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
PDF
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
PPTX
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
Blockchain Transactions Explained For Everyone
CIFDAQ
 
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
July Patch Tuesday
Ivanti
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
Python basic programing language for automation
DanialHabibi2
 
Fl Studio 24.2.2 Build 4597 Crack for Windows Free Download 2025
faizk77g
 
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 

Implementing SR-IOv failover for Windows guests during live migration

  • 1. Implementing SR-IOV Failover for Windows Guests During Migration Annie Li - Principal Software Engineer, Oracle Yan Vugenfirer - CEO, Daynix http://github.com/virtio-win/
  • 2. Agenda • Virtio-win drivers • Windows guest terminology • The problem • Different solutions • Failover solution with virtio-net on Windows guest
  • 3. • Upstream: https://github.com/virtio-win/kvm-guest- drivers-windows/ • Drivers for the major virtio devices: • virtio-net • virtio-blk, virtio-scsi • virtio-balloon, virtio-serial, virtio-vsock, virtio-input, virtio-rng • Panic, fw-cfg • INF files (pci-serial, sm-bus on Q35) Drivers for Windows
  • 4. • WDF drivers for the “simple” devices • Miniport architecture for network and storage • NDIS • Storport • Scsiport VirtIO Drivers for Windows
  • 5. • Supported OS • Windows XP, Vista, 7, 8, 8.1, 10 (up to recent builds) • Widows Server 2003, 2008, 2008R2, 2012, 2012R2, 2016, 2019 VirtIO Drivers for Windows
  • 6. • Send PRs - https://github.com/virtio- win/kvm-guest-drivers-windows/ pulls • Code changes should pass WHQL • We are running WHQL CI on upstream (HCK-CI) How to Contribute
  • 8. NDIS (Network Driver Interface Specification) Architecture NotifyObject Network configuration subsystem HW device HW device Protocol drivers Protocol Driver Miniport Drivers Network Device Miniport driver Miniport edge Protocol edge Intermediate drivers Protocol driver Intermediate driver Protocol driver Miniport driver NDIS User space
  • 9. • NDIS miniport driver • Basic driver package: • INF file – installation description • SYS file – driver binary • PDB file – symbols for debugging • CAT file - package digital signature Virtio-net (NetKVM) Driver for Windows
  • 10. The Problem – Live Migration and SR-IOV
  • 11. Overview of SR-IOV Migration Solutions • Previous efforts and vendor specific HW solutions • Hyper-V and Windows • Linux and VirtIO
  • 12. Previous Efforts • KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han, Huawei • KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao, Alibaba Cloud; Jerry Jiang & Ken Xue, AMD • KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live Migration and VMM Fast Restart - Jason Zeng, Intel
  • 13. Overview of Software Solutions • Windows NIC Teaming • Windows MUX Intermediate driver • Hyper-V Solution
  • 14. Windows NIC Teaming • Similar to bond in Linux • Provides failover capability • Configured through GUI or Powershell Cmdlets in user space
  • 15. Windows MUX Intermediate Driver • Kernel space solution with various models • One-to-two model for SR-IOV live migration
  • 16. Windows MUX Intermediate Driver TCPIP VF miniport driver NotifyObject Open Adapter1 Open Adapter2 Network configuration subsystem VirtIO network device Virtual Function network device Protocol driver Protocol Driver Miniport Driver Network Device Virtual Adapter NetKVM Miniport edge Protocol edge One-to-Two MUX Intermediate Driver Internal Binding Other Protocols ndisndis ndis ndis ndis • Netkvm is Open Source driver code for VirtIO network • VF miniport driver is provided by vendor NDIS(Kernel Space) User space
  • 17. Network Binding of NIC Teaming or MUX
  • 18. Hyper-V VM Network • Network virtual service client(NetVSC ) • Synthetic data path • SR-IOV data path • Two Installation files(INF) • Share same driver binary
  • 19. Hyper-V SR-IOV VF Failover TCPIP Other Protocols VF Miniport Driver Hyper-V NetVSC Driver Hyper-V NetVSC Miniport Driver Hyper-V NetVSC Protocol Driver Miniport Driver 1:1 binding Hyper-V Virtual Network Device Virtual Function Device Virtual Network Device Protocol Driver Protocol Driver ndisvf ndisvf • No Bond/Teaming • No NotifyObject • No new Virtual Adapter NDIS(Kernel space)
  • 21. Comparison Summary • MUX Driver model: • Complicated, New virtual adapter, Restore offload, NotifyObject. • Hyper-V model: • Simplified, Appropriate for Hyper-V • 2-netdev model in KVM
  • 22. Windows Network of VirtIO and VF TCPIP Other Protocols VF Miniport Driver NetKVM Miniport Driver VirtIO Network Device Virtual Function Device Protocol Driver Miniport Driver Virtual Network Device ndis ndis NDIS(Kernel sapce)
  • 23. TCPIP Other Protocols VF Miniport Driver VirtIO NetKVM Driver NotifyObject Network configuration subsystem Virtual Function Device Protocol Driver Miniport Driver Virtual Network Device 1:1 binding ndis ndis ndis ndis VirtIO SR-IOV VF Failover VirtIO Network Device VirtIO Protocol Driver NetKVM Miniport Driver User Space NDIS in Kernel Space • Netkvm is Open Source driver code for VirtIO network • VF miniport driver is provided by vendor
  • 24. Protocol Driver in 2-netdev Model • Behaves like a bridge • VF adapter is coupled to VirtIO adapter with the same MAC address • Handling TX/RX network data • Object identifiers(OIDs) are wrapped and forwarded, offloads are propagated
  • 25. Network Binding of VirtIO SR-IOV
  • 26. Known Issues • DHCP issue • Only happens in specific scenario • Statistics is missing • NetKVM driver needs to keep the statistics of packets sent to or received from VF driver • Old Windows system • Windows Server 2003 and Windows XP • Need to add VF PNP to Notification Object code or to the registry • Possible race during boot?
  • 27. VirtIO and SR-IOV Failover • VirtIO specification specifics • Feature bit called VIRTIO_NET_F_STANDBY. It is appropriate for 3-netdev model in Linux, but not for Windows 2-netdev model.
  • 28. Installation • Before • INF for Miniport • After • INF for Miniport • INF for Protocol driver definition and Notify Object • Not part of the guest tools installer: https://github.com/virtio-win/virtio-win-guest-tools-installer
  • 29. WHQL Certification • Before • Certification of miniport • Automatic review • After • Two steps certification • Automatic review for miniport • Manual review of protocol driver and the final package
  • 30. Performance From VM to Remote Host(NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
  • 31. Performance From Remote Host to VM(NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
  • 32. Performance From VM to VM on Remote Host (NetPerf, 32 vcpus, 32 TCP Streams, MTU=9000) 0 25 50 75 100 Default Default 16k 32k 16k 128k 32k 64k 32k 256k 64k 128k 128k 128k 256k 256k VF 2-netdev NetKVM • x axis: “TX/RX Buffer size” and “TX/RX Socket Buffer size” • y axis: Network throughput from 0 to 100G
  • 34. Links – source code • virtio-win drivers source code: • https://github.com/virtio-win/kvm-guest-drivers- windows • MiniPort and Protocol driver: • https://github.com/virtio-win/kvm-guest-drivers- windows/tree/master/NetKVM • Notification object: • https://github.com/virtio-win/kvm-guest-drivers- windows/tree/master/NetKVM/NotifyObject
  • 35. Links – download binary drivers • https://docs.fedoraproject.org/en-US/quick-docs/ creating-windows-virtual-machines-using-virtio- drivers/index.html
  • 36. Links – related presentations • KVM Forum 2015 Live Migration with SR-IOV Pass-through - Weidong Han, Huawei • https://www.youtube.com/watch?v=vnwEnzVp9Zo • https://www.linux-kvm.org/images/9/9a/03x07-Juniper-Weidong_Han- LiveMigrationWithSR-IOVPass-through.pdf • KVM Forum 2018 - Live Migration Support for GPU with SR-IOV - Zheng Xiao, Alibaba Cloud; Jerry Jiang & Ken Xue, AMD • https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Live- Migration-Support-for-GPU-with-SRIOV-Challenges-and-Solution-Zheng-Xiao- Alibaba-Cloud-Jerry-Jiang-Ken-Xue-AMD.pdf • KVM Forum 2020 (parallel session) - Device Keepalive State for Local Live Migration and VMM Fast Restart - Jason Zeng, Intel • https://sched.co/eE3W