Windows guest debugging presentation from KVM Forum 2012
8 likes15,943 views
The document provides a comprehensive guide on Windows guest debugging, covering topics such as device management, crash dumps, driver verifier, and debugging tools like WinDbg and DebugView. It includes detailed instructions for configuring various debugging settings, managing drivers, and handling blue screen crashes, along with commands for executing these tasks. The document serves as a resource for troubleshooting and maintaining system stability in a Windows environment.
![NetKVM parameters from
command line
Description Command line Output
List devices netsh netkvm show devices
Names and index for each
device
Restart device netsh netkvm restart [idx]
Should be done after
parameters are changed
Show parameters netsh netkvm show parameters [idx]
List the names of the
parameters
Show detailed
information about
parameter
netsh netkvm show paraminfo [idx]
[param]
Current value of the
parameter
Change parameter
value
netsh netkvm setparam [idx] [param]
[value]](https://image.slidesharecdn.com/windowsguestdebugging-kvmforum2012-130204094652-phpapp02/85/Windows-guest-debugging-presentation-from-KVM-Forum-2012-18-320.jpg)
Windows guest debugging presentation from KVM Forum 2012
- 1. Windows Guest Debugging Yan Vugenfirer, [email protected] Daynix Computing LTD
- 2. Agenda Device manager Logging and DebugView Introduction to BSOD Crash dumps Crash on demand Driver verifier WinDbg Installation and configuration Postmortem debugging !ndiskd Configuring guests Live debugging
- 4. Device Manager Run -> devmgmt.msc Devices and drivers management and installation Scan for HW changes Effective tool to check resource allocations of the devices, drivers versions and different drivers settings.
- 5. Device Manager - Views Change the views of device manager Show non PNP devices
- 6. Device Manager - Views View -> Devices by connection This view can be used to check buses and devices relations
- 7. Device Manager - Views View -> Resources by type Effective to check IRQ sharing On OSes below Vista virtio devices will use legacy PCI interrupts Virtio-serial and ballon use legacy interrupts also on newer OSes.
- 8. Device Manager - Views View -> Resources by connection IO and memory address space with relations between buses and the devices
- 9. Device Manager - Show non- present devices Show devices drivers that are installed but the device is curently not present in the system Very useful for removing crashing drivers from the system From command line: SET devmgr_show_nonpresent_devices=1 START devmgmt.msc
- 10. Device manager - properties Double click on the device Click on “General” tab You can view device location on the bus If the driver is not started you can see the reason in status text-box. For example - no digital signature on x64 OS from Vista and up
- 11. Installing self signed drivers Drivers need to be at least self signed starting from Vista x64 bit To enable test signing: Bcdedit.exe –set TESTSIGNING ON Disable test signing Bcdedit.exe –set TESTSIGNING OFF Install test certificate on the guest Reboot needed after enabling and disabling test siging commands More information on test signing: http://msdn.microsoft.com/en-us/library/bb530195.aspx
- 12. Device Manager - resources Double click on the device Click on “Resources” tab Shows device resources (IRQs, IO and memory ranges)
- 13. Device manager - driver tab Double click on the device Click on “Driver” tab Get driver version View the driver binaries View the versions of driver binaires
- 14. Device manager - details Double click on the device Click on “Details” tab A lot of additional details like: PNP ID of the device, PM policy, location of different device keys in registry and etc
- 15. VirtIO - NetKVM parameters Advanced tab in device manager Change logging levels Enable packet statistics printout Other parameters starting with “TestOnly” prefix Full list: http://www.linux-kvm.org/ page/ Internals_of_NDIS_driver_for_VirtIO _based_network_adapter#List_of_ configurable_parameters
- 16. NetKVM - log levels 0 - Basic configuration and unload trace. Critical errors. 1 - Warnings, corner cases 2 - Network packet trace 3 - More verbose trace of packets 4 - Virtio library, DPC 5 - ISR trace 6 - Registers readwrite.
- 17. NetKVM parameters from command line Part of WHQL requirements to supply command line utility for drivers that expose parameters through “advanced” tab readme.doc - supplied with driver package explains the usage. Integrated with “netsh” Windows networking tool To register (before the first use): rundll32 netkvmco.dll,RegisterNetKVMNetShHelper Unregistration (removal) rundll32 netkvmco.dll,UnregisterNetKVMNetShHelper
- 18. NetKVM parameters from command line Description Command line Output List devices netsh netkvm show devices Names and index for each device Restart device netsh netkvm restart [idx] Should be done after parameters are changed Show parameters netsh netkvm show parameters [idx] List the names of the parameters Show detailed information about parameter netsh netkvm show paraminfo [idx] [param] Current value of the parameter Change parameter value netsh netkvm setparam [idx] [param] [value]
- 19. Debug view and tracing
- 20. DebugView Shows debug space of Windows (kind of dmesg). Can extract debug trace from crash dump if it was running before the crash Can log debug trace during boot. Download from - http://technet.microsoft.com/en-us/ sysinternals/bb896647.aspx
- 21. DebugView - enable output to debug space To enable debug space outputs from Vista and up: Through registry (using regedit): Add “Default” value of the type DWORD equal to 0xFFFFFFFF under “HKEY_LOCAL_MACHINESYSTEMCurrentControlSet ControlSession ManagerDebug Print Filter” Reboot During live debugging ed nt!Kd_DEFAULT_MASK 0xFFFFFFFF
- 22. Debug view - capture kernel Capturing kernel output disabled by default To enable: Capture -> Capture Kernel
- 23. Debug view - log boot Run DebugView Enable boot logging Run DebugView after reboot to collect the logs
- 24. Debug view - crash dump Run DebugView before crash dump creation After crash and reboot - point DebugView to crash location for processing
- 25. Virtio debugging with serial port Uncomment “ENABLE_COM_DEBUG=1” in SOURCES file. Comment out “ENABLE_PRINT_DEBUG” Rebuild the driver On the host side -serial tcp::<port>,server,nowait nc localhost <port>
- 26. Blue screens
- 27. Bug check System crash called “Bug check” Blue screen Crash dump will be saved Ability to debug the bug check if debugger is connected
- 28. Blue screen inside Bug check code Bug check parameters - specific to each bug check Additional information - can be located also below bug check code and parameters Crash dump saving counter
- 29. Common bug checks Bug Check 0xA: IRQL_NOT_LESS_OR_EQUAL Bug Check 0x8E: KERNEL_MODE_EXCEPTION_NOT_HANDLED Bug Check 0xD1: DRIVER_IRQL_NOT_LESS_OR_EQUAL Bug Check 0x9F: DRIVER_POWER_STATE_FAILURE Bug Check 0xC4: DRIVER_VERIFIER_DETECTED_VIOLATION Bug check codes reference: http://msdn.microsoft.com/en-us/ library/hh994433(v=vs.85).aspx
- 30. Crash dumps
- 31. Crash dump creation During blue screen Windows will save crash dump Minidumps are not always useful Also - remove the option to reboot on crash Right click on “My computer” -> “Properties”->”Advanced system settings”->”Startup and Recovery” -> “Settings” Chose “Kernel memory dump” Remove “Automatically restart” The windows on different OSes can be different but the menu names are the same
- 33. Crash dump locations After the crash search in: %windir%memory.dmp %windir%MinidumpMinidump<timestamp>.dmp When you have Windows error message after reboot - do not close it. Check minidump path and copy it.
- 34. Crash dumps From Windows 7 and up - OS will auto-delete large crash dumps To keep crash dumps: Key: HKLMSystemCurrentControlSetControl CrashControl Value: “AlwaysKeepMemoryDump”:DWORD set to 1
- 35. Crash dumps on demand Crash Windows on demand to examine driver status Configure through registry: Key: HKEY_LOCAL_MACHINESystem CurrentControlSetServicesi8042prtParameters, Value: CrashOnCtrlScroll, equal to a REG_DWORD value of 0x01 Key: HKEY_LOCAL_MACHINESystem CurrentControlSetServiceskbdhidParameters, value: CrashOnCtrlScroll, equal to a REG_DWORD value of 0x01
- 36. Crash dumps on demand Press Ctrl+ScrollLock+ScrollLock You will get MANUALLY_INITIATED_CRASH (0x000000E2) bug check. Read more (also how to redefine the shortcuts): http:// msdn.microsoft.com/en-us/library/ff545499.aspx
- 37. Crash dumps on demand (2) Issue NMI from QEMU monitor Setup guest: HKLM/System/CurrentControlSet/Control/CrashControl, DWORD value NMICrashDump 1 Restart Enter QEMU monitor (Alt+Ctrl+2, to go back Alt+Ctrl+1) Issue “NMI” command
- 38. Driver verifier
- 39. Driver verifier Built-in into OS (substitutes OS functions with own layer). To activate run “verifier.exe” from command line Need a reboot after the settings are changed Effective in finding deadlocks, memory leaks, memory corruptions and incorrect parameters to OS functions.
- 40. Driver verifier - activation Run from command line: verifier.exe Select “Create custom settings”
- 41. Driver verifier - activation
- 42. Driver verifier - activation Change selection to “Select driver names from a list”
- 43. Driver verifier - activation Select loaded driver from the list Or select driver from its location on files system After clicking finish - reboot is needed to activate the settings.
- 44. Driver verifier - bug checks 0x000000C4 Some can be ignored if debugger is connected Provide additional information for debugging
- 45. Driver verifier - current settings Run verifier.exe Click on “Display existing settings” radio button Click on “Next” button
- 46. Driver verifier - current settings
- 47. Driver verifier - removal Don’t forget to reboot
- 48. WinDbg
- 49. WinDbg We can examine crash dumps with WinDbg Remote live kernel debugging Limited live kernel debugging using one guest only User mode debugging including remote debugging
- 50. Installation Install as part of the WDK http://msdn.microsoft.com/en-US/windows/ hardware/hh852362 Install as part of the SDK http://msdn.microsoft.com/en-US/windows/ hardware/hh852363 Download symbols for different Windows OSes http://msdn.microsoft.com/en-us/windows/ hardware/gg463028
- 51. Symbols configuration View OS functions and variable names Set Microsoft symbol server and local paths Set environment variable: “_NT_SYMBOL_PATH” Sample symbol path: C:symbolslocal;SRV*C: symbolswebsymbols*http:// msdl.microsoft.com/ download/symbols
- 52. Symbol configuration Another way to set the symbols path is through WinDbg menu File ->”Symbol File Path...” You can instruct WinDbg to reload the symbols Also using windbg command: .reload
- 53. Post mortem debugging Run WinDbg.exe Load dump file through: File -> “Open Crash Dump...” Might be slow first time
- 54. Loading crash dump Information about loaded symbols OS information Hint to use “!analyze -v” Bug check code and parameters
- 55. Magic wand - !analyze -v
- 56. !analyze -v First command that you should run (or click) Bug check and its parameters explanation Hints for the next steps and how those parameters can be used for further debugging Module name that probably caused the crash Stack trace
- 57. WinDbg - help Help -> Contents Switch to “Index” tab for specific command help
- 58. WinDbg - views
- 59. WinDbg - Views View To display window From command line Memory Alt+5 d <address> Call stack Alt+6 kb Disassembly Alt+7 u <address> Registers Alt+4 r Watch Alt+2
- 60. WinDbg - View memory Menu: View -> Memory Command (different variation of “d”): db <address>
- 62. WinDbg - View disassembly
- 63. Windbg - Reference symbols <modulename>!<symbol_name> Example: netkvm!DriverEntry
- 64. WinDbg - Reference symbols
- 65. Useful command Command Description !scsikd Scsi related commands. Use !scsikd.help to find more .reload Reload symbols lm List loaded modules ~<cpu number> Switch between CPUs. For example ~0 !irpfind Search for IO request packets !irp Display information regarding specific IRP .reboot Restart remote during live debugging
- 66. !ndiskd - NDIS debug extensions !ndiskd.ndis Show ndis.sys build information !ndiskd.help Show help !ndiskd.miniports Show NDIS miniports !ndiskd.miniport <addr> Show info about specific mimiport !ndiskd.nbl Show NET_BUFFER_LIST info !ndiskd.oid Show pending OID requests
- 69. From !ndiskd to NetKVM
- 70. Configuring live debugging Host side Run debugget host VM with : “-serial tcp::<port_number>,server,nowait” For example: “-serial tcp::4445,server,nowait” Run target VM with: “-serial tcp: 127.0.0.1:<port_number>”
- 71. Configuring live debugging (2) Host side (new format) Run debugget host VM with : “-chardev socket,id=serial0,path=/tmp/debug- serial0,server,nowait -device isa- serial,chardev=serial0,id=serial0 ” Run target VM with: “-chardev socket,id=serial0,path=/tmp/debug-serial0 - device isa-serial,chardev=serial0,id=serial0”
- 72. Configuring live debugging Guest side - Windows XP and Windows 2003 Edit c:boot.ini Duplicate the default boot line and add: “/debug /debugport=COM1 /baudrate=115200” Reboot
- 73. Configuring live debugging Guest side - Vista and up Use bcdedit.exe bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200 bcdedit /debug ON Reboot Read more: http://www.linux-kvm.org/page/ WindowsGuestDrivers/GuestDebugging
- 74. WinDbg - Connecting to target VM File -> Kernel Debug... Select previously specified serial port and baud rate
- 75. WinDbg - Breaking into target VM Ctrl+C Ctrl+Break Press pause button in the toolbar
- 76. WinDbg - break points Command Description bl List breakpoints bp Set breakpoint bu Unresolved breakpoints (will persist between module reloads) ba Breakpoint on access bc Clear breakpoint Read more: http://msdn.microsoft.com/en-us/library/ windows/hardware/ff552204(v=vs.85).aspx
- 77. WinDbg - stepping through the code Command Description g Start executing the given process or thread t Trace - executes single instruction or source line p Step - executes single instruction or source line. Subroutines are treated as single step. Check different variation and parameters of the above commands
- 78. Saving the dump file Sometimes the system doesn’t save dump file. What to do? Connect WinDbg Use “.dump /f <local_file_name>” to save dump file Wait for it. It will take a long time
- 79. Live debugging on same VM File -> Kernel Debug...
- 80. More links More on debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/ hh406283(v=vs.85).aspx Use the Microsoft Symbol Server to obtain debug symbol files - http:// support.microsoft.com/kb/311503 Setting break points with WinDbg - http://msdn.microsoft.com/en-us/library/windows/ hardware/ff552204(v=vs.85).aspx Working with source files - http://msdn.microsoft.com/en-us/library/windows/hardware/ ff556911(v=vs.85).aspx Live kernel debugging - http://msdn.microsoft.com/en-us/library/windows/hardware/ hh451166(v=vs.85).aspx Debugging techniques - http://msdn.microsoft.com/en-us/library/windows/hardware/ ff541398(v=vs.85).aspx Enabling NDIS debug tracing - http://msdn.microsoft.com/en-us/library/windows/ hardware/ff542961%28v=vs.85%29.aspx
- 81. More tools and links regedit.exe (just in case) Sysinternls tools (home of DbgView) - http:// technet.microsoft.com/en-US/sysinternals Wireshark - http://www.wireshark.org/download.html Extra debugging tools - http://msdn.microsoft.com/en- us/library/windows/hardware/ff543998(v=vs.85).aspx
- 82. Q&A