VMware Performance Troubleshooting
- 1. VMware Performance TroubleshootingPresented by Chris Kranz
- 2. Topics CoveredIntroductionRoot Cause AnalysisPerformance CharacteristicsCPUNetworkingMemoryDiskVirtual Machine optimisationESXTopvm-supportService ConsoleResource GroupsDesign GuidelinesCapacity Planner limitations and cautionsConclusionReference Articles
- 3. IntroductionMultiple layers of virtualisation are used to increase service levels, availability and manageabilityHowever, multiple layers of virtualisation often mask performance and configuration issues making it more of a challenge to troubleshoot and correctThe worst out come is that performance issues after a virtualisation project lead to the perception that VMware results in reduced performance and future confidence in VMware can be affected
- 6. Typical HostTypical 3 VMs per core, 24VM’s per HostEach has 2GB of RAM = 48GB of RAM
- 9. Do not rely on guest tools, butCan show high CPU, & Memory UtilisationMeasurement of Latency & throughput of Disk & Network InterfacesUse the virtualisation layer, to diagnose cause:Guest is unaware of virtualisation workloadThe way in which guest OS’s account time is differentNo visibility of available resourcesMonitoring Performance
- 10. esxtop (service console only)resxtop (remote command line utilities)Performance graphs in vCentrePerformance Analysis Tools
- 11. esxtop can be run:Interactively Batch (eg. esxtop -a -b > analysis.csv)Load batch into windows perfmon or MS ExcelTwo keys to rememberH : helpF : fields to displayesxtop
- 12. esxtop basicsHost ResourcesName of Resource Pool, Virtual Machine or WorldNumber of Worlds
- 13. Performance CharacteristicsCPUNetworkingMemoryDiskSlow ProcessingHigh CPU WaitPacket LossSlow NetworkSlow ProcessingDisk SwappingLog StallsDisk QueueSlow Application PerformanceReduced User ExperienceData Loss and Corruption
- 14. CPUESX SchedulerBasic World StatesRead / Run / WaitCPU StatesReady / Usage / WaitServiceConsoleVirtualMachineLimits / Shares / Reservations
- 15. CPUHigh %RDY + High %User can imply over commitmentesxtopPCPU(%): CPU utilization
- 17. %RDY: Ready Time
- 18. %RUN: Run Time
- 19. %WAIT: Wait and idling timeCPUVI-ClientUsed Time > Ready Time: Possible CPU over-committmentUsed TimeReady Time
- 20. CPUFurther Investigation%MLMTD shows this VM has been limited
- 21. CPUFurther InvestigationHigh ready time caused by CPU resource limit
- 22. VMware Memory Management Transparent Page Sharing
- 23. VMware Tools Balloon Driver to force the VM to swap to disk
- 24. Virtual Machine Page FileMemoryBallooning vs. SwappingBallooning driver causes the host to swap pages that it chooses to diskESX Swapping will swap any pages to disk.
- 25. Ballooning can be disabled (0 value) or controlled on a per Virtual Machine basis using:sched.mem.maxmemctlDefault is set to 65%, can be controlled at host level.Only is an issue in resource contention scenarios. (or VM’s with low latency eg Citrix)Memory
- 26. Memory - HostVI Client shows memory usage of the host. This is calculated as “consumed + overhead memory + Service Console”.Performance charts are a very good way of showing the Virtual Machine memory breakdown. Consumed Memory
- 27. Ballooned Memory
- 28. Shared Memory
- 29. Swapped MemoryMemory - GuestHost Memory = Consumed + Overhead MemoryGuest Memory = Active Memory for Guest OS
- 30. Memory – Guest Overhead
- 31. MemoryVirtual Machine Memory Metrics – VI Client
- 32. MemoryHost Memory Metrics – VI Client
- 33. MemoryPMEM: Total physical memory breakdownVMKMEM: Memory managed by vmkernelCOSMEM: Service Console memory breakdownPSHARE: Page sharing statisticsSWAP: Swap statisticsMEMCTL: Balloon driver dataesxtop
- 34. Memoryesxtop / VI Client metrics : Virtual Machines
- 35. Memoryesxtop / VI Client metrics : Host Usage
- 36. MemoryVI Client memory usage graph
- 37. MemoryTroubleshooting Memory usage issues
- 38. NetworkingSwitch Assisted Teaming (IP Hash)
- 39. VLAN Trunking
- 41. Speed & Duplex (1000Mb / Full)
- 42. Port Fast
- 43. BPDU Disabled
- 44. STP Disabled
- 46. Jumbo FramesNetwork configuration is more likely to blame than resource contention
- 47. NetworkingesxtopTransmit and Receive in Mb/sTransmit and Receive in Packets
- 48. NetworkingesxtopDropped Packets TransmitDrop Packets Received
- 49. DiskVarying Factors File system performance
- 50. Disk subsystem configuration (SAN, NAS, iSCSI, local disk)
- 51. Disk caching
- 52. Disk formats (thick, sparse, thin)ESX Storage StackDifferent latencies for different disks
- 53. Queuing within the kernelK: KernelD: DeviceG: Guest
- 54. DiskVI Client statisticsQuite Coarse Statistics Disk read / write rate (KB/s)
- 55. Disk usage: sum of read BW and write BW (KB/s)
- 56. Disk read / write requests (per 20s interval)
- 57. Bus resets / Command aborts (per 20s interval)
- 58. Per LUN or aggregated statsDiskesxtop statisticsAggregated stats similar to VI Client Disk read / write per sec (READS/s, WRITES/s)
- 59. MB read / write per sec (MBREAD/s, MBWRTN/s)Latency Statistics Kernel Average / command (KAVG/cmd)
- 60. Device Average / command (DAVG/cmd)
- 61. Guest Average / command (GAVG/cmd)Queuing Information Adapter Queue Length (AQLEN)
- 62. LUN Queue Length (LQLEN)
- 63. VMKernel (QUED)
- 65. %Used (%USD = ACTV/LQLEN)DiskSAN Rough EstimatesPurely looking at a single ESX host, roughly:Throughput (in MBps) = (Outstanding IOs * Block size in KB) / latency in msecFC, rough maximums:Effective Link Bandwidth = ~80/90% of Real BandwidthEffective (2Gbps) = 200 – 230 MBpsEffective (4Gbps) = 410 – 460 MBpsEffective (8Gbps) = 820 – 920 MBpsiSCSI / NFS / FCoE, rough maximums:Effective Link Bandwidth = ~70/80% of Real BandwidthEffective (1GigE) = 90 – 100 MBpsEffective (10GigE) = 900 – 1000 MBps
- 66. DiskDesired Latency CalculationsDesired Larency in msec <= (Outstanding IOs * Block size in KB) / Throughput per hostExample:Number of Hosts: 16Effective Link Bandwidth: 90 MBpsThroughput per host: 90 / 16 = 5.6 MBpsDesired Latency: (32 * 32) / (5.6) = 182.86 msec
- 67. DiskSAN Cache enabledHigh throughputVI ClientSAN Cache disabled Poor throughput
- 68. DiskesxtopLatency is quite highAfter enabling cache,Latency is reduced
- 69. Virtual Machine OptimisationDeploy all machines from an optimised template! VMware tools MUST be installed
- 70. The disks MUST be block aligned to the storage (even when using NFS and SAN)
- 71. Where possible, always separate data disks from OS disks
- 72. Windows performance settings should be optimised for application performance
- 73. Guest operating system timeouts should be set as defined by the SAN vendor
- 74. Pagefileshould be separated where appropriate (this can impact VMware SRM however)
- 75. Unused Windows services should be disabled (wireless config, print spooler, audio, etc.)
- 76. Last access update time should be disabled (unless where required)
- 77. Logging of the VM should be disabled (only enabled for troubleshooting)
- 78. Remove any unused virtual hardware (floppy drives, USB, etc.)
- 79. Disable screen savers and power saving features, including logon screen saver
- 80. Enable Remote Desktop, avoid using the VI Client for remote administration
- 81. Install standard applications into template (bginfo, AntiVirus, any host agents, etc)
- 82. Multiple-CPU’s should be allocated sparinglyVirtual Machine OptimisationBlock alignment is vital to good disk performance!
- 83. esxtopCommand Options when inside esxtop
- 84. esxtopCommand Line Optionsfrom the console
- 85. esxtopExpand the default window size for your session to get all statistics
- 86. vm-supportCreates a packaged zip file containing the following sections: boot
- 87. contains the grub configuration
- 88. etc
- 89. contains the Console OS configuration files (cron, tcpwrappers, syslog, etc)
- 90. proc
- 91. contains much of the hardware configuration modules and variables
- 92. tmp
- 93. contains a lot of the ESX specific configuration output
- 94. var
- 95. contains log files and any core dumps
- 96. vmfs
- 97. contains the structure of the VMFS datastores
- 98. esx3-installation (where appropriate)
- 99. contains a copy if the previous esx3 configuration variablesvm-supportUsing vm-support to extract performance information:vm-support –S –d <duration> -i <interval><duration> and <interval> are in secondsThe output from this can then be replayed in esxtop for review after it has been extracted.esxtop –R <path_to_vm-support_output>
- 100. Service Console PerformanceMultiple Service Console networks – for network resiliency
- 101. Increased Service Console memory – upto 800MB
- 102. Use host agents supplied by your vendors
- 103. Make storage recommended tweaks such as HBA Queue Depth and IO timeouts
- 104. Minimal use of the VI Client console – RDP or SSH instead
- 105. Properly sized vCenter server – 64bit OS where possibleResource GroupsDynamically reallocate resource sharesAdditional VM, shares allow you to over-commit resources and have a graceful re-allocationRemove a VM and exploit extra resources across all remaining VM’s
- 106. Design Guidelines Full Resilience / Multiple paths
- 107. Standard configuration across all aspects (ESX, Storage, Networking, etc.)
- 109. Learn from others mistakes
- 110. Follow guidelines from vendors best-practices
- 111. Rule out the basics before requesting supportCapacity Planner & P2V Cautions and Limitations Peak CPU usage can sometimes be misleading
- 112. Back-end storage system performance
- 113. P2V machines will require block-aligning to the storage
- 114. P2V machines will still require guest OS optimisationConclusion Performance issues can often be traced with simple root cause analysis using basic tools (VI Client / esxtop)
- 115. Performance tools help diagnose issues and help rule out non-issues
- 116. Performance tools are useful in different contexts, not always either/or
- 117. Real-time data and troubleshooting: esxtop
- 118. Historical data: VI Client
- 119. Coarse resource / cluster usage: VI Client
- 120. Detailed resource usage: esxtop
- 121. Combine information from various tools to get a complete picture
- 122. Always benchmark your systems first so you not what the optimal performance is that you can receiveReference Articleshttp://www.vmware.com/pdf/esx3_memory.pdfhttp://www.vmworld.com/docs/DOC-2370http://blogs.vmware.com/performance/http://communities.vmware.com/docs/DOC-5420http://kb.vmware.com/kb/1008205http://communities.vmware.com/community/vmtn/general/performancehttp://www.vmware.com/products/vmmark/http://www.vmware.com/pdf/vsphere4/r40/vsp_40_san_cfg.pdfhttp://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdfhttp://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdfhttp://www.vmware.com/pdf/GuestOS_guide.pdfhttp://www.vmware.com/resources/techresources/10066http://www.vmware.com/resources/techresources/10059http://www.vmware.com/resources/techresources/10062