I’ve started to see reports recently of I/O errors when running very high I/O workloads on Windows 2008 and Windows 2008 R2 VM’s. Mostly this was during artificial benchmark tests run against MS SQL and Exchange 2010 with Jetstress. However it could impact production workloads. Upon further investigation it appears these I/O errors are a known defect with a certain version of the PVSCSI driver that comes with VMware Tools and can affect vSphere 4.0 U1, 4.1 and 5.0. Here I’ll cover more about this potentially serious issue and how to fix it.
This problem is described in VMware KB 2004578 – Windows 2008 R2 virtual machine using a Paravirtual SCSI adapter reports the error: Operating system error 1117 encountered along with the versions of the PVSCSI driver that are impacted and a link to the fix. Microsoft has also included a knowledge base article on their site with regard to this, refer to MS KB 2519834 – SQL Server reports “Operating system error 1117 (I/O Device Error)” on VMware ESX environments that are configured to use PVSCSI adapters.
Although the referenced KB articles describe a situation with SQL it is possible for this to happen under any high I/O workload on the impacted versions of Windows, including for example Exchange. The information available right now doesn’t mention Windows 7 VM’s. But Win7 VM’s are generally less susceptible to the same high I/O workloads as Exchange and SQL servers. Even though Win7 VM’s are less susceptible to the same load conditions that would cause this issue the PVSCSI driver in Win7 is still affected by this problem and should be updated. In the case of VDI desktops could be re-provisioned if they experienced this issue.
What makes this issue potentially serious is that in the worst case (rare) scenario this problem could lead to data corruption. This makes it very important that you upgrade or patch your vSphere environment to address this defect. With ESXi 5.0 the patch is included with Update 1. For ESXi 4.1 you should deploy Patch 04 described in VMware KB 2009144 – VMware ESXi 4.1 Patch ESXi410-201201402-BG: Updates VMware Tools.
On completion of the ESXi Patch process, you will be required to update VMware Tools (System Restart Required) on both required VMs and Templates. You should then confirm the PVSCSI StorPort device driver version has been updated to 220.127.116.11 or later. For more information about the VMware Paravirtual SCSI adapter and supported VMs, refer to VMware KB 1010398 – Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.