If you are a Zerto customer they have released a very critical patch that fixes an IO path bug where SCSI sense codes were modified before responses were sent back to guest VM’s. Un-patched, the bug could result in data integrity issues. This has been seen in the field, particularly with Microsoft SQL and Oracle databases where the storage is under constant load and where a storage controller upgrade was being performed online. So it’s very imperative that you review the Zerto release notes and upgrade to Zerto 5.0 U2 ASAP.

The underlying problem was related to SCSI sense codes that the storage system was sending to the guest OS during certain storage controller failover operations. The bug prevented proper processing of the SCSI sense code (the code was being modified and truncated by Zerto software), which erroneously resulted in the OS thinking a write operation had been completed when in fact it had not. This issue was seen on Nutanix systems due to the frequency that storage controllers (CVM’s) are updated. The Nutanix business critical apps team was able to reproduce the issue and worked with Zerto to get the problem reproduced and corrected. We have confirmed Zerto 5.0 U2 has resolved the issue. Zerto is a great partner, and they were extremely helpful in resolving this issue and coordinating with their impacted customers.

I’ve personally been involved in a number of very successful large scale (many PB of data) projects where Zerto has been used for migration and DR and have not experienced this issue. I’ve also been involved in helping recover from this issue for a number of impacted customers. It’s good to know that when found this issue was fixed and assistance provided to impacted customers jointly by Zerto and Nutanix.

Derek Seaman was first to write about this issue being resolved here.

Here is an image of the Zerto Release Notes. Although it mentioned Nutanix in the notes any storage system could potentially be at risk due to the way the bug was caused. The full release notes for Zerto Replication 5.0 U2 are found here. 

With all of the excitement around vSphere 5.1 last year most of us had forgotten to mention the little 2012 Christmas present that VMware left for customers that are still running vSphere 5.0. What I’m referring to is vSphere 5.0 Update 2. Now that Partner Exchange in Las Vegas is over I’ve had time to revisit this release and the importance of it for customers running business critical apps.

There are a number of important defect fixes and enhancements in vSphere 5.0 Update 2 and vCenter 5.0 Update 2. I won’t go through all of them. But I do recommend you read the release notes: ESXi 5.0 Update 2 Release Notes, vCenter 5.0 Update 2 Release Notes, vCenter Update Manager 5.0 Update 2 Release Notes.

The top 10 fixes or enhancement highlights (in no particular order) for the vSphere 5.o Update 2 release from my perspective are as follows:

1. Fixed: Long running vMotion operations might result in unicast flooding. When using the multiple-NIC vMotion feature with vSphere 5, if vMotion operations continue for a long time, unicast flooding is observed on all interfaces of the physical switch. If the vMotion takes longer than the MAC address table ageing-time, the source and destination host start receiving high amounts of network traffic. (MW: If you have Monster VM’s this one is particularly important as you need all the Multi-NIC vMotion Bandwidth you can get to ensure quickest migration and maintenance mode times. This fixes the problem I wrote about in my article The Good, The Great, and the Gotcha with Multi-NIC vMotion in vSphere 5 and removes the need for the Workaround for Multi-NIC vMotion Unicast Flooding in vSphere 5).

2. Fixed: Large number of UDP packets are dropped when you use the VMXNET3 adapter. Large number of UDP packets are dropped when you use the VMXNET3 adapter with a Linux guest operating system installed on an ESXi 5.0 host.

3. Support for Windows Server 2012, Windows 8, RHEL 6.2 and RHEL 6.3 customization.

4. Fixed: vCenter Server might stop responding when there is excess memory consumption. When a client connects to vCenter Server, and if it creates an extra PropertyCollector using the vmodl.query.PropertyCollector.createPropertyCollector vCenter API, vCenter will not deallocate the memory for this extra PropertyCollector and its associated objects even after the client logs out. This will result in vCenter Server memory leak. When vCenter Server memory utilization level exceeds 10GB, vCenter Server goes out of memory and stops responding.

5. Fixed: Intermediate certificate is unable to configure and load in vCenter Server 5.0. In vCenter Server, after installing the intermediate certificate chain, the following security warning is displayed every time you try to log in – “Invalid certificate”.

6. Fixed: The network bandwidth is not shared fairly among virtual machines of a network resource pool. The network bandwidth is not allocated fairly to all the virtual machines of a network resource pool due to limitations in the current implementation.

7. Fixed: When you disable coalescing on ESXi, the host fails with a purple screen. In ESXi, when VMXNET3 is used as vNIC in some virtual machines and you turn off packet coalescing, the ESXi host might fail with a purple screen as the virtual machine is booting up.

8. Fixed: Adding a new hard disk to a virtual machine that resides on a Storage DRS enabled datastore cluster might result in Insufficient Disk Space error. When you add a virtual disk to a virtual machine that resides on a Storage DRS enabled datastore, if the size of the virtual disk is greater than the free space available in the datastore, SDRS might migrate another virtual machine out of the datastore to allow sufficient free space for adding the virtual disk. The Storage vMotion operation completes but the subsequent addition of virtual disk to the virtual machine might fail and an error message similar to the following might be displayed: Insufficient Disk Space.

9.  Fixed: Applying a host profile in vCenter Server 5.0 fails with the error. A general system error occurred: In vCenter Server, when you apply a host profile in vCenter Server 5.0 it fails with the following error – “A general system error occurred”.

10.  ESXi 5.0 host fails due to a world slot memory leak. When the ESXi host attempts to create a world group heap without releasing the memory associated with world slot, the process fails due to a world slot memory leak.

Note: If you’re upgrading from vSphere 4.x and you made use of the VMware VMI Paravirtualization Interface that you will need to review VMware KB 1013842. vSphere 4.1 was the last release to support the VMI paravirtualiztion interface. I know I have at least one large customer with hundreds of VM’s in this category.

Final Word

If you’re already on vSphere 5.0 it is very easy to upgrade to vSphere 5.0 Update 2 using vCenter Update Manager. I would highly recommend you do the upgrade, of course following due process and having it go through your dev / test environments first prior to product. But the benefits of the upgrade are very clear. Especially for environments that are running Business Critical Applications such as Oracle, SQL Server, SAP, Exchange, and other major enterprise applications. As always feedback is welcomed.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.