In my article vSphere 5.5 Jumbo VMDK Deep Dive I covered the new PDL AutoRemove feature of vSphere 5.5 briefly, including it’s impact on vSphere Metro Stretched Cluster Configurations (vMSC). The reason I’m writing this article is to let you know that there are some impacts to vMSC configurations with the default settings, and in these types of environments you’ll need to modify the defaults.

I had a discussion with Cormac Hogan, Ken Werneburg and others regarding the impact of PDL AutoRemove on vMSC Clusters while at VMworld Barcelona. But before we get to our conclusions lets go over some of the basics.

What is a PDL?

PDL is an acronym for Permanent Device Loss. It is different from the APD or All Paths Down condition in that the device has been administratively removed or suffered a catastrophic hardware failure for one reason or another and the storage will send a SCSI sense code to ESXi telling it as much. In the case of APD the paths to a storage device have failed and the storage device can no longer be contacted for some reason, no sense codes are available. In most cases you’ll want any VM’s running on a PDL’d device to be killed, assuming it was removed for a reason, or in the case of a vMSC cluster that the VM’s will be starting up at the other location. In this case no further IO’s will be sent to the device.

In vSphere 5.0 and 5.1 there were particular advanced settings that you had to change in order to activate the PDL behaviour. These settings apply from vSphere 5.0 Update 1 and 5.1. A ESXi Host Advanced Setting disk.terminateVMOnPDLDefault set to 1 and this setting would terminate the VM’s or kill the VM’s in case of a PDL, and the vSphere HA advanced setting das.maskCleanShutdownEnabled would be set to true. This allows HA to trigger a restart for a virtual machine that has been killed automatically due to a PDL.

Expected PDL Behaviour and APD conditions are covered in VMware KB 2004684 – Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.x and in Duncan Epping’s article Permanent Device Loss (PDL) enhancements in vSphere 5.0 Update 1 for Stretched Clusters.

This brings us nicely onto PDL AutoRemove introduced in vSphere 5.5.

vSphere 5.5 PDL AutoRemove

vSphere 5.5 introduces PDL AutoRemove, which automatically removes a device in a PDL state from a host. A PDL state on a device implies it cannot accept more IOs, but needlessly uses up one of the 256 device per host limit. The new default behaviour will free any PDL devices up from the number of devices per host limit. The reason for doing this is that the devices in PDL state are generally not coming back. This is a great default behaviour for all environments that are standalone or consist of only one location.

Ok, so that sounds like a good thing. But what about in the case of a vMSC configuration?

PDL AutoRemove with vMSC Configurations

Here is what Duncan Epping had to say about it recently on Twitter:

@DuncanYB – If you are upgrading to 5.5 in a vMSC environment, please for now set “Disk.AutoremoveOnPDL” to “0”. So that it is disabled.

Why do we need to change the default I hear you ask? Well it’s quite simple really. In the case of a vMSC environment the PDL’s are likely temporary because one site has become orphaned from the other, in which case a failover is occurred. If the devices in a PDL state are removed permanently when the failure or configuration error of the vMSC environment is fixed they will not automatically be visible to the hosts again. This will require a manual rescan in order to bring the devices back into service. The whole reason for having a vMSC environment is that it handles these types of things automatically. So you don’t want to have to do manual rescans all the time. For this reason the PDL AutoRemove functionality should be disabled on all hosts that are part of a vMSC configuration. Please note that this is recommended for Uniform or Non-Uniform vMSC configurations. Any vMSC configuration that could cause a PDL should have the setting changed.

Duncan Epping also covers this in his article Disable “Disk.AutoremoveOnPDL” in a vMSC environment! Including a link through to KB 2059622 – PDL AutoRemove feature in vSphere 5.5.

Final Word

When considering a vMSC architecture don’t forget to architect for operations of the environment, including updates and changes after it’s gone into production. You need to have a way of testing things non-disruptively to production, including firmware updates, changes to configuration of the underlying components that make up the vMSC. This testing should be done prior to the changes being made to production. You’ll also need a way of testing the vMSC solution functionality periodically, so that it actually works the way you expect it to, prior to a disaster striking. vMSC environments allow fantastic disaster avoidance when architected, implemented and operated correctly and when there is a justified reason for them, which outweighs the complexity. But operations is critical and I often see vMSC solutions implemented without due thought of how they will be operated, managed and tested long term.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.