There is a lot of FUD about Data Corruption, Torn IO, Write Ordering and other aspects when using NFS as a datastore in VMware vSphere, even when the VM’s are configured to use Virtual Disks. This seems surprising, especially given some very large VMware vSphere based clouds are built on NFS storage presented as datastores for use with VM’s, and that for years numerous companies have been running business critical apps on NFS, presented as datastores, or otherwise. Many of you may not know that VMware has actually patented the process for presenting NFS as a datastore to VM’s that use Virtual SCSI disks (US7865663), so that it emulates the SCSI protocol. You also may not know that not all storage systems, even when using block based storage such as FC, FCoE or iSCSI, honour all of the techniques to keep your data safe. A lot of it comes down to individual storage system implementation. Enterprise storage systems that take data protection seriously and implement the appropriate IO protections are all suitable for running business critical apps, even when presenting NFS for use as a datastore to VMware vSphere. So what do you need to know?

Fortunately in this fight against the FUD, Josh Odgers (www.joshodgers.com) has done all the hard work and research and written some excellent articles on this topic. He’s also working with VMware to have knowledge base articles either written or updated. But here are the highlights and then links to all of Josh’s articles.

Fight the FUD: NFS as a Datastore with VMware vSphere

1. Emulation of the SCSI Protocol

When using NFS as a datastore VMware emulates the SCSI protocol including resets and aborts. This is a complete abstraction and the virtual machines using virtual SCSI disks have no knowledge of the underlying storage. The protocol retains it’s IO integrity, regardless of how the underlying storage is presented. This is a patented and thoroughly tested process.

2. Forced Unit Access and Write Through

Forced Unit Access and Write Through ensures that IO’s are written to persistent media and that when a write is acknowledged it is actually written and protected. This is honoured in the VMware vSphere hypervisor, but is also the responsibility of underlying storage systems. Not all storage systems honour this requirement completely. Data loss can result if the IO’s are not persistently written. This can be a problem for any storage system, regardless if it is presenting block storage or NFS. Every system is different and implements data protection in different ways. If you are running business critical apps on your storage systems then you should ask your vendor how they meet this requirement and protect your data.

3. Write Ordering

Writes must be written in the order they are sent. Otherwise you can end up with data corruption where data is read that wasn’t written, or overwritten because of a write out of order. VMware specifically covers write ordering in KB 1012143 and says ” Write ordering and write-through integrity for NFS storage are both satisfied with NFS in an VMware ESX environment. An NFS datastore, when mounted on an ESX host, goes through virtual SCSI emulation. A virtual machine disk (VMDK) file on an NFS datastore appears as a SCSI disk within the virtual machine’s guest operating system, which is no different than one residing on a VMFS volume over FCP or iSCSI protocol. Therefore, write ordering and write-through integrity are no different than those with block based storage (such as iSCSI or FC protocol).”

4. Torn Writes

Torn writes are where only part of a multi-sector update are written successfully to disk. If there is a problem during a multi-sector write operation, and the write is acknowledged, it can result in silent data corruption. Storage systems implement torn write protection in different ways. The important thing to note is that this problem is related to the underlying storage system implementation and not the storage protocol or the hypervisor. Therefore there is as much risk on block based storage as there is on NFS and it comes down to your vendors data protection implementation. How torn writes are protected against is something you should know for your storage system if you are the virtualization admin or the storage admin. Not all storage systems are created equal when it comes to data protection and IO integrity with regard to torn writes. But this is not a problem caused by VMware vSphere, or using NFS as the datastore.

5. Data Corruption

Data corruption can occur for a number of reasons and depends on the underlying storage system implementation. Corruption handling mechanisms in your storage systems are essential. Corruption handling via sector and ideally block based checksums is essential on writes. Using a checksum on Read helps detect corrupted data. Backup / Recovery solutions are essential. Enterprise grade storage solutions which use checksums to verify data integrity on write and reads have a much lower risk of data corruption, regardless of media type and storage protocol. JBOD style deployments using SATA drives have a significantly higher risk of data corruption. See Josh’s article below for all of the data based on a study of over 1million disks in production.

Josh Odgers Fights the FUD of VMDK’s on NFS Datastores:

Part 1 – Emulation of the SCSI Protocol
Part 2 – Forced Unit Access (FUA) & Write Through
Part 3 – Write Ordering
Part 4 – Torn Writes
Part 5 – Data Corruption

Final Word

This article focused on important storage requirements that protect the integrity of IO and your data when it’s in flight and at rest. This was with regards to Virtual Machines that are using SCSI Virtual Disks where the underlying storage is presented to the VMware vSphere hypervisor as NFS, not where the Virtual Machine directly accesses an NFS system. Many of the requirements come down to individual vendor implementation of their storage systems, regardless of the protocol being used. VMware operates a storage system certification process for a reason, to reduce the risk to you, and it ensures that NFS is properly implemented for systems that use it with VMware vSphere. You can have confidence that using NFS as a datastore with VMware vSphere is suitable for even business critical applications such as Oracle, SQL Server and Exchange. If you’re a Nutanix customer then Nutanix will fully support you when running any application where NFS is presented to VMware vSphere as the datastore, and this includes Exchange.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Just wanted to give my readers a heads up that there appears to be a bug with vSphere 5.5 U1 that impacts any NFS connected storage and can cause random disconnects. The timing of this is a coincidence that it comes right on the coat tails of my previous article Hardware Fails, Software Has Bugs and People Make Mistakes – Usually You Get All At Once! During the disconnects VM’s will appear frozen and the NFS datastores may be greyed out. This appears to impact all vendors and all environments on 5.5. U1 accessing NFS. UPDATED: There is a public KB on this problem from VMware: KB 2076392 – Frequent NFS APDs after upgrading ESXi to 5.5 U1, and VMware is working on it. According to the KB you may experience blue screens of death in Windows, and read only file systems in Linux. I’ve also experienced kernel panics and reboots in Linux as a result of this bug.

NetApp was first to report it. The recommendation from most vendors (Including Nutanix as per Field Advisory FA-17) at this point is to not upgrade to vSphere 5.5 U1 and stay on vSphere 5.5 GA. If you have upgraded to 5.5 U1 then you may need to downgrade back to 5.5GA. I have tested and been able to reproduce this problem on vSphere 5.5 U1 on two completely different vendors systems.

For further information you can see Datacenter Dude Nick Howell’s article – NFS Disconnects in VMware vSphere 5.5 U1, and keep an eye out on the VMware knowledge base article. It is recommended you subscribe to the KB article so that you are made aware of any updates.

[Updated 12/06/2014] There is an updated KB article regarding the NFS Disconnection Bug – KB 2076392, and VMware has issued a patch to correct the problem. vSphere 5.5 Patch 04. It is strongly recommended that you apply patch 04 if you use NFS and wish to leverage the fixes and improvements in vSphere 5.5 Update 1. 

If you’re using block based storage, i.e. VMFS, and a EMC VMAX then you might want to read Cormac Hogan’s article with regard to a situation with VMAX, VAAI and the Unmap command.

—

This post appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

A group of interested parties from a cross section of industry vendors has started a discussion around the merits of supporting Exchange Server on VMDK’s on top of NFS. Technically there is no reason why it should not be supported. It works great and Exchange and Windows don’t even know they’re virtual. They just talk SCSI to their virtual disks. Supportability might be a concern so having some sort of a certification or QA program is important. But provided robust, performant and reliable storage systems are used and Exchange continues to use block SCSI protocol to a VMDK why does it matter that the underlying protocol to the physical storage is NFS? I believe this provides choice to customers, and in fact many customers have been choosing this for a long time and running Exchange like this in production. They’ve continued to get support under their premier agreements. But I believe this should be extended to all customers for any qualified storage platforms. If you agree and you want to see choice of protocol provided the abstraction is solid and proven then support the movement. Vote Exchange on VMDK on NFS!

—

This post appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.