Most VMware admins know by now that auto tiering storage systems should have SIOC configured differently than if they have storage systems that don’t do auto tiering. Especially if it’s auto tiering at block level. It’s also recommended to have IO metric collection disabled because it will get different results based on where the blocks on the datastore physically reside. What you may not know is that VMware has enabled IO metric collection for every datastore by default with ESXi 6.0. This can cause some weird latency spikes on your storage system that can be very hard to isolate if you don’t know what you’re looking for or what might be causing it. So here is how you fix the problem.

If like a lot of people you still primarily use the C# client for vSphere then you will have to switch to a web browser to fix this problem. This will require you log into the vSphere Web Client. Once you are logged in select Storage -> <Datastore> -> Manage. You should see something similar to the following image:

Click on the Edit button and then click the check box next to Disable Storage I/O statistics collection as in the image below:

Click Ok and repeat the process for your other datastores.

If you would like to do the above more automatically by using a PowerCLI Script, please check out this great blog post by Burke Azbill here.

Final Word

Hopefully the above helps you reduce any unexpected latency spikes on your datastores and storage systems you use with ESXi 6.0. As the number of storage systems with auto tiering and multiple storage types increases over time this becomes even more relevant. Given the trends in the industry it is a wonder VMware made this a default setting in the first place. As storage systems become more intelligent there is less and less need for features such as Storage IO Control, which means admins should be able to focus on other more important things, rather than worrying about noisy neighbor problems. FYI, If you are a Nutanix customer this also applies to you. I found this little gem while running an NCC check on a cluster that I was getting periodic unexplained latency spikes. Needless to say the problem has now been resolved after making this change.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2016 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Yesterday I wrote an article about an apparent conflicting support statement between EMC and VMware titled Storage I/O Control with FAST Auto Tiering Support Statement Conflicts. Less than 8 hours later I have an answer thanks to some great guys at VMware and EMC.

Thanks to one of the great team at VMware – Manish Patel a.k.a. @Mandivs I have got a response from one of the EMC TechBook document authors. It turns out that the reference to SIOC not being supported with FAST is incorrect and will be completely removed from the next version of the document, which is due out in a couple of weeks. It feels so good to be connected to these guys! Here is the response straight from one of the authors of the EMC TechBook that caused me some confusion. This is a quote from Cody Hosterman regarding the TechBook note: “This note is incorrect in the Techbook and is being pulled entirely in an upcoming version. It somehow arose from the fact that SIOC would rarely be useful with FAST VP-enabled devices and in a properly configured environment shouldn’t be necessary at all on a VMAX. But there are some extreme situations where it could help so it is supported. Please disregard the note.”

While researching this problem I had responses from Andrew Mitchell, Duncan Epping and Scott Lowe and I would like to thank them for helping get to the bottom of this.

Please note that this discussion is around Storage I/O Control used in combination with EMC FAST. This is not a discussion around using I/O metrics in Storage DRS in combination with EMC FAST. VMware’s current recommendation is that Storage DRS should be used for initial placement and load balancing based on space utilization only and I/O metrics balancing should be disabled when the arrays are using FAST. In the case with the project I’m working on we are configuring Storage DRS in manual mode and will be using it for initial placement and making load balancing recommendations only, and based on utilization. Any implementation of those recommendations will be manual. This is in line with the recommendations.

For further information regarding Storage DRS interoperability with array features check out this great blog – Storage DRS and Storage Array Feature Interoperability. I would also recommend that you check out this article from Chad Sakac – vSphere 4.1, SOIC, and Array Auto-Tiering.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

I’ve been working on a vSphere 5 Architecture Design for business critical applications for a large financial institution and have run into a conflict in the support statement between VMware and EMC with regard to the support of FAST (Fully Automated Storage Tiering) on VMAX when used with Storage I/O Control. The VMware SAN Compatibility Guide on page 6 says “All the storage devices listed on vSphere 4.1 Storage HCL are supported for use with SIOC. There is no special certification requirements for the SIOC support.” This is where things start to get interesting.

If you read the VMware SAN Compatibility Guide you would think that FAST on VMAX with Storage I/O Control was no problem. Further to this VMware KB 1022091 – Troubleshooting Storage I/O Control says “Before using SIOC on datastores that are backed by arrays with automated storage tiering capabilities, check the VMware Compatibility Guide to ensure that your automated tiered storage array is certified to be compatible with SIOC.” So based on the KB if you checked back with the VMware SAN Compatibility Guide you would still think that Storage I/O Control with FAST on VMAX was no problem. But is it?

However if you visit the EMC site and you look up the latest edition (version 7) of the white paper on how to configure and use Symmetrix Storage with vSphere titled Using EMC Symmetrix Storage in VMware vSphere Environments you will find on page 154 a cautionary note saying “Storage I/O Control is not supported on devices whose datastores are managed by EMC FAST.” This is where some head scratching might start.

I can find no explanation for this apparent conflicting support statement between VMware and EMC on this topic. I am also unable to find any definitive statements in a document or KB from VMware that explicitly mentions Storage I/O Control and EMC FAST or the supportability thereof. I am following up with good contacts inside EMC and VMware to see if I can get to the bottom of this. The word from one of the authors of the EMC TechBook is that this is still the current ‘recommendation’ and although there is a new version being released shortly this advice is not changing. Recommendations are one thing and support is another, so I think it’s important to have absolute clarity on this. Especially when Storage DRS may be involved in the future, even in Manual Mode.

So why would I want to use FAST with SIOC in the first place? Firstly to get the best use of storage between the available tiers in use, FC and SSD in this case. Secondly because the customer requirements state that all VM’s are of equal importance and need equal fairness with regard to access to storage. By using Storage I/O Control it will prevent the noisy neighbor problem where one VM will impact the performance of others. Even with FAST in use this should provide fair access to I/O resources for the VM’s that may be sharing a datastore. SIOC should also reduce the impact of momentary spikes of IO latency if these are observed when FAST is migrating blocks between tiers and still ensuring fairness.

Update (less than 24 hours later): Thanks to one of the great team at VMware – Manish Patel a.k.a. @Mandivs I have got a response from one of the EMC TechBook document authors. It turns out that the reference to SIOC not being supported with FAST is incorrect and will be completely removed from the next version of the document, which is due out in a couple of weeks. This is a quote from Cody Hosterman regarding the TechBook note: “This note is incorrect in the Techbook and is being pulled entirely in an upcoming version. It somehow arose from the fact that SIOC would rarely be useful with FAST VP-enabled devices and in a properly configured environment shouldn’t be necessary at all on a VMAX. But there are some extreme situations where it could help so it is supported. Please disregard the note.”

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.