Most VMware admins know by now that auto tiering storage systems should have SIOC configured differently than if they have storage systems that don’t do auto tiering. Especially if it’s auto tiering at block level. It’s also recommended to have IO metric collection disabled because it will get different results based on where the blocks on the datastore physically reside. What you may not know is that VMware has enabled IO metric collection for every datastore by default with ESXi 6.0. This can cause some weird latency spikes on your storage system that can be very hard to isolate if you don’t know what you’re looking for or what might be causing it. So here is how you fix the problem.
If like a lot of people you still primarily use the C# client for vSphere then you will have to switch to a web browser to fix this problem. This will require you log into the vSphere Web Client. Once you are logged in select Storage -> <Datastore> -> Manage. You should see something similar to the following image:
Click on the Edit button and then click the check box next to Disable Storage I/O statistics collection as in the image below:
Click Ok and repeat the process for your other datastores.
If you would like to do the above more automatically by using a PowerCLI Script, please check out this great blog post by Burke Azbill here.
Final Word
Hopefully the above helps you reduce any unexpected latency spikes on your datastores and storage systems you use with ESXi 6.0. As the number of storage systems with auto tiering and multiple storage types increases over time this becomes even more relevant. Given the trends in the industry it is a wonder VMware made this a default setting in the first place. As storage systems become more intelligent there is less and less need for features such as Storage IO Control, which means admins should be able to focus on other more important things, rather than worrying about noisy neighbor problems. FYI, If you are a Nutanix customer this also applies to you. I found this little gem while running an NCC check on a cluster that I was getting periodic unexplained latency spikes. Needless to say the problem has now been resolved after making this change.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2016 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Hello Michael!
Thnx for the useful article. Any suggestion why that happens? I mean latency spikes in storage system.
Interesting….I just checked my hosts and we have this enabled as well ( unchecked ). However we are running ESXi 5.5 and vCenter 6.0.
Thanks for the feedback. Unless you need it, might be a good idea to disable it.
Any ideas on how to disabled this via Powershell or Cli? Trying to find the right command to use.
Unfortunately disabling via PowerCLI doesn\’t seem to be trivial. I haven\’t been able to find an easy command to use either.
Hi Shaun, I've just updated the post with a link to how to disable stats collection using PowerCLI.
Maybe I’m misunderstanding things but are you saying having the stats on affects performance of running vms or just gives false alerts?
The way I read this option it’s just collecting statistics (used for SDRS I guess) but doesn’t actually do anything to affect performance since Storage I/O control is off. At worst it’s reporting false positives and feeding false data to SDRS (so the latter might make bad decisions). If you don’t use SDRS, is there any harm in having this on?
Yes it can impact performance and send false data to systems that do auto tiering.
[…] Yes, Vmware has SIOC and SDRS, but these don’t really work with modern flash arrays or HCI environments. In fact if you are using Vmware with Nutanix or any other platform with advanced storage auto-tiering capabilities, it is advisable to turn SIOC off […]
I’m using autotiering with SIOC manual threshold reasonably high – depending on tiers, but usually 30ms works pretty well. In this configuration you can still benefit from SIOC to some extend – till autotiering kicks in and migrates extends to the higher tier.
AFAIK Storage spikes are caused by SIOC injector which is used if you have Percentage of peak throughput configured with SIOC, which is unfortunately default settings nowadays.
[…] Yes, Vmware has SIOC and SDRS, but these don’t really work with modern flash arrays or HCI environments. In fact if you are using Vmware with Nutanix or any other platform with advanced storage auto-tiering capabilities, it is advisable to turn SIOC off […]