Most VMware admins know by now that auto tiering storage systems should have SIOC configured differently than if they have storage systems that don’t do auto tiering. Especially if it’s auto tiering at block level. It’s also recommended to have IO metric collection disabled because it will get different results based on where the blocks on the datastore physically reside. What you may not know is that VMware has enabled IO metric collection for every datastore by default with ESXi 6.0. This can cause some weird latency spikes on your storage system that can be very hard to isolate if you don’t know what you’re looking for or what might be causing it. So here is how you fix the problem.

If like a lot of people you still primarily use the C# client for vSphere then you will have to switch to a web browser to fix this problem. This will require you log into the vSphere Web Client. Once you are logged in select Storage -> <Datastore> -> Manage. You should see something similar to the following image:

Click on the Edit button and then click the check box next to Disable Storage I/O statistics collection as in the image below:

Click Ok and repeat the process for your other datastores.

If you would like to do the above more automatically by using a PowerCLI Script, please check out this great blog post by Burke Azbill here.

Final Word

Hopefully the above helps you reduce any unexpected latency spikes on your datastores and storage systems you use with ESXi 6.0. As the number of storage systems with auto tiering and multiple storage types increases over time this becomes even more relevant. Given the trends in the industry it is a wonder VMware made this a default setting in the first place. As storage systems become more intelligent there is less and less need for features such as Storage IO Control, which means admins should be able to focus on other more important things, rather than worrying about noisy neighbor problems. FYI, If you are a Nutanix customer this also applies to you. I found this little gem while running an NCC check on a cluster that I was getting periodic unexplained latency spikes. Needless to say the problem has now been resolved after making this change.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2016 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Heads Up! If you’ve updated to ESXi 6.0 U1b, build 3380124 and you have lots of templates, you may run into some problems if you update VMware Tools to the latest version. I just upgraded my environments to the latest VMware patches ESXi 6.0 U1b (build 3380124), that has just come out. As you do usually when there is a new hypervisor build you upgrade VMware Tools. Well that proved to be a big problem for my VM templates that I use to provision new systems. But I’ve got a workaround.

As soon as VMware Tools is updated on any templates you will no longer be able to clone those templates. If you’ve updated any templates with the version of VMware Tools that comes with ESXi 6.0 U1b then you need to uninstall it and reinstall the prior version that came with ESXi 6.0 build 3247720. After the couple of reboots that you have to go through with an uninstall and reinstall of VMware Tools you will find that you can now clone VM’s and have them automatically customized. I ran into this problem on Windows 2008 R2 Server. So I know it will impact this guest OS. I haven’t tested other OS’s yet, but others could be impacted. I’ve logged a support call with VMware to address this problem. In the meantime, the workaround is fine. The Official VMware KB Article 2142982 explains the situation.

[Updated 14/01/2016] After further testing I have narrowed down the problem area to new installs where the complete option is selected, and any upgrades where the complete options was previously selected, or where the VMCI / NSX Guest Introspection Driver is included. I have been able to successfully clone from a new VM Image that has had a fresh install of Windows 2008 R2 and VMware Tools without the VMCI / NSX Guest Introspection Driver, or where VMware Tools was installed twice / installed and repaired on the same VM, when the complete option was previously selected. This seems to be similar to what other of you have also reported. I have completed the Upgrade Scenario testing as well and confirmed that after an upgrade, if the complete install option was previously selected the VM will not clone due to the same VMCI driver problem. If VMCI driver is removed by running VMware Tools Install again and selecting Modify and unselecting VMCI, then you will be able to close the VM.

This update just in from VMware Support “VMware Engineering have confirmed that the issue is dependent on the install/upgrade sequence. Specifically, the issue is aligned to the version of deploypkg.dll in the vmtools package. GSS and Engineering are mapping the ESX and vmtools update versions to the deploypkg.dll  versions to confirm which upgrade sequences are problematic. A KB article will be published once this information is finalised.“

Thanks to VMware Support for getting to this stage very quickly. The VMware KB Article 2142982 has now been published. 

Final Word

I guess someone has to take the risk and patch their systems to the latest versions first, especially as these were security patches with a critical severity. Fortunately like all good IT environments I only did my test systems first. This is the whole point of having infrastructure test systems. You can test infrastructure hardware and infrastructure software changes first before putting them into production. The old saying goes that software eventually works and hardware eventually fails, but these days a lot of your hardware is also software, especially in a virtualized software defined datacenter. It pays to have appropriate test systems and test plans to mitigate the risks associated with software updates and changes of all types, including infrastructure software. Thanks to all of you in the community that contributed to this effort and commented on this blog post. I have been relaying your feedback during my discussions with VMware Support.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

While a lot of people (me included) are excited about the technical speeds and feeds of the vSphere 6 launch, there is something much more fundamentally important about this release. Some technical highlights include 64 node clusters, 8000vm’s per cluster, 480 pCPU’s & 12TB RAM & 2048 VM’s per Host, 128 vCPU & 4TB RAM per VM support, SMP FT (up to 4 vCPU FT), enhancements to NIOC, VVOLs, SIOC enhancements etc, and much more. The reason this release is more fundamentally important though is related to the same reason that Amazon with AWS went from nothing to cloud leader. It’s not just about the technology, but what the technology enables, changing the business model, reducing friction, enabling flexibility. What might seem like a relatively small feature on the surface has the potential to change the landscape in hybrid cloud SDDC. If you’d like to know more about this, and all of the goodness coming as part of the launch of vSphere 6, keep reading.

VMware is about to release the latest version of the flagship vSphere product in what I predict will be a defining moment for the Mobile / Cloud Era. For the first time you will be able to live migrate, without any disruption, between private cloud datacenters, to public cloud, and over long distance, a true hybrid cloud and software defined datacenter. You will be able to implement improved quality of service for all applications with additional SLA guarantees, and scale to unprecedented levels. All while reducing management overheads and complexity across the entire ecosystem. With the policies following the virtual machines and virtual applications regardless of where they are physically located. 

This release has been baking for a while and for good reason. There is a big commitment to product quality, which was evidenced by the first ever public beta for VMware vSphere.  This is a major release, and is well deserving of the 6.0 version number. A lot of hard work has gone into this release by thousands of people. I was able to test a lot of the functionality during the beta and it was great to be able to contribute to the product. 

So why do I think this is such a defining moment? The world is changing with the massive explosion of mobile smart phones and the applications that support them. Billions of users are now demanding their applications wherever and whenever they want. So not only are the users mobile, their applications need to be. The applications need to be able to scale massively and on demand, and move to wherever it makes sense.

Previously migrating workloads from a private cloud or private SDDC to a cloud provider and to support a hybrid cloud required the systems being migrated to be shut down. You could migrate templates and power them on and update load balancer records, but that’s not quite the same as being able to dynamically live migrate any workload from your datacenter to a cloud without any downtime or disruption, and across long distances. If you really wanted to deliver cloud workloads and mobile workloads at scale, they had to be written for a particular cloud environment. Then you are stuck in a hotel California, where you can check out, but can never leave. This is the fundamental difference, and the fundamental technical change that is potentially enabled by vSphere 6, which in turn will deliver business and commercial disruption to current models.

The enhancements to VMware vMotion have the potential to change the way organisations run their datacenters, applications and interact with cloud service providers. It is conceivably possible to migrate workloads between different clouds on demand, based on various business rules and policies, provided they are based on vSphere 6.

So where does the comparison to Amazon and AWS come from? The reason I believe AWS became successful it not because of technology, it’s because it changed the economic and business model of consuming infrastructure. It reduced the friction, made everything on demand, and delivered to development and applications teams, in a way that was transparent. With the VMware vMotion enhancements allowing Cross vCenter vMotion and Long Distance vMotion, Cloud Service Providers can provide even less friction, on demand, run anywhere appropriate type of service. Some of the tyrannies of the network and live migration are being demolished. This again can change the way infrastructure is consumed and make it easier for app teams to deliver. But this needs to be blended with a commercial construct that also supports it.

At VMworld in 2014 Bill Fathers, Father of vCloud Air, reported that some 6% of workloads were running in Cloud environments. I believe part of the reason is because of the difficulty in migrating workloads to a Cloud, and between different Clouds, without disruption, and without having to change the underlying apps. With the changes that VMware is starting to deliver from vSphere 6, conceivably this could rapidly change the adoption of VMware compatible Clouds. There is still much to do in terms of the Network, which is still one of the barriers to Cloud, but this will go a long way. Soon you will be scaling workloads on demand to support the billions of mobile users and migrating those workloads to the cloud of choice that makes sense, almost anywhere in the world.

I was recently at the Singapore VMUG User Conference and listening to one of my colleagues, Scott Drummonds, talk about Cloud and locality. Locality is important because there are orders of magnitude computational difference the further the users are from their applications and data. How this relates back to the VMware vSphere 6.0 launch and vMotion Across vCenter and Long Distance is that it will now be possible to migrate workloads on demand closer to where the users are, especially as we start to see Cloud services become more local, and more miniaturised over time.

Final Word

At first glance vMotion Across vCenters and Long Distance vMotion may not seem that revolutionary. When you put this all in the context of the Hybrid Cloud and Software-Defined Datacenter that VMware has been building towards for over 5 years it is easier to see that this actually delivers a fundamental change in the way infrastructure resources can be used. It won’t be long before the live migration of VM’s is happening as depicted in the image above. What new possibilities will this open up for businesses? What impacts will this have on data sovereignty? Your thoughts and comments are appreciated.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.