Heads Up! If you’ve updated to ESXi 6.0 U1b, build 3380124 and you have lots of templates, you may run into some problems if you update VMware Tools to the latest version. I just upgraded my environments to the latest VMware patches ESXi 6.0 U1b (build 3380124), that has just come out. As you do usually when there is a new hypervisor build you upgrade VMware Tools. Well that proved to be a big problem for my VM templates that I use to provision new systems. But I’ve got a workaround.
As soon as VMware Tools is updated on any templates you will no longer be able to clone those templates. If you’ve updated any templates with the version of VMware Tools that comes with ESXi 6.0 U1b then you need to uninstall it and reinstall the prior version that came with ESXi 6.0 build 3247720. After the couple of reboots that you have to go through with an uninstall and reinstall of VMware Tools you will find that you can now clone VM’s and have them automatically customized. I ran into this problem on Windows 2008 R2 Server. So I know it will impact this guest OS. I haven’t tested other OS’s yet, but others could be impacted. I’ve logged a support call with VMware to address this problem. In the meantime, the workaround is fine. The Official VMware KB Article 2142982 explains the situation.
[Updated 14/01/2016] After further testing I have narrowed down the problem area to new installs where the complete option is selected, and any upgrades where the complete options was previously selected, or where the VMCI / NSX Guest Introspection Driver is included. I have been able to successfully clone from a new VM Image that has had a fresh install of Windows 2008 R2 and VMware Tools without the VMCI / NSX Guest Introspection Driver, or where VMware Tools was installed twice / installed and repaired on the same VM, when the complete option was previously selected. This seems to be similar to what other of you have also reported. I have completed the Upgrade Scenario testing as well and confirmed that after an upgrade, if the complete install option was previously selected the VM will not clone due to the same VMCI driver problem. If VMCI driver is removed by running VMware Tools Install again and selecting Modify and unselecting VMCI, then you will be able to close the VM.
This update just in from VMware Support “VMware Engineering have confirmed that the issue is dependent on the install/upgrade sequence. Specifically, the issue is aligned to the version of deploypkg.dll in the vmtools package. GSS and Engineering are mapping the ESX and vmtools update versions to the deploypkg.dll versions to confirm which upgrade sequences are problematic. A KB article will be published once this information is finalised.“
Thanks to VMware Support for getting to this stage very quickly. The VMware KB Article 2142982 has now been published.
I guess someone has to take the risk and patch their systems to the latest versions first, especially as these were security patches with a critical severity. Fortunately like all good IT environments I only did my test systems first. This is the whole point of having infrastructure test systems. You can test infrastructure hardware and infrastructure software changes first before putting them into production. The old saying goes that software eventually works and hardware eventually fails, but these days a lot of your hardware is also software, especially in a virtualized software defined datacenter. It pays to have appropriate test systems and test plans to mitigate the risks associated with software updates and changes of all types, including infrastructure software. Thanks to all of you in the community that contributed to this effort and commented on this blog post. I have been relaying your feedback during my discussions with VMware Support.
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
I am not sure why VMware released VMTools 10.0.0 with 6 u1b considering 10.0.5 is out since November. 10.0.5 have many bug fixes included. Perhaps using that version would be a fix for you.
You’d think that “someone” would be VMware….
We run an app that defaults to OpenGL and falls back to CPU rendering. Had real problems with VMware Tools 9.10.5 and 10.0.0…app would start but crash when doing its OpenGL check. Coincidentally (not!) that appears to be right about the time VMware started putting OpenGL “improvements” into Tools, which became official in 10.0.5 (check file sizes and dir. structure 9.10.5 through 10.0.5 to see what I mean). Was very disappointed with seeming lack of QC.
PS – falling back to 9.10.1 or moving up to 10.0.5 did clear up our issue (for anyone that runs across a similar situation).
Where do people download 10.0.5 from? I didn\’t see it on packages.vmware.com.
Thank you. Upgrading to the 10.0.5 tools was the solution for me as well.
Where is VMware QA these days?
[…] Please be aware of a bug with VMware Tools and Templates. Read the following post for more details: Heads Up! Do Not Upgrade VMware Tools on Hosts with ESXi 6.0 U1b. […]
can you post the build you can see in the vmware tools about page so we can compare with is downloadable separately from VMware website?
I’ve run into an issue using the Introspection drivers in VMware Tools 10.0.0 and 10.0.5, 9.10.5 seem to be fine though. If using vCNS and an SVA the guest VMs will hang logging in and hang logging out. Turn the SVAs off everything is fine and using the old VMware Tools from 6.0U1 work fine. (The do eventually log in/out)
Chad, is that with both file and network introspection components of VMware Tools installed?
I tried with disabling NSX Network Introspection Driver and it didn’t matter, I also tried 10.0.5 and the same issue. Also had issues with VDP backing up guests and I think I resolved that with setting disk.UUID to false and then back to true. The disks had no serial number after the VMware Tools upgrade.
I just installed fresh tools on 2008 R2 and that NSX Introspection driver takes an age. Really slows down the install.
Sorry for the typos, I just meant to compare what is the current download from the website since now it can be downloaded separately from the ESXi ISO
Customization doesn’t work with 10.0.5, after downgrading to 10.0.0 it works fine.
After couple retries I found that simple reinstall 10.0.5 also solved this customization error.
I’ve deployed vCenter 6.0 Update 1b (build 3339084) managing ESXi 6.0 Update 1b (build 3380124), Windows 2012 R2 with Update Datacenter VM.
vSphere installs VMware Tools 10.0.0 build 3000743.
Brand new environment, no upgrades.
Guest customization with Active Directory domain add is working thus far.
How big is the VM? I had it working on a very small VM, but not large VM's with Windows 2012 R2. But I haven't got it to work with Windows Server 2008 R2. The logs are with VMware to analyze. I repro'd it for them on Webex yesterday.
I just required two attempts to install VMTools from ESXi 6.0 U1b on a Windows 2008 R2 Server VM that I'd built from scratch. Something is just not right about that build of VMTools.
Jason are you saying that the VMware Tools that are downloadable from the website are newer than the ones coming with the ISO of ESXi 6.0 U1b?
Quick question. Other than downloading the tools iso to install in the OS. How do you replace the version that installs when you’re in vCenter and push tools that way?
Store the tools centrally, and reconfigure the hosts see this blog:
[…] of today – 1/10/15 there is an issue with 6.0 U1b where if you update the tools in a template you might have issues. So I am going to […]
we have experienced a degradation of performance in our VMs after upgrading them to Virtual Hardware 11.
There is one KB representing similar symptoms:
We have managed to solve the problem temporally by downgrading Virtual Hardware version to 10.
Before we apply the solution from the KB I would like to know from you guys if anybody is having similar issues?
We are experiencing the same problems. We are also planning to downgrade to Version 10 because we also discovered that this would solve the Problem.
Where did you find the KB? Your Link is dead.
we can confirm now that the fix proposed in the KB-2129176 (http://kb.vmware.com/kb/2129176), fixes the Peformance issues we were having with applications with Microsoft SQL based workloads but we also had big Performance issues with our Exchange Organization…
The following KB describes TSO and LRO
Instead of disabling RSC per affected VM… we decided to disable LRO/RSC for all virtual machines on an all our ESXi hosts using:
esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0
esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0
Note:This will disable LRO for all virtual machines on the ESXi host. Virtual machines will need to be powered off and back on or vmotioned between hosts for the changes to take effect.
And now we are OK with good performance ratios.
I hope this will help you too.
I’ve had a SR open since 1/28/2016 and they still haven’t provided a resolution, I’m glad I stumbled on this post. I have templates running vmware tools version 9541 (9.10.5) where cloning was working (with customization) prior to installing U1b but does not work post U1b. If I haven’t touched those templates and they are not running the “bad” tools version 10240 (10.0.0), why can’t they clone anymore? That seems to indicate that something closer to the ESXi level happened in U1b that isn’t talking correctly to machines running some versions of vmware tools.
I also created a fresh Windows 7 VM (hardware 11) with tools 10.0.5 and I cannot clone with customization from that template either.
[…] VM fails to complete on a ESXi 6.0 Update 1b host I think Michael was the first to find this issue but we have a KB article now to […]
Version 10.0.6 of VMware tools is now available. The readme suggests that it fixes this issue.
After trying with the newer 10.0.6 version still seeing certain templates that are not working with Guest customization so perhaps this isn’t the fix after all.
I checked the release notes and even if the problem seems to be the same described in this post the build numbers mentioned don’t match.
Can someone who experienced the problem try the new build and share what happens?
I tried the new build 10.0.6 with view 6.2.2 and don’t see the problem anymore. Recomposing without problems.
10.0.8 is coming out soon to deal with Guest Introspection and other ESX host anomalies.
I hadn’t heard anything about 10.0.8 but thank you. I wonder if that will be rolled into the ESXi patch for VSAN 6.2 that my support engineer said would be released “soon” (nearly 4 weeks ago).
They (VMware) told me that for VM/ESXi/NSX/Guest Introspection performance issues, the culprits are;
vnetflt.sys and vsepflt.sys
Installing new VMtools will address those files. Have to reboot VM’s afterwards.
Monitor this link, hopefully soon, lol;
This is the link to monitor, not sure what happened;