| Unique Visitors |
Powered By 
| Unique Visitors |
VMware has announced that it will turn off TPS in upcoming version of it’s hypervisor ESXi and vCloud Air hybrid cloud service. This is due to a security bug, considered a very rare possibility and only exploitable in very controlled and largely misconfigured environments. TPS also known as Transparent Page Sharing is a memory management technique that allows multiple VM’s to share a read only copy of the same memory page. When a VM needs to update or write to a page a new copy is created. The idea is that if there are many VM’s with similar memory pages on the same physical host server it will de-duplicate the pages and only store one copy. The result is that you can run more VM’s per physical server while still achieving very good performance.
TPS has for a long time been used as a competitive advantage by VMware over all of the other hypervisors. But realistically it hasn’t been in wide use by most customers for some time (since ESX 3.5) as the amount of RAM per host has increased, because of the use of large memory pages (2MB instead of 4KB) in Nehalem and above processors, and because most customers don’t want to run their systems at 100% utilization so that they can handle bursts of activity. When using large pages TPS only kicked in when systems were over 96% memory utilization, at which point large pages would be broken down into small pages that could be shared. However this has been a popular technique with service providers and with virtual desktop environments, and in some test and development environments, where over commitment of memory may have been acceptable.
The security problem was found by recent research that leverages Transparent Page Sharing (TPS) to gain unauthorized access to data under certain highly controlled conditions. The research demonstrated that by forcing a flush and reload of cache memory, it is possible to measure memory timings to try and determine an AES encryption key in use on another virtual machine running on the same physical processor of the host server, if Transparent Page Sharing is enabled. This is effectively a VM escape, where code executed within one VM can break the hypervisor isolation and read data from another VM’s memory. Certainly not a good situation if said VM contains credit card data, as we’ve already had enough breaches recently. The conditions under which this could be exploited would be rare in the real world, especially as most environments don’t use TPS actively, even if it is enabled. Even so, I believe in being secure by default, and even though the number of conditions that have to simultaneous by true for this to be exploited would be very rare, if this were exploited the impact could be high. So I believe that VMware is taking the right approach to this research by disabling TPS.
I have been a proponent for leaving TPS enabled in the past, even though a few others have previously recommended it be disabled for performance reasons. My argument was that TPS is a good safety net if all else fails, even if during normal operations it is not used. Also performance was never proven to be a factor. I put this argument in my article Blueprint for Successful Large Scale Oracle Virtualization on vSphere when an EMC paper recommended disabling TPS. To quote that article “Disabling TPS can have disastrous consequences, including causing additional host swapping, which can result in extremely poor performance, much worse than disabling it could ever possibly gain.” So this begs the question, now that it’s being disable by VMware what impact will it have?
Without TPS you will have to have much more conservative memory usage per host. If you business requirements dictate, you will have to be able to sustain maintenance and failure without causing memory overcommitment. If there is a failure or maintenance that causes temporary or prolonged overcommitment of memory you will have a lot more guest OS swapping, due to ballooning, and also host swapping may occur, which would greatly impact performance. Memory swapping is the enemy of performance, and this also adds significantly to poor performance on shared storage if it occurs. But this is possibly better than the alternative security bug.
If you have an existing VMware vSphere environment this will mean you need to evaluate the level of resource usage you have today, your standard operating procedures for maintenance, and the settings of VMware HA Admission Control for failure. If you don’t have sufficient available memory to operate your environment in the case of failure or maintenance, then you may need to upgrade the amount of RAM per host or purchase additional hosts. With any additional hosts you’d need additional licenses. Frank Denneman has a good take on the capacity planning implications in his article here.
TPS will be disabled by default from the following VMware vSphere Releases:
VMware’s official statement on this problem is contained within KB 2080735 Security considerations and disallowing inter-Virtual Machine Transparent Page Sharing. This KB also contains the steps to disable TPS on older versions of VMware vSphere that will not be covered by patches.
If you want to check whether you have TPS enabled or not on your existing versions, and if you want to disable it you can use the following PowerCLI examples (explicitly provided without any warranty, use at your own risk):
Check if TPS is Enabled on all hosts connected to a vCenter Server, Mem.ShareScanGHz returns > 0 if enabled.
Connect-VIServer <YourvCenter>
Get-VMHost –State Connected | Get-AdvancedSetting –Name Mem.ShareScanGHz | Format-Table –Property Entity,Name,Value -AutoSize
Disconnect-VIServer
Disable TPS on all hosts connected to a vCenter Server by setting Mem.ShareScanGHz = 0, check the setting has been applied correctly
Connect-VIServer <YourvCenter>
Get-VMHost –State Connected | Get-AdvancedSetting –Name Mem.ShareScanGHz | Set-AdvancedSetting –Value 0
Get-VMHost –State Connected | Get-AdvancedSetting –Name Mem.ShareScanGHz | Format-Table –Property Entity,Name,Value -AutoSize
Disconnect-VIServer
So if TPS is vulnerable to data leakage and VM escape attacks what about the recently announced Project Fargo, AKA VMFork? VMFork allows a running VM to be quiesced and rapidly cloned by using a similar copy on write technique to share a read only copy of the parent VM memory, and sharing the parent VM’s read only disk, with updates being written to a delta disk. This allows a VM to be cloned and get up and running on the network with it’s own personality in a matter of a few seconds, with the VM memory and disk effectively being deduped at the same time. This doesn’t just have applicability to VDI environments, but web server environments, Dev and Test environments and many other use cases. I’m sure VMware won’t let VMFork out in the wild until issues such as the VM escape bug with TPS are addressed. Kit Colbert, VMware CTO for End User Computing, has said to me that VMFork is much more secure than TPS, so it may not suffer from the same problems.
VMware is not alone with a VM escape vulnerability being discovered. There was also a security bug made public regarding the Xen hypervisor that allowed a VMescape, where code executed within one VM could escape the encapsulation of the hypervisor to a neighbour VM or dom0. This is covered at the VUPEN Vulnerability Research Team’s blog site.
Final Word
Nothing is fully secure. You can never guarantee that your system isn’t vulnerable to attack. All you can do is take appropriate measures to reduce the risk of attack, implement technical controls and monitoring and auditing processes. Implement separation of duties, least privilege access, and role based access controls. Implement the guidelines that make sense based on your business requirements from the VMware and other vendors hardening guides. Comply with the security standards for your industry / company that make sense. Stay on top of critical security patches and implement them as soon as practicable, especially for any environments containing public facing or highly secure systems.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
If you thought Ebola was deadly to humans wait till you get a load of the latest security issue impacting the world wide web and most everything connected to it including potentially your phone, lights, servers and the list goes on (excluding Windows systems). If Heart Bleed wasn’t bad enough at the start of the year the new Shell Shock bug certainly is. It is what I would term the Mother of All Bugs (MOAB). It impacts almost all Unix, Linux and Mac systems and allows a remote attacker to execute arbitrary code and potentially steal your data, credit cards and other information. So how serious is this? Well the NIST CVE Alert Rating on this is a 10 for severity, and a low for complexity to exploit (read my 7yr old could exploit this bug). So basically the worst possible kind. Oh, but wait, there’s more…
According to this article, there are already worms exploiting this bug. So the impact could be wide spread for the vulnerable systems. I would expect most major vendors to come out with security advisories very promptly for this, after they have assessed their systems. For those of you running VMware, they have posted a blog here, and an advisory here. As things stand if you’re running VMware tools on top of Windows, such as vCenter for example, then you are not vulnerable. Also ESXi is not vulnerable as it uses ash shell via BusyBox instead of Bash. However any virtual appliances may well be vulnerable, including the vCenter Server Appliance. I would recommend keeping and eye on VMware KB 2090740 for the latest updates. For home users, lock up your networks tight and try to prevent anyone getting in the virtual front door, until such time as there are widely available fixes.
This bug highlights the importance of keeping patches up to date and staying across the alerts from the likes of NIST. Be aware of this bug and get patched and protected as soon as you can. Not everyone has a vaccine for this one yet, but hopefully it’s not far away. This bug could cause a lot of change to the way systems are designed, implemented and secured. What’s to say another bug of this nature isn’t just around the corner? Better to be prepared.
Final Word
This is probably one of the highest impact and most wide spread bugs with the highest severity that I’ve seen in over 20 years in IT (reminds me of the original internet worm). As the Internet of Things (IOT) spreads bugs of a similar nature will have a much wider impact and much more sever consequences. Security of your systems is going to become an ever more serious issue and this is why Micro Segmentation, and using technologies such as VMware NSX and vCloud Networking and Security will become so important. In addition to more intelligent firewalls, such as from Palo Alto Networks. As much as we give Microsoft a hard time over security and patches, neither Heart Bleed and Shell Shock impacted Windows systems.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Back in 2010 I was helping a large company troubleshoot their virtualized SAP environment, which was experiencing instability and performance problems. One thing we noticed was that the buffers on the NIC’s were periodically overloading due to the large amount of small packets. This was on vSphere 4.0, with Windows 2003 64bit OS at the time and using VMXNET3. Unfortunately at that stage the VMXNET3 driver for Windows didn’t support increasing the send or receive buffers and as a result we had to switch over to E1000 and increase the TX and RX buffers, which resolved the problem (in addition to adding memory reservations to the VM’s). However since vSphere 4.1 it has been possible to modify the buffers in VMXNET3 to resolve these sorts of issues. I have been experiencing this myself in my home lab and have as a result modified the buffers, but it appears I may not be alone in experiencing this.
I thought this was just something I had done in my lab environment. But after reading Michael White’s Newsletter and the VMware KB 2039495 – Large packet loss at the guest OS level on the VMXNET3 vNIC in ESXi 5.x / 4.x, it appears I’m not alone in this. Fortunately it is easy to make the necessary modifications to the buffers and resolve the majority of the packet loss issues as follows:
In my environment I’ve also set my Large RX Buffers to 8192 and my TX Ring Size to 4096.
If you suspect that your virtual machines may be dropping packets or losing packets then you should consider adjusting the RX and TX buffers. This may well lead to increased performance and more importantly application stability. Sometimes in addition to increasing the buffers you may need to reserve the memory if it’s a very important app. This will ensure it can receive the resources it needs.
Final Word
In most cased the default settings are fine. In some cases there are some adjustments needed. This is one of the cases, if you are experiencing this problem, where adjustments are needed. There is no patch as such to address this problem at this time. But VMware will hopefully make improvements to its drivers and IP stack in future versions of vSphere.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
I was recently doing some maintenance and upgrades on some of the hosts in my lab environment when I had the need to enter maintenance mode and reboot my hosts from the command line. I have used esxcli a bit, which is handy if you’re going for your VCAP-DCA exam, but I’d not rebooted a host from command line. Fortunately it was easy to find out how to do it, and it’s well documented. I thought I’d let you know where to find it, and what the commands are, just in case you ever need to know.
Firstly, to put a host into maintenance mode there are two options. ESXCLI, which I was familiar with, or vicfg-hostops, which I’ve only just discovered.
To enter and exit maintenance mode:
esxcli <conn_options> system maintenancemode set –enable true, where <conn_options> are the connection options such as –server=<servername>
To exit maintenance mode
esxcli <conn_options> system maintenancemode set –enable false
In addition to the above using vicfg-hostops you can issue the following
vicfg-hostops <conn_options> –operation enter or –operation exit to enter or exit maintenance mode.
All of the above is documented in the official VMware vSphere CLI Documentation – Entering and Exiting Maintenance Mode.
To reboot a host:
To reboot a host from the command line we would use the vicfg-hostops –operation reboot command, to force a reboot you could add the –force option, similarly to shutdown a host you use –operation shutdown, which can also be used with the –force option.
This is also nicely documented in the official VMware vSphere CLI Documentation – Stopping, Rebooting, and Examining Hosts.
Final Word
The VMware CLI options have come a long way over the years and you can really quickly and easily automate a lot of tasks using esxcli, PowerCLI and the other command line tools. I would encourage all of you to check them out. They are covered in the blueprint for the VCAP-DCA exam, so if you wanted to achieve the advanced level administration certification from VMware you should learn how to use these tools, and how much easier they could make your job if you’re a vSphere Admin.
—
This post appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.