As part of the development of Virtualizing SQL Server with VMware: Doing IT Right (VMware Press), which I co-authored with Michael Corey and Jeff Szastak, I needed to provide guidance around virtual networking. To do this I figured it would be a good idea to do some performance testing of various different virtual network adapters in VMware vSphere 5.5, as there wasn’t much in the way of performance data around. In all I would have performed approximately 600 individual test runs. All of the important details and much more (including tuning advice to get optimal performance) can be found in the book. But I thought I’d share with you some of the highlights of the results.
As more and more companies start to look at hyperconverged, web scale (such as Nutanix - where I work) or just converged solutions for your vitualization platform there is a need to make sure you know what your looking at and go in eyes wide open. There are a number of different aspects to evaluate, none of the different options are the same. There are no right or wrong answers, as many different solutions may meet your requirements, or be best suited to your requirements. The idea behind this article is to give you a list of questions to consider and to ask any potential vendor. This list is designed to be vendor neutral, and there are really no right or wrong answers, it’s just so you understand what you’re getting for some important aspects, based on my experience.
One of the little known and infrequently used features of vSphere since version 4.1 is the ability to connect a USB device to an ESXi host and then mount that device to a VM, and still allow vMotion to work without any problems. This is usually used for USB dongles required for software licensing, but can be used with a number of other devices. More often these days the USB connectivity is being used from a VMware Horizon View client to connect a USB device to a desktop. But if you’re 9000 miles away from your desktop, and you’ve been asked to connect a console cable to a physical network switch in the same location and run some debug commands, how can you do that? Well I figured it out and it makes for a good story.
Back in 2010 I was helping a large company troubleshoot their virtualized SAP environment, which was experiencing instability and performance problems. One thing we noticed was that the buffers on the NIC’s were periodically overloading due to the large amount of small packets. This was on vSphere 4.0, with Windows 2003 64bit OS at the time and using VMXNET3. Unfortunately at that stage the VMXNET3 driver for Windows didn’t support increasing the send or receive buffers and as a result we had to switch over to E1000 and increase the TX and RX buffers, which resolved the problem (in addition to adding memory reservations to the VM’s). However since vSphere 4.1 it has been possible to modify the buffers in VMXNET3 to resolve these sorts of issues. I have been experiencing this myself in my home lab and have as a result modified the buffers, but it appears I may not be alone in experiencing this.
Nutanix will be once again making WebScale Waves at VMworld 2014 at booth 1535. This year we are including a big focus on enterprise applications, including SAP, Oracle, SQL Server, Exchange and Java. I will be spending time on the booth along with many other experts at Nutanix and taking 1:1 meetings, in addition to the sessions that we’ll be presenting at VMworld. My co-authors and I may also be signing copies of Virtualizing SQL Server with VMware: Doint IT Right at the Nutanix booth, so bring your copy along. To find out how you can learn more about greatly simplifying your IT infrastructure with Nutanix WebScale IT and where to party at VMworld, read on.
VMware ESXi hypervisor like many systems that implements TCP/IP uses TCP Delayed Acknowledgement to try and improve network efficiency. Essentially the idea of Delayed Ack is that a system can acknowledge every other full segment received, and must acknowledge by a certain threshold. The actual threshold can vary and be up to 500ms. While this works well for general communications and consistent streams of traffic and where full TCP segments are being sent, it can cause negative performance impacts for IP based storage systems. The main impact can be increased latency, due to the Delayed Ack Timeout, if small IO’s are being sent and received. For iSCSI on ESXi this is documented in VMware KB 1002598, however for NFS it isn’t as straight forward. I’ll take you through how to disable Delayed Ack for NFS.
I don’t like it when I get Purple Diagnostic Screens a.k.a. Purple Screen of Death or PSOD for short. Fortunately these are fairly rare. However there is one I came across just recently with a customer running vSphere 5.1 U1 and it is quite nasty. The PSOD was caused by TCP Heap exhaustion on an ESXi 5.1 U1 host. The host has the recent patches, and the usual search of the knowledge base didn’t really turn much up. The customer is running NFS, although the symptoms may not be tied only to NFS, any host based IP storage protocols (NFS or iSCSI) could be impacted. I’ll briefly tell you what we have found out, the logs to watch out for and some KB’s that will be helpful and steps you can take to prevent this from happening.
The printing presses at VMware Press have been churning out copies of Virtualizing SQL Server with VMware: Doing IT Right, which I’ve co-authored with Michael Corey and Jeff Szastak. We had a great technical reviewer for this book in Mark Achtemichuk (Mark A – VCDX-050). There will be enough copies for everyone, and we’ll be signing them at VMworld(s), and vForum Sydney if you’d like your copy autographed (and hopefully SQL PASS events also). This is 512 pages that combines decades of experience into a book, in a single place, which is the definitive guide for virtualizing SQL Server on VMware. This project took us over 12 months to complete, and many late nights and full weekends. If you ask my family they almost didn’t see me for the entire year. This is the third book project I’ve been involved with, after being technical reviewer for VCDX Bootcamp and Virtualizing and Tuning Large-Scale Java Platforms, both also by VMware Press. Duncan Epping has done us a great honour and written a fantastic foreward for the book.
Although I have dedicated this book to my wife, Susanne, and my four sons, Sebastian, Bradley, Benjamin, and Alexander , for their ongoing support (and putting up with my absence during this project). I’ve also dedicated this book to the VMware Community.
I was also lucky to have some great sounding boards during this project in addition to my fantastic co-authors. Kasim Hansia, VMware Strategic Architect and SAP expert, Cameron Gardiner, Microsoft Senior Program Manager Azure and SQL, and Josh Odgers (VCDX-090), Nutanix Senior Solutions and Performance Architect.
Below I have included the full Table of Contents so you can take a look at what we cover and see the value that we’ve packed into this book. I hope you enjoy the book, get a signed copy, and succeed when virtualizing SQL Server.
For anyone that has been following VMware for any amount of time will know they like to keep their Beta programs very limited and usually most people don’t know what’s going on until it’s announced in public at an event. However without a wider audience involved in the use and testing of VMware vSphere in beta form there is less chance to catch bugs and the GA product quality may not be as good as what it could have been.
As you may or may not know for a few years you have been entitled to unlimited usage of Suse Linux Enterprise Server for VMware as part of your VMware vSphere Licenses and Support and Subscription. This is the same Suse Enterprise Linux that everyone was used to, just distributed for VMware under and OEM agreement. This meant that in a vSphere environment you could use as much SLES as you wanted and get all the patches and updates you need for no additional charge over and above you existing VMware SNS. If you wanted phone support on top that was a very reasonable and small fee per host. Unfortunately for those of you didn’t know about this, or for whatever reason didn’t take advantage of this then you might miss out.