If you’re upgrading from vSphere 5.1 to vSphere 5.5 and you ARE NOT using Custom CA SSL Certificates then you might run into an error. The error will be encountered during the upgrade of SSO, and specifically the Lookup Service, and only occurs in specific conditions, such as when using the default VMware Self-Signed Certificates. If you run into this problem your upgrade process will roll back, but leave behind some upgrade files that need to be cleaned up. This article will briefly touch on the recommended solution to this problem.

Many of you will recall the many articles that I wrote regarding updating the default self-signed SSL Certificates in vSphere 5.0 and 5.1 to Custom CA Certificates. If you haven’t seen these articles and you’re interested in SSL Security you can check out Updating CA SSL Certificates in vSphere 5.1 and Updating CA SSL Certificates in vSphere 5. To make the process of updating certificates easier VMware created the VMware Certificate Automation Tool and VMware Partner VSS Labs created vCert Manager.  If you’re using CA Signed Certificates for your SSL communications between the various vCenter components then you won’t strike the problem I described above. So now might be a good time to review my previous articles and/or use one of the automation solutions that are available.

This issue during the upgrade from vCenter 5.1 to vCenter 5.5 is described in the VMware KB Article – Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511). You will likely see an error such as “Warning 25000. Please verify that the SSL certificate for your vCenter Single Sign-On 5.1 SSL is not expired. If it did expire, please replace it with a valid certificate before upgrading to vCenter Single Sign-On 5.5.” or the vCenter Upgrade will simply fail and roll back. In the vim-sso-msi.log you will see an error message like the following:

“Action 10:06:03: PostInstallScripts. Importing Lookupservice data…
CustomAction DoUpdateAndMigrateTasks returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)”

As described in the VMware KB this issue does not affect you if:

• You are using custom certificates (recommended)
• You are using the vCenter Server Virtual Appliance (only applies to the Windows version of vCenter Server)
• You are performing a fresh install of vCenter Server 5.5
• You are upgrading from:
  • vCenter Server 5.0
  • vCenter Server 4.x
  • a fresh install of vCenter Server 5.1 Update 1a or later

For the recommended fix please refer to the VMware KB – Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511). The fix will require modifying the Windows Registry. Before any upgrade of vCenter is attempted it is recommended that you take a backup and potentially have a snapshot in place for the vCenter Database and the vCenter Server system itself so you have a point you can roll back to. This should be standard practice in most VMware environments, as should testing the upgrade process, as should upgrading test environments and management environments before upgrading production. Given that this impacts environments that have self-signed certificates it has the potential to impact a large number of customers, however as it only impacts customers upgrading from vCenter 5.1 prior to version Update 1a to vCenter 5.5, the number of impacted customers is reduced.

Final Word

The easiest way to get around this problem if you’re using vCenter 5.1 would be to either run through the registry fix described in the KB article. Upgrading to vCenter 5.1 U1b will not correct the issue as it doesn’t correct the certificate. You may also choose to completely rebuild your vCenter with a fresh install of vCenter 5.5 against your existing database. Alternatively you may choose to update your vCenter Server to use CA Signed Certificates, which will also improve the security of your critical management infrastructure. Regardless of the option you choose make sure you have a backup of the vCenter Database and vCenter so you can roll back if needed.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Recently I posted some updates to the supportability of Windows Server 2012 Clustering and SQL Server in particular in my articles titled Windows Server 2012 Failover Clustering Now Supported By VMware With Some Caveats and The Status of Microsoft Failover Clustering Support on VMware vSphere 5.1. VMware KB 1037959 Microsoft Clustering on VMware vSphere: Guidelines for Supported Configurations explains the configurations that are supported and provides guidelines. The reason for this article is because some people might think because these configurations are now supported on top of vSphere that somehow the configurations are also supported with vCenter and other VMware Products. Unfortunately this is not the case. In this article I’ll cover some background and where things are today.

In October 2012 I published an article titled Clustering Support on vCloud Director and vCenter Databases, which explains which products supported clustering of their databases. I didn’t cover every product, just a selection of the popular products, it also didn’t cover SQL Server Replication, Mirroring or AlwaysOn. For things like vCenter, VMware supports vCenter Server Heartbeat as the means of protecting both the vCenter Server itself and the vCenter Server Database (as well as a selection of related components). But you can’t use vCenter Server Heartbeat to protect the vCloud Director database for example, see Using vCenter Heartbeat to Protect Non-vCenter SQL DB? Think Again!.

Clustering of the Database has not been tested or validated by VMware to date prior to vCenter 5.5. VMware will attempt to help customers with their configurations even if it includes a clustered database, unless the clustering technology is believed to be at fault. If the clustering technology is believed to the cause of the fault you would need to contact the vendor of said clustering technology for further assistance.

So where does SQL Server 2012 AlwaysOn Availability Groups come into this picture? SQL Server 2012 AlwaysOn Availability Groups is basically a database availability technique using database replication with automated failover between the members of the availability group (depending on how you configure it). This doesn’t require shared disk clustering, which is why it got added as supported recently on top of Windows Server 2012 and on vSphere 5.1 platform. But in terms of using this for the databases that support your VMware products you’ll be in the same boat as mentioned above with clustering of the databases. VMware has not tested and validated the use of SQL Server 2012 AlwaysOn Availability Groups with their products. It may well work fine, like clustering does for many databases, but it could well break things as well. VMware will likely provide best efforts support up to the point that they believe that AlwaysOn Availability Groups might be causing an issue, at which point it’ll be time to log a call with Microsoft.

[Updated 24/12/2013] As of vSphere 5.5 and vCenter 5.5 VMware has tested and now supported the use of SQL Server Database for vCenter on a traditional Microsoft Failover Cluster solution. This does not however include the use of Always on Availability Groups. Support for Always on Availability Groups may be included in a future release. Two KB articles cover this topic area. KB 1024051 Supported vCenter Server high availability options (Always On Availability Groups would only be covered by third party support as per other cluster configurations), and KB 2059560 Enabling Microsoft SQL Clustering Service in VMware vCenter Server 5.5.

Final Word

So we are all clear. SQL Server 2012 AlwaysOn Availability Groups is not currently tested, validated or supported by VMware for use with VMware’s products. This situation may or may not change in the future. If you wish to use this type of availability technique to protect your VMware products, such as vCenter DB, I would strongly recommend extremely thorough testing and that you weigh up the support risks. It would likely be a lot cheaper and easier to use vCenter Server Heartbeat, which would be my recommended solution for availability of the vCenter Server Database at least, and the other DB’s that vCenter Server Heartbeat supports. SQL Server 2012 AlwaysOn Availability Groups is supported to run on top of the vSphere 5.1 platform to provide database services to any application that supports this method of availability. As always feedback is appreciated.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

This is just a quick article to let you know that VMware vSphere 5.1 now appears on the Microsoft SVVP list. There have been quite a few customers asking about this. It didn’t impact support of any customers that have a Microsoft Premier Support Agreement, but could have impacted customers that don’t have a Premier Support. Anyway, to cut a long story short VMware and Microsoft sorted out the holdup and now got the list updated. You can check out the new entry here – Windows Server Catalog – VMware vSphere 5.1.

[Updated 25/06/2013] A couple of people have mentioned that I haven’t explained in this was the Microsoft SVVP is. So here goes. SVVP stands for the Server Virtualization Validation Program. It’s basically Microsoft’s way of validating hypervisor technology with their server products. But as I’ve said above it’s only relevant if you don’t already have a Premier Support Agreement. So know you know what SVVP is.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Upgrading to vSphere 5.1 has been a bit of a hot topic ever since it became a GA product in 2012. The problems some customers have experienced during the upgrade process to vSphere 5.1 have been well documented and VMware certainly received a lot of feedback which was taken on board. But many upgrades still went through without any major dramas at all. I completed a couple of small scale upgrades without any problem just after the GA. The real trouble was when you started to try and integrate into larger scale and more complex environments with more complex requirements. So is now the time to upgrade if you haven’t already?

The short answer is yes. VMware has released vSphere 5.1 Update 1 (and an additional patch) and vCenter 5.1 Update 1A. These updates provide fixes to the majority of problems that were experienced with the GA release. Now that this release is out I think it’s a good time to upgrade if you haven’t already. You can read the release notes here for a full list of what is included. If you need a reminder of what’s new and improved in vSphere 5.1 and why you should move to it you can check out the What’s New in vSphere 5.1 document.

If you do choose to upgrade I recommend that you consider the best options for deployment of Single Sign-on and keep your architecture as simple as possible. There are good reasons to split out Single Sign-on from the vCenter Server in some cases, such as if you have multiple vCenters. I would recommend keeping the other services such as Inventory Service on the vCenter Server itself. If you need high availability configurations in addition to what is provided by vSphere HA for vCenter and SSO you should choose to use vCenter Server Heartbeat. If you don’t already have vCenter as a virtual machine running out of a Management Cluster then now is the perfect time to introduce this best practice to your architecture. If you want to use Linked-mode then you will need to configure a multi-site SSO configuration and I would recommend that you test it in a test environment before implementing it in production.

Upgrading from a previous version of vSphere to 5.1 really isn’t a minor upgrade as it requires an architectural shift due to the introduction of SSO. This requires some planning and preparations in order to be successful. But this is the ideal opportunity to review your architecture and your designs and make sure they will support your requirements for the next 3 to 5 years. It’s also the ideal time to start designing an architecture that will support Monster VM’s. In the long run SSO improves security and integration for the vCloud Suite and will make it easier to administer. Future upgrades should also be much less onerous.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

A little while ago I wrote an article titled 5 Tips to Prevent 80% of Virtualization Problems. This article was all about storage and how to configure your storage and the dangers to watch out for. This is because problems in virtualized environments are predominantly caused by or related to storage in one way or another. In that article I explained the impact of queue depths on performance and also some of the dangers of making the HBA device queue depths too high. What I didn’t know at the time I wrote the previous article was that the default queue depth for QLogic HBA’s was changed between vSphere 4.1 and 5.x. This article will being you up to date on the changes and the impacts of the change in default values between vSphere 4.x and 5.x.

The HBA device queue depths are important as I outlined in 5 Tips to Prevent 80% of Virtualization Problems because it has a big impact on the number of parallel IO’s that a VM can issue and that can be serviced. It also has an impact on the number of LUNs that you need to support to achieve the same performance. If your queue depth is too low you can have very high latency when your VM’s are trying to issue a high number of parallel IO’s as they’ll all queue up inside the hypervisor. If your HBA device queue depth is too large you could have lots of IO’s either queueing up in the HBA itself, or you could overload your storage array. So you need to strike the right balance.

This article is a follow up to Cormac Hogans article titled Heads Up! Device Queue Depth on QLogic HBA’s that was published in response to some queries VMware had received from one of their Technical Account Managers. I would recommend you read Cormac’s article for the reasons why the change to the defaults was made, as I won’t cover that here.

The following are the default HBA device queue depths when using QLogic HBA’s for FibreChannel or FCoE SAN connectivity:

• ESXi 4.1 U2 – 32
• ESXi 5.0 GA – 64
• ESXi 5.0 U1 – 64
• ESXi 5.1 GA – 64

Note: This change does not effect Emulex HBA’s, only QLogic.

The VMware KB 1267 – Changing the queue depth for QLogic and Emulex HBAs, which documents the process for changing device queue depths for QLogic and Emulex HBA’s has been updated to include the default queue depths for the adapters per vSphere version.

So is this change to the default really significant? I think it’s significant in that it wasn’t documented anywhere and in fact the QLogic HBA documentation still lists the default as 32. It’s also significant due to the impact of an overload condition can be quite a dramatic negative storage performance hit, which could take a while to troubleshoot. But for a very long time it had been a common best practice for VMware to recommend changing the HBA device queue depth on QLogic HBA’s to 64 from the default of 32. In most cases this had a positive impact on performance with reduced IO latencies. If you are using Storage I/O Control it will dynamically adjust queue slots between different VM’s on a shared datastore and you don’t need to worry about the device queue depths.

Storage I/O Control takes away the worry and will adjust performance to ensure the latency thresholds are met (by default 30ms). If you have vSphere Enterprise Plus (4.1 and above) and you have Multiple VM’s per Datastore you should be making use of Storage I/O Control. The device queue depth is used when there is only one VM per datastore and Disk.SchedNumReqOutstanding is used when there are multiple VM’s per datastore, in which case the per device queue depth is ignored. As Paudie O’Riordan, one of VMware’s Senior Staff Technical Support Engineers says “let the computer (SIOC) make the decision, not the finger and the wind”. 

However there are a few cases where the queue depth of 64 had a detrimental impact and that was largely when non-virtual systems were sharing the same storage array as the vSphere hosts. In this case the vSphere hosts got a far larger proportion of the array’s IO resources and this could impact the performance of the non-virtual systems. I would recommend that where possible you don’t share storage arrays between your virtual and non-virtual environments, which would avoid these types of impacts. In cases where that is not possible you will need to carefully consider the quality of service and storage IO isolation requirements and impacts that high performance vSphere hosts could have on the overall storage array.

The Queue Depth for all devices on the QLogic HBA is a total of 4096. So if you have a per device (per LUN) queue depth of 32 you can support 128 LUN’s at full queue depth, without queueing in the HBA. If you increase the queue depth to 64 (as is the new default in 5.x) then you can support only 64 LUN’s at full queue depth. You can still have more LUN’s configured based on the assumption that not all LUNs will be using all the queue depth all at the same time, so you can effectively overcommit queues in essence. But it would pay to consider the impact of a large queue depth if all VM’s do start issuing IO’s. As Cormac says in his article “If you hit the adapter queue limit, then you won’t be able to reach the device queue depth, and may possibly have I/Os retried due to queue full conditions.”

Now this is only for the HBA queue depths. What about the target ports or storage processor ports on the array? A lot of array storage processor ports will have a queue depth of 2048. You should check with your storage vendor what the Target Port Queue Depth is for your array, if any. As you can see if the HBA is only configured to issue IO’s to one target port a single HBA could easily overwhelm the storage processor and this could cause a QFULL. Fortunately your design should have LUN’s configured across multiple target storage processor ports and multiple storage processors to reduce the risk of overloading. So what happens in a QFULL scenario? Well you can read the QLogic Document titled Execution Trottle and Queue Depth with VMware and QLogic HBA’s. In essence the vSphere Host will set the queue depth to the minimum, which is 1.  You can just imagine what this would do to your performance.

Final Word

I recommend you read Cormac’s article Heads Up! Device Queue Depth on QLogic HBA’s and read the QLogic document Execution Trottle and Queue Depth with VMware and QLogic HBA’s. Overall the change in default queue depth for QLogic HBA’s should be positive for performance in most environments. In some environments however you may need to adjust the settings to reduce the risks that I have outlined here. It’s far better to be armed with this knowledge than suddenly have your storage performance fall off a cliff and not know what might have caused it. If you have vSphere Enterprise Plus (4.1 or above) and you have Multiple VM’s per Datastore you should be making use of Storage I/O Control. 

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

One of the most important documents for any vSphere administrator or architect has been released. The vSphere 5.1 Hardening Guide is now available. The guide was announced on the vSphere Blog by Mike Foley – vSphere 5.1 Hardening Guide – Official Release. I’d like to thank Mike and the rest of the VMware Security Team that was involved in putting this invaluable resource together. It has been reformatted from the previous version to make it easier to use. I think you’ll all like the new improvements.

This is an essential resource for enterprise and secure environments. I also hope that all VMware customers take the time to review this as a lot of the recommendations can be easily implemented and greatly improve the security of your environments. Of course this isn’t the only measure you need to take to protect your environment, keeping up to date with patches of your VMware software is also important as part of your overall security strategy.

Please bare in mind that the guide has been created to be applicable to all VMware environments, but that not all settings or recommendations will be relevant to all environments or all customers. You should review the settings and make a determination as to which settings are applicable to you. VMware has made this process easy by defining different profiles within the guide. So you should look at the profile (explained on the intro page of the guide) that best fits your environment and then review the settings that fit into that profile.

The guide itself is available here. The change log is available here. The central location for all of the VMware hardening guides is http://vmware.com/go/securityguides and this will be the permanent home for the vSphere 5.1 hardening guide also.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

The number of enquiries I’ve been receiving regarding Microsoft Failover Clustering, especially for Microsoft SQL Server Databases has skyrocketed in the past few weeks. I have been receiving a number of enquiries from customers and also from partners including cloud service providers. As a result I thought I’d write this article to help you understand what the current status is of support for Microsoft Failover Clustering on VMware vSphere 5.1 (GA) and with regard to some VMware products.

Background Reading

Firstly there are two main VMware knowledge base article that outline the support statements of Microsoft Failover Clustering and Microsoft Cluster Services on VMware vSphere. They are as follows:

Microsoft Cluster Service (MSCS) support on ESXi/ESX (1004617)

Microsoft Clustering on VMware vSphere: Guidelines for Supported Configurations (1037959)

This article only applies to vSphere 5.1. The rule book has been rewritten with vSphere 5.5, check out my article on vSphere 5.5 Windows Failover Clustering Support.

Clustering and VMware Solutions

In addition to the above there are specific mention of clustering configurations for the VMware technologies that support it, such as for the vCloud Director SQL Database, which was introduced in vCD 5.1 and covered in my article Clustering Support on vCloud Director and vCenter Databases. The golden rule is this. If VMware does not specifically document a clustering solution as being supported then it is NOT supported. vCenter Server from version 4.0 to current 5.1 GA does not support a clustered Database, be it Oracle RAC or SQL Server. It has not been tested by VMware and is therefore not supported. This may well change in the future as VMware recognises the need to provide alternative high availability solutions for the vCenter Database and I will update this article accordingly. However currently the supported high availability solutions for the vCenter and its database are VMware HA, and vCenter Server Heartbeat. Clustering of the vCenter Server itself is also not supported by VMware but is covered by KB article 1024051 – Supported vCenter Server high availability options.

Customers with production support who wish to run Oracle RAC for the DB for vCenter (not SSO, as that doesn’t work) can get support from the VMware Oracle Support Team under VMware’s Expanded Oracle Support Policy. But they will be limited by the capabilities of vCenter itself, if any. I do know a number of customers running vCenter DB (Not SSO) on Oracle RAC in an active/passive service configuration and it has been fine for years. Also I expect the official support statement to change in the future as the testing for vCenter and RAC is completed.

Not supported does not always mean something doesn’t work. But it does mean it hasn’t been tested by VMware and therefore VMware can’t stand behind the configuration as a supported solution. If it’s not documented as supported, then it’s not supported.

The Status of Microsoft Failover Clustering Support on VMware vSphere 5.1

VMware has done a lot of work to enhance support for Microsoft Failover Clustering and its predecessor Microsoft Cluster Services on VMware vSphere 5.1 to support larger cluster sizes. You can now support up to 5 nodes in a virtual Microsoft Failover Cluster on vSphere 5.1. This is great news for environments where two nodes was not enough, even when combined with the additional availability of VMware HA. I’ve implemented a number of solutions where Microsoft Failover Clustering was used successfully in the cases where it was justified and within the limits that were supported. Strong justification and support constraints are two things I’d like you to think about as you read further.

You can still do hybrid Physical and Virtual clusters, and you can also still do cluster-in-a-box with VMDK’s (dev / test of cluster functionality itself not for high availability). VMware Site Recovery Manager is also supported to protect Microsoft Failover Clusters from a DR perspective and there are a number of different configurations you can use such as multi-node to single node, or multi-node to multi-node. This really does make DR for the cluster easy, less error prone, and of the recovery plan itself once it is initiated is automated and provides audit reporting. VMware HA is fully supported, however VMware recommends you implement anti-affinity rules to ensure cluster nodes are prevented from start up on the same physical host.

So what are the gotcha’s or caveats I hear you ask? Well there are a few gaps in support that you should be aware of when developing your solution architecture. I’ll also cover some of the other valid options you have for high availability later as well and some of the impacts of using Microsoft Failover Clustering. This list is in no particular order.

1. Clustering Across Boxes (i.e. traditional clustering for high availability purposes) is not supported with the use of VMDK’s or Virtual Mode RDM (vRDM). You must use Physical Mode RDM’s (pRDM) due to the requirement of persistent SCSI reservations.
2. Due to the requirement to use pRDM’s there is no support for doing backups with vSphere API’s for Data Protection (vADP). So you must use in guest agents for backup.
3. The is no support for vMotion or DRS with Microsoft Failover Clusters as they use shared disks and a shared SCSI bus. Any attempt to migrate a cluster node will be met with an error message.  This doesn’t mean you can’t deploy a Microsoft Failover Cluster inside a VMware DRS cluster, because you can and it’s fine, it just means that DRS can’t automatically migrate the Microsoft Failover Cluster nodes automatically because vMotion isn’t supported.
4. Windows Server 2012 Failover Clustering is not supported currently. Period. Not even with in-guest iSCSI. [Updated 21/06/2013] Except with non-shared disk access, in-guest iSCSI, or in-guest SMB storage access. MS SQL Server 2012 on top of Windows Server 2012 with AlwaysOn Availability Groups is supported as it does not require shared disk. See the VMware KB Microsoft Clustering on VMware vSphere: Guidelines for Supported Configurations (1037959) and my article Windows Server 2012 Failover Clustering Now Supported By VMware With Some Caveats.
5. There is no support for Native iSCSI (where an RDM is presented via the host iSCSI initiator or iSCSI HBA to a guest)
6. There is no support for Fibre Channel over Ethernet (FCoE). Even if the FCoE Converged Network Adapter (CNA) presents itself as a normal HBA to the host, the use of this configuration with Microsoft Failover Clustering is not supported. With one exception – Two node cluster configuration with Cisco CNA cards (VIC 1240/1280) and driver version 1.5.0.8 is supported on Windows 2008 R2 SP1 64-bit Guest OS in vSphere 5.1 Update 1.
7. The use of Round Robin Multipathing for your Path Selection Policy (PSP) is not supported.
8. If you are deploying a hybrid physical node – virtual node Microsoft Failover Cluster the Physical Node can’t use Multipathing software.
9. No support for VM snapshots, which is one of the reasons that vADP backups don’t work.
10. No support for Storage vMotion due to the use of pRDM’s.

Some of the above restrictions, especially lack of vMotion and DRS support make it very difficult for cloud service providers that are using vSphere to offer Microsoft Failover Clusters as a service. The reason is obvious. One of the main benefits of having an Infrastructure as a Service is completely non-disruptive hardware upgrades and maintenance. This is not possible with Microsoft Failover Clusters with the current constraints. If cloud service providers wanted to offer a Failover Clustering option they would need to notify customers to shut down their cluster nodes each time the firmware, drivers, or hypervisor version needs to be updated on their hosts. This of course also applies in your private cloud. Downtime would be required on the nodes each time the hosts need to be updated due to the lack of vMotion and DRS capabilities.

Even with the limitation though the advantages of virtualizing your clusters still outweigh the drawbacks. You still benefit from VMware HA, and the performance and reliability you’ve come to expect. You also get the benefit of being able to use VMware SRM for Disaster Recovery.

Options and Alternatives for High Availability

Failover Clustering is inherently complex. It doesn’t always provide high availability either. There are scenarios where downtime is still required and that downtime might be as much as would be expected just using VMware HA. Because the underlying disks are shared any storage loss or corruption will affect the entire cluster. A cluster if very static and hard to move about between hosts or from your private cloud to a cloud provider if you wished. These are some of the reasons it’s not always the best option.

When considering clustering and you think you’re protecting against Guest OS or Host failure think about the last time you saw a Blue Screen of Death (BSOD) from a VM, or you had a host fail. Hardware reliability is greatly improved and most BSOD’s are caused by drivers. With the standard drivers used when you virtualize your servers you are very unlikely to get a BSOD, at least based on the VMware drivers. There will always be exceptions but this is the case based on my experience for the vast majority of workloads, including those with high availability requirements. Microsoft Failover Clustering is not a DR mechanism (generally), so you need additional measures to provide DR, however some of the alternatives can provide HA and DR in the one solution.

Microsoft Failover Clustering can provide flexibility at times around OS patching. But even this use case has alternatives that provide the same level of availability. I give you an option around rolling patch upgrades below.

If you want to provide high availability to vCenter and the vCenter components then the options are VMware HA and vCenter Server Heartbeat. If you are looking to provide high availability to SQL Databases (ones that are not being used for VMware products that don’t support database clustering) then you have a number of options and alternatives, again these are in no particular order.

1. In guest iSCSI initiation. This is fully supported and will still allow vMotion and DRS migration to occur. Please Refer to the VMware Guide Titled – Setup for Failover Clustering and Microsoft Cluster Service – Update 1, ESXi 5.1, vCenter 5.1. The guide reads as follows on page 9 “Use of software iSCSI initiators within guest operating systems configured with MSCS, in any configuration supported by Microsoft, is transparent to ESXi hosts and there is no need for explicit support statements from VMware.”   Although VMware hasn’t specifically tested this with Windows Server 2012 Failover Clustering there aren’t the same restrictions as this is relying on standard in guest support for the clustering, which as per the guide is transparent to ESXi and does not require any specific support statements from VMware. Provided Microsoft Supports it (Direct Guest Initiated iSCSI for Windows Failover Clustering), which they do, then it’s fine. I would still recommend in guest agents for backups in this case. Incidentally Cisco has a great guide on how to deploy this configuration – Microsoft SQL Server 2012 Failover Cluster on Cisco UCS with iSCSI-Based Storage Access Deployment Guide. This option allows more than the 5 cluster nodes supported by vSphere normally and in fact you could configure a cluster up to the maximum number of nodes supported by Microsoft.  This option is not supported with the use of VM snapshots. You can read more about Snapshot Limitations Here.
2. If you’re wanting high availability above 99.9% for an application such as Exchange or SQL Server you can use the built-in replication technologies such as DAG’s, Database Mirroring, Log Shipping, or Always On Availability Groups depending on the version. These are fully supported by VMware, have full support for vMotion, DRS and HA, and can also provide a DR mechanism. They also are supported with use of VMware vSphere API’s for Data Protection (vADP) for backups. DAG’s and Mirroring or Always On Availability Groups can be used for high availability as well as disaster recovery. The failover can also be completely automated. They also provide additional protection against disk based corruption where clustering would completely fail. You should check if your software vendor supports the Microsoft SQL Client (if using SQL Server) and these automated failover options. Unfortunately VMware doesn’t support Database Mirroring or Always On Availability Groups at this time for SSO or vCenter databases.
3. You could choose to keep things simple and just rely on VMware HA. This is a very viable solution for up to 99.9% availability. This is a great solution for the vast majority of cases. I know of a single VM with 528GB RAM and 32 vCPU’s being protected by VMware HA and it runs the entire SAP system and Oracle DB for a very large organization and has done so reliably and performed exceptionally well meeting their SLA’s. When at all possible I recommend keeping things simple. Unjustified and unnecessary complexity adds the risk of downtime and higher probability of human error.
4. If you need more availability than VMware HA alone can provide  you could add VM and Application Monitoring and Application HA. This will cover the cases of individual application services failing within the guest.
5. To cover the use case of failover during in guest patching you can use vCenter Orchestrator in combination with hot add and hot remove and clone operations of virtual disks to patch the OS disks or application disks of a single VM while it’s still running and then fail over to the patched version. This requires some advanced understandings about how the OS and hypervisor work together and would best be done along side VMware PSO, but it is possible. This would achieve very similar availability profile during the rolling patch process as Failover Clustering would.
6. If you wanted to build a Microsoft Failover Cluster inside of a VMware vCloud Director vApp you could also achieve this. You would need to create the cluster nodes connect them to the shared storage by using in guest iSCSI initiators. They could either connect through an external network out to the iSCSI storage or you could make an iSCSI Target VM as part of the vApp. This would give you Microsoft Failover Clusters as a Service inside an Infrastructure as a Service environment running on top of vCloud Director, with self service, and on demand. There would definitely be some scripting involved but this could be a viable solution, and with orchestration you could also set appropriate anti-affinity rules each time one of these clusters was deployed. No restrictions on HA, vMotion or DRS, it would just work. This option allows more than the 5 cluster nodes supported by vSphere normally and in fact you could configure a cluster up to the maximum number of nodes supported by Microsoft. This option is fully supported by VMware. The idea of using an iSCSI VM inside a vApp came from Andrew Mitchell (@amitchell01 also a VCDX) a colleague from the VMware APJ CoE. This option would not support VM Snapshots. Supportability of vADP for backups is unclear as vADP does get around some of the same limitations for snapshots. But this is not recommended as an option for Production vApps, only development and testing. 

Final Word

This article covered the current status as of vSphere 5.1 GA. It is very likely that improvements will be made in future releases to address some of the limitations highlighted above. VMware understands what it needs to do in order to deliver the Software Defined Datacenter and to support Business Critical Applications. I’m sure they’re already working hard to improve platform support for Microsoft Failover Clusters, even if it is only needed in a very small minority of cases. In the meantime my recommendations are to use VMware HA and VM Monitoring or App HA unless there is a very strong justification for something in addition to this. If you have that strong justification then leverage the built in application high availability and protection options. Failover Clustering due to it’s complexities and risks is a last resort and inferior in most cases to application level HA.

[Updated 10/09/2013] As of vSphere 5.5 Microsoft Windows 2012 Failover Clustering is fully supported using Fibre Channel, FCoE, iSCSI or any of the in-guest storage IO access methods. Failover Clustering is also supported for the vCenter Database as of vCenter 5.5.  I cover the enhancements to clustering in more detail in a separate article – vSphere 5.5 Windows Failover Clustering Support.

I’d be very interested to get your feedback on this article and hear some of your experiences running Failover Clusters in VMware vSphere.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

I’ve been getting feedback and questions from a number of different places of people wanting to disable Single Sign-on in vSphere 5.1 for various reasons (with vCenter). This is mainly due to difficulties around implementation of SSO in combination with other VMware solutions, such as VMware View, vCloud Director. My response to the questions is very simple. DON’T DO IT! At least not with vCenter itself. vSphere 5.1 and vCenter was not designed to run without SSO and this is definitely not supported and will likely result in a broken environment. This brief article will give you some tips on how you can be successful with SSO.

Firstly some reference material you should read:

VMware KB; vCenter Single Sign-on FAQ: Q: Can I disable SSO and revert to the old method of authentication in vCenter Server? A: NO.

VMware KB: Troubleshooting Issues with Single Sign-on in a VMware View Environment

VMware vCenter Blog: vCenter Single Sign-on Deployment Options

VMware vCenter Blog: vCenter Single Sign-on Availability

vSphere 5.1 Gotcha with Single Sign On (SSO)

vSphere 5.1 Generally Available – Important Upgrade Considerations

VMware KB: Backup and Restore of vCenter Single Sign-on Configuration

VMware Support Insider: SSO Best Practices and KB’s

Tips for Single Sign-on Success

1. Keep it simple. Put SSO on the vCenter if you have a single vCenter. The resource consumption is minimal and it is easier to protect and manage. If you have a multi-vCenter environment consider putting SSO on a separate standalone VM, potentially protected with vCenter Server Heartbeat, else just use vSphere HA. For most environments vSphere HA will be sufficient for availability requirements. Only use a multi-site configuration where it is absolutely essential and if necessary get VMware Support and PSO advice.

2. Read the vCenter Single Sign-on FAQ and references above.

3.  Do not cluster the vCenter Single Sign-on Database, Database clustering isn’t supported and won’t work (instead of sometimes where it isn’t supported but does work). Both MS SQL deployed on a cluster and Oracle RAC are not supported at the moment for vCenter Single Sign-on. You can protect the SSO Database by using vSphere HA or vCenter Server Heartbeat. As the DB is small and not very resource intensive you could have it on the same system as SSO itself.

4. Make sure you’re running the latest version of vCenter and Single Sign-on. A lot of enhancements and bug fixes have been provided in vSphere / vCenter 5.1b.

5. Test your deployment of vCenter Single Sign-on thoroughly in your lab or test environment first. Make sure you are specifying the correct Windows AD DN and credentials. Be aware of how SSO works in a multi-domain / multi-forest environment. It would be a good idea to maintain a separate lab or sandpit from your production vSphere environment for testing purposes (doesn’t have to be big at all).

Where can you disable SSO? 

You may  disable SSO integration from vCloud Director. SSO is only used for vCloud administrator authentication, not tenant authentication. If you already have a management AD for vCloud Director authentication then SSO integration is optional. If SSO authentication is used client devices will need access to the SSO system and this would normally be blocked by firewalls. This may complicate the solution. Be aware though that (from what I understand) if you disable SSO you won’t be able to use SAML. So this should not be done lightly, and in any case should be thoroughly tested.

Final Word

Don’t disable SSO in vCenter / vSphere 5.1. Keep it simple. That way you’ll have less problems and you’ll benefit from the integration that SSO provides now and in the future. As always feedback is appreciated. If for some reason you want to not use SSO for vCenter right now the only supported method is to stick with vCenter / vSphere 5.0 U2. If you are on an earlier versions of vSphere than 5.0 I would strongly recommend you consider upgrading to 5.0 U2, which includes great performance and functionality enhancements.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

vCenter Heartbeat is the only supported and validated solution for providing high availability to vCenter Server, and can also protect the core components that go with it (such as SSO, VUM, Inventory Service etc). I strongly recommend vCenter Server Heartbeat be considered for environments where the management infrastructure availability is critical. I’ve used vCenter Server Heartbeat in a number of implementations. But it hasn’t always been easy to implement. The good news is that now thanks to the team at VMware and VMware KBTV we have a video that takes you through the installation and validation of vCenter Server Heartbeat in a vSphere 5.1 environment. In this article I’ll give you some insight into when you might want to deploy vCenter Server Heartbeat, and you can learn how by viewing the video.

As your environment grows and becomes more critical, as you start to virtualize business critical applications, deploy private or hybrid cloud, or virtual desktops, your management infrastructure such as vCenter also increases in criticality. Loss of access to vCenter can mean loss of major functionality, not just for support and troubleshooting but also provisioning, operations, and monitoring. In a 24/7 environment where availability is paramount this can have a major impact on your organization. This is where vCenter Server Heartbeat comes in. It will protect your vCenter system, and can also protect all the core components like SSO, VUM, Inventory Service, and even the vCenter MS SQL Database, by making it highly available. All of these components can be protected by a single heartbeat license even if they are deployed on separate servers or VM’s. It can be used in two different deployment modes that either provide local-site HA on the LAN or inter-site disaster recovery across the WAN. A list of all the components that vCenter Server Heartbeat can protect were listed in my article titled vSphere 5.1 and vCenter Server Heartbeat 6.5 Deployment Considerations, which also goes into some key design considerations for 5.1 environments.

This video is a first installment of a series answering the most common questions asked by the VMware user community when deploying vCenter Server Heartbeat. Whether deploying in High Availability or Disaster Recovery deployment modes, this video will offer key points, tips and considerations for a successful deployment. The video is originally published on the VMware Support Insider Blog – vCenter Heartbeat Installation and Validation. I hope you get a lot out of this video and watch out for more videos in the series.

Final Word

vCenter Server Heartbeat is the best way to protect and provide high availability to vCenter and many of the core components. As I’ve previously written Cluster of the vCenter Server Database is not supported by VMware as a means of providing high availability, and neither is Database Mirroring. But Heartbeat does so much more than just protect the database, it provides intelligent application level high availability that includes monitoring performance metrics, so it knows when to switch over to the stand by or when to restart a service. With the proper design and implementation vCenter Server Heartbeat can provide worry free high availability protection for your core VMware management components. I recommend you consider vCenter Server Heartbeat for your environment and wish you luck with your implementation if you go down that road.

Two additional articles I’d recommend you read that cover vCenter Server Heartbeat are below:

Clustering Support on vCloud Director and vCenter Databases

Using vCenter Heartbeat to Protect Non-vCenter SQL DB? Think Again!

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

The people in VMware Technical Marketing and Engineering have been very busy as usual and have recently published an excellent and deep paper on the VMware vSphere 5.1 CPU Scheduler. This paper is an update from previous papers that have been written about it. Getting the most out of your CPU’s and tuning the environment for peak performance from a CPU perspective starts here.

If you have a design to know how the VMware CPU scheduler works and why it is so efficient and high performance then this is the paper for you. This will cover all of the essentials like faireness, scalability, NUMA, and optimizations and tuning. If you are responsible for virtualizing Business Critical Apps then this is essential reading in my book. Enjoy.

VMware vSphere 5.1 CPU Scheduler Performance

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2013 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.