If you have an application that needs very high service levels for availability (99.999%), 24/7/365, including maintenance and patching, then OS Clustering with shared storage is a proven solution. However it can be very complex to set up and maintain Fibre-channel, iSCSI or direct attached shared SCSI solutions. In some cases complex configurations are required not just in the hardware, but also the Operating System of the Guests. If you add in virtualization to the mix, the complexity level can increase more, with the need for physical mode raw device maps. Ironically, increased complexity can decrease overall availability, especially with an increase in the probability of human error. So how do we increase availability, decrease complexity, and provide a simple solution for OS Clustering with shared storage?

Nutanix AOS 5.17 with AHV has the answer. Nutanix AHV allows shared storage for Guest OS Clustering without any complex back end storage support or configuration, unlike other hypervisors from leading vendors, which still require Fibre-channel storage if you wish to use virtual disks. From Nutanix AOS 5.17 onwards you are able to configure a shared volume group and directly attach it to 2 or more VM’s and set up Guest Clustering without any complex in guest OS storage configuration at all. There is no complex storage back end, as that is all provided automatically by Nutanix AOS, and no in guest storage configuration, that might ordinarily be required if using iSCSI. This makes the use of Guest Clustering incredibly simple, as well as being very easy to automate, and significantly less difficult to support and troubleshoot.

The process for creating a Guest OS Cluster has 3 main steps:

1. Create 2 or more VM’s with the OS of your choice
2. Create a Volume Group with the number and size of virtual disks that you want for your clustered applications and attach it to the VM’s
3. Configure the clustering software inside of your chosen OS and deploy the applications

Here is an example of how the Volume Group might look in the storage section of Prism for AHV:

While it’s possible to have up to 256 vDisks or Volumes within a Volume Group it is recommended to have 32 or less. If you need more Volumes you can create more Volume Groups.

When you attach a Volume Group to VM’s they will be listed in the Volume Group page within the Storage section of Nutanix Prism Element as follows:

If you wish to have a mixed virtual + physical cluster you can choose to enable external client access to the Volume Group. Any physical / external clients can then use iSCSI Initiator to connect to the clusters Target Data Services IP (DSIP) and mount the volumes.

In the example above I created a 4 node Windows 2016 Cluster, which will host SQL Server 2016 as the primary application. The VM’s are listed below, along with an AD Domain Controller:

After the Failover Cluster Manager components and tools are installed you can configure the Failover Cluster. Note: as part of the cluster creation a verification wizard is executed to ensure compliance with the strict rules needed to form a cluster, including shared storage tests for SCSI fencing and persistent reservations. The nodes in this case were displayed as follows within Failover Cluster Manager:

The next step is to install SQL Server on the cluster nodes and assign all the necessary dependent resources, which would look like the following:

I installed a second SQL Server instance in the same Failover Cluster so I could do comparisons between different configurations. You can see that in the image below:

After I created the cluster I did a series of tests including using tools such as HammerDB and Benchmark Factory for Databases. During the tests I performed live migrations to ensure the cluster didn’t blink in spite of the load, and it worked flawlessly.

Final Word

Nutanix AOS 5.17 and AHV makes creating guest clusters simple and quick and supports both Linux and Windows Guest OS types. You can now configure your fav clustering solutions without the traditional complexity and that means it’s way easier to automate. A Nutanix AHV cluster can now support any number of cluster nodes supported by the OS vendors. The next step in the evolution of this will be when AHV supports Metro Cluster across sites, along with volumes, which will allow for geo distributed guest clusters with greatly reduced complexity compared to the traditional implementations. The cluster example in this article with Windows 2016 and SQL Server 2016 was created just by following the standard Microsoft Documentation and directly attaching a Nutanix Volume Group on AHV directly to the 4 Windows VM’s that would form the cluster. That’s it, no special tuning or complexity needed.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster. Copyright © 2012 – 2020 – IT Solutions 2000 Ltd and Michael Webster. All rights reserved. Not to be reproduced for commercial purposes without written permission.

HammerDB is a very popular benchmark tool for testing multiple different database engines, including Oracle, SQL Server and PostgreSQL. This is a brief article to bring your attention to some ways you may improve your results and get more valid benchmark data.

We do a lot of testing with HammerDB at Nutanix. We run multiple database engines through HammerDB for each release of software we release. One of the great things about running databases on Nutanix is that once you find out how much performance you get from a given configuration of a node or database, you can scale it linearly and get the same experience. This fact was demonstrated by a test that Gary Little (Performance Engineering at Nutanix) did some time ago and published in his article SuperScalin’: How I learned to stop worrying and love SQL Server on Nutanix – Recommended Reading. Over the years we’ve found some things when using HammerDB that can improve consistency and reliability of test results.

Sometimes SQL Server might only use a single NUMA node due to the way connections and transactions are assigned in the engine. Gary describes the problem and solution in his article SQL Server uses only one NUMA Node with HammerDB. The solution is a slight modification to how the HammerDB scripts execute.

Another issue we’ve frequently run into when testing large SQL Server databases that needs lots of cores and memory is that not all the cores get used. Often you might find only 20 CPU’s are being used by the SQL Engine, even though you are using Enterprise edition. This is due to the ISO and installer being used not being the Core Edition of SQL Server Enterprise. You need Core Edition to allow more than 20 CPU’s to be used by the database engine.

Another common issue that we have discovered and Gary has documented in his article HammerDB: Avoiding Bottlenecks In Client, is settings in the actual HammerDB client. Specifically around how the client logs data. Check out Gary’s article for the settings you should uncheck before running a test.

Other basic issues we’ve found is the system under test not being sized properly for SQL Server, or the VM’s running SQL Server not being properly aligned to the NUMA nodes and configurations of the hardware being tested. an 8 vCPU VM isn’t optimal if running on a 12 CPU socket, whereas a 6 vCPU VM would be better. We have found that right sizing the VM’s produces far better results. In this case, 8 vCPU’s might produce less transactions per minute and less new orders per minute and higher response times than if 6 vCPU’s was used. On an 18 core socket, 9 vCPU’s or 6 vCPU’s are also good options. Some people find it difficult to use an odd number of vCPU’s, but the scheduler works fine.

Final Word

Benchmarking SQL Server with HammerDB can help determine what performance you might reasonably expect from a platform running different types of databases and show how a platform scales, but it isn’t exactly the same as your real world workloads. The best way of determining how your workloads will perform is by testing them on a the platform with test cases you have valid comparisons for and under similar conditions. Always account for maintenance tasks, such as backups, stats update, reindex etc and allow headroom for growth. I hope the above help you and wish you all the best benchmarking with HammerDB.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster. Copyright © 2012 – 2020 – IT Solutions 2000 Ltd and Michael Webster. All rights reserved. Not to be reproduced for commercial purposes without written permission.

Firstly thank you to all of the 2,587 attendees that made the trip to Las Vegas for the second annual Nutanix .Next Conference. It was a sold out event and was great to meet a lot of new customers and partners and catch up with some familiar faces from last year. This year my team and I helped put together some of the demos for the day 1 general session, specifically for Oracle and SQL Server on an All Flash Lenovo HX Cluster, in addition to presenting a few of the breakout sessions. There was a clear progression of Nutanix not only being good at VDI, but being great for all virtual workloads, including large SQL Server databases, and now with Acropolis Block Services (Based on iSCSI) being able to support any workloads that for some reason need to remain physical.  There is also a clear trend towards all flash systems, and Nutanix announced that almost all platforms have a configuration option for all flash. I explain why Nutanix has some unique benefits with all flash configurations in Your Network Is Too Slow For Flash And What To Do About It. This article takes a look at some of the highlights from the conference specifically around business and mission critical applications, large databases, and all flash solutions.

This year I had the pleasure of presenting a deep dive databases on Nutanix break out along with Leopoldo Salgado, Director IT Infrastructure from Kellogg. Kellogg runs a lot of different workloads on Nutanix, including Oracle and SAP. Thank you to all those that attended and filled in the survey. It was one of the top 5 break outs of the conference. I was also lucky enough to co-present All Flash, All Apps on Nutanix with Tim Isaacs, Sr Director of Product Management and Chris Cate, CIO of ValPak. This session also made it into the top 10 most highly rated.

For the Business Critical Apps – SQL Server and physical Oracle RAC components of the the day 1 general session we had a cluster of 6 x Lenovo HX All Flash Appliances running Nutanix AHV, and 2 x physical Lenovo 3850-X6 systems, running Oracle Linux 7. This is in addition to the 100 Citrix XenDesktop’s running on the same cluster while it was under load, and one of those Citrix desktops was used for the keynote demo itself. The image below shows the architecture and setup for this part of the keynote.

As luck would have it this was the 13th time we’d run through the demo for the keynote and it turned out to be lucky number 13 as everything went flawlessly. The cluster was under heavy load of approximately 150K IOPS with a mix of the VDI Desktops, SQL Server VM’s running batch jobs and the Oracle RAC environment running a transactional workload. Even though databases generally don’t reduce well with data reduction features in this environment we saw approximately 3:1 data reduction, excluding any smart clones and data avoidance. The image below depicts one of the earlier trial runs for the demo.

Even with all of this load the cluster performed well, the latency was low, and the end user desktops were very responsive. The demo infrastructure was hosted by Lenovo in their labs in Durham, North Carolina. There were a number of us connected into desktops on this infrastructure during the demo to monitor the systems as it progressed. For those interested in watching the video from this demo of the day 1 keynote it starts just on the hour mark and I have included a link to the YouTube video below.

One of the key reasons why we didn’t suffer noisy neighbor problems and why the desktops, SQL and Oracle Databases could coexist on the same cluster is due to the way Nutanix uses data locality to try and ensure reads are accessed locally to where the VM’s exist. This helps reduce network saturation that could otherwise occur on all flash systems, leaving more network capacity for real end user workloads. When you compare the various different flash technologies and their potential impact on network utilization it becomes  obvious that the only way to future proof your architecture is with a system that has data locality as part of it’s core. The image below compares different flash technologies with different network technologies. Without data locality it is impossible to take advantage of modern flash to its potential, and without data locality you also experience the noisy neighbor problems where one workload on one host can interfere with the performance of other hosts and virtual machines in a larger cluster. This is prevented in the Nutanix architecture, due to the way we intelligently place data as close to the applications as possible. If you want to run large database servers then the Nutanix NX8150, Dell XC730-24 and Lenovo HX7500 systems are the best suited to large database workloads.

One of the best things about working on the keynote demo’s this year was that all the functionality and performance that we were showing in the demo as I described above was delivered in the Nutanix AOS 4.7 release the week following the .Next conference. This means that the additional functionality and performance is only a simple one click upgrade away for all of the Nutanix customers. Some of the more futuristic announcements as part of the day one keynote are still in a future release, but the bulk are available today in the GA of AOS 4.7.

In addition to Acropolis Block Services, which allows bare metal workloads such as Oracle RAC to run on the storage of a Nutanix cluster through iSCSI there was another feature released that didn’t get much attention. That feature was TRIM or Unmap, which is now natively available in the Nutanix architecture, and will allow customers to automatically reclaim space when data is deleted from within a guest VM. This one feature provides more useable capacity without sacrificing performance and is yet another example of the continuous innovation that Nutanix bring customers so they can improve their existing investments on the same hardware. The TRIM process runs as a background task in Curator and when the guest VM marks blocks as reclaimed Curator will release them when it runs a partial or full scan. Because of the way this feature has been implemented there is no impact to performance of running VM’s, even when the background tasks are running and reclaiming space.

For those of you who couldn’t make it to .Next Conference and would like to watch the full Day 1 AM Keynote, I have included it below.

There is still way more to come this year from Nutanix. The innovation will not stop. There is another major release scheduled for later on this year and customers and partners will get to hear more about some of the other interesting things we’ve got planned further out by attending our European edition of .Next in Vienna in November. If you pre-register you could save 40% on admission.

Join @nutanix #NEXTConf EMEA Pre Register to Save 40% – https://t.co/rhEkqMbhrp pic.twitter.com/BxON38yulQ

— Lord WebScale Webster (@vcdxnz001) July 5, 2016

As we say goodbye to Las Vegas for this year of .Next Conference US we plan to bring .Next Conference to Washington, DC next year. Mark this date in your diary and be prepared for an even bigger event as Nutanix bring more of the enterprise cloud goodness to our customers and partners.

Let the countdown to @nutanix #NEXTConf 2017 begin! Gaylord National in DC, June 28-30. pic.twitter.com/GAcMVJH6I6

— Howard Ting (@howardting) June 23, 2016

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2016 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

It was good to read recently on Duncan Epping’s blog Yellow Bricks that database clustering support for vCloud Director is added in version 5.1. vCloud Director now supports both Oracle RAC and also Microsoft Cluster Services or Microsoft Failover Clustering for MS SQL Server for it’s database. This was previously not supported in vCloud Director 1.0 and 1.5. But what is the support situation for all of the other components when it comes to database clustering, including vCenter Server?

The announcement regarding clustering support for vCloud Director 5.1 databases can be found in kb 2037802. It is really great that VMware has made it clear what the support situation is with no ambiguity.

Here is my understanding of the current state of play with regard to clustered database support (RAC and MSCS/MSFC) based on publicly available information (or lack thereof), this is for the main components of the VMware Cloud Suite that would generally be deployed in most environments:

Single Sign-On (SSO) – Not Supported, vCenter Server Heartbeat is the supported and recommended solution for database high availability

vCenter Server – Not Supported, vCenter Server Heartbeat is the supported and recommended solution for database high availability.

vCenter Server Virtual Appliance – Not Supported, No Database HA currently Supported

vCenter Update Manager – Not Supported, vCenter Server Heartbeat is the supported and recommended solution for database high availability

vCloud Director – Supported from version 5.1 both Oracle RAC and MSCS/MSFC

vCenter Chargeback – Oracle RAC Only from v1.6.2 (refer release notes)

vCenter Orchestrator – Oracle RAC 11g (refer kb 1022828), or vCenter Server Heartbeat

So what do I mean when I say Not Supported? I mean there is no explicit support statement from VMware that they have tested and verified the configuration with clustered databases at the time of writing this article. VMware will provide best efforts support and still help customers troubleshoot their environments. But like most vendors if they suspect a problem caused by the clustered database or clustering technology will refer the customer back to the vendor of that technology. I know many customers that are running vCenter with Oracle RAC databases at the back end successfully and have never had any problems, provided they set it up correctly and tested it, including failover scenarios (should be active/passive configuration). Likewise with a clustered MS SQL database customers have run that successfully as well with vCenter. In many cases the clustering of the database is completely transparent to the application. But that is not always the case depending on how the components connect to and communicate with their database.

When it comes to components that don’t use traditional ODBC connectivity to the database, such as SSO and the vCenter Virtual Appliance, support for clustered MS SQL databases is a bit more difficult. This seem to be primarily why in my opinion SSO is not currently supported with a MS SQL clustered database. For vCenter Server Virtual Appliance it doesn’t support MS SQL at all, but does support Oracle. Oracle RAC Support for the vCenter Server Virtual Appliance is currently going through validation from what I’ve heard and will in the future be supported.

Remember when it comes to support and VMware if something is not included in public documentation, including the product documentation and KB articles, and is not in the product interoperability matrix, then it is not supported. If the support statement isn’t explicit then you can assume that VMware will do everything they can to help (based on my experience and reading relevant kb’s), but it’s on a best efforts basis only.

I’ve mentioned vCenter Server Heartbeat a couple of times above. Some of you may also remember the article I wrote titled Using vCenter Heartbeat to Protect Non-vCenter SQL DB? Think Again!. To ensure you’re aware of what components are supported for protection with vCenter Server Heartbeat I have included them below. This is directly from the vCenter Server Heartbeat Product Documentation. The case remains that if the component isn’t listed below it is not supported and only the databases relevant to the components below (if any) are supported for protection by vCenter Server Heartbeat.

vCenter Server Versions 5.1

■ VMware vCenter Inventory Service

■ VMware ADAM

■ VMware USB Arbitration Service

■ VMware vCenter Server

■ VMware vSphere Client

■ VMware vSphere Web Client

■ VMware vCenter Update Manager

■ VMware vSphere Update Manager Download Service

■ VMware vCenter Orchestrator Configuration

■ VMware vCenter Orchestrator Server

■ VMware vSphere ESXi Dump Collector

■ VMware Syslog Collector

■ VMware vSphere Auto Deploy

■ VMware vSphere Authentication Proxy

■ VMware vCenter Host Agent Pre-Upgrade Checker

■ VMware vCenter Single Sign On

■ VMware vSphere Profile-Driven Storage Service

■ RSA SSPI Service

■ View Composer 1.1, 2.0, 2.7, and 3.0

Note Remote deployment of View Composer is supported starting with View Composer 3.0

■ VMware View Composer

■ VMware Universal File Access

■ vCenter Converter Enterprise

When considering vCenter Server Heartbeat for your environment I would recommend you get VMware involved in the design and deployment, or at least someone (Partner / Consultant) that is experienced with it’s implementation (blatant plug: like my company). It provides great features and excellent availability but does introduce additional complexity. VMware or your preferred partner can help you navigate the complexity and ensure a robust handover to operations to ensure you get what you expect.

Final Word

Depending on the component there are different solutions to provide database high availability. There is no standard across the board between the different VMware components that make up the new vSphere 5.1 Cloud Suite. I expect as the suite develops we will start to see a lot more standardisation in this respect and to see more availability options. Careful consideration needs to be given to providing high availability when deploying the suite within current constraints. As and when the availability options change I will update this post.

Given that this has been based on the easily accessible and publicly available information it’s possible I’ve got something wrong. If so I’d be happy to correct it, so please let me know.

Note: Anyone who is wondering what MSFC is it’s the new term Microsoft use to name their clustering technology. In the past (Windows 2003 and previous) it was called Microsoft Cluster Services, and now it’s called Microsoft Failover Clustering. Many people use the term MSCS to describe both, but MSFC is vastly improved over the previous MSCS.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.