Hi Forbes, some good points you've raised. We actually find apps running on the local node with the CVM can be more efficient as their CPU's aren't waiting around as long for IO and consequently get more value from the local controllers. This is also true of other local caching technologies. Plus this adds more predictability, you know what you're going to get as the architecture scales. The real point is that the hypervisor isn't a bottleneck for IO at all, regardless of the actual underlying architecture. Even for a SAN Device, which has a much longer IO path, the hypervisor isn't a bottleneck. So regardless if you have something in the kernel, or in a VM, the key consideration isn't actually performance. Both approaches use local system resources to service IO. So you're not losing out on those resources, and DRS will still distribute your workloads so they get the resources they require. Also the platform is distributed, so you are getting the benefit of all of those resources across all of the nodes to provide you value, including not having a dedicated rack of disks that can't run VM's.

In answer to your question how much resources does the Nutanix Controller or CVM take up, it depends on how it's been configured, being a VM you have the option to change it's size if you desire and you can have control over it. But the default configuration is 8 vCPU and 16GB RAM. It will only use those resources to deliver the performance if applications require it (proportional to the value it delivers), so under a high load it could use the resources assigned. Also if you want to enable additional features, such as dedupe, you would increase the vRAM to 24GB. Features all have a resource cost, as does running any sort of hyperconverged environment. However the important thing is that the benefits outweigh the costs and the overall efficiency is higher. In the experiment I ran for this article, to show how well the DB workloads scale up and out, my CVM is configured with 8 vCPU and 32GB RAM. The workload is over 70% write, and 100% random, so doesn't benefit from read cache, I saved many TB of storage by using VAAI primitives, and I saved many more TB of storage by using compression, all while seeing great performance. I also produced per transactions per minute that similar tests done on other hyperconverged platforms, demonstrating the value of the approach (while saving 60% storage using compression and much more using VAAI).

My actual problem with this whole in kernel debate is that it harms the credibility of the hypervisor as a place to run business critical applications, which is an area I've focused 10 years on. Getting confidence in customers to run business critical apps on a hypervisor is not a trivial process, and performance is one area that still constantly comes up. If you can't get performance from a VM on the hypervisor (VSA or otherwise), how can you know you'll get performance for the high IO business critical apps. So I think it was a big mistake VMware taking this path with their messaging. I know through experience of virtualizing some of the largest systems on the planet that the hypervisor is a great place to run high IO apps, and I have confidence in ESXi as a platform for great performance of any app, which is one of the reasons I have confidence that it's a great place to run a storage controller. In addition to all the reasons I outlined in http://longwhiteclouds.com/2015/02/27/in-kernel-o….

I suppose the question I ask myself is if guys like Duncan, and Frank (who have probably forgotten more about how the vSphere kernel operates than most know) believe that kernel integration introduces less performance penalty with respect to IOPS (but probably more importantly less latency), I suppose I tend to believe that.

Satyam Vaghani (Godfather of VMFS and various VMware storage innovations) probably has a good idea about kernel mode vs VSA for servicing IO. It’s doesn’t go unnoticed that he choose kernel mode integration for his FVP product, rather than depending on VSA’s.

I think your test works well in a small controlled environment that you setup. I would like to see the results of hundreads of dissimilar workloads. With the Nutanix concept of locality (which is great) the local CVM would be servicing a greater # of transactions. It’s difficult to believe that scheduling wouldn’t start to pose some issues. Enough that latency (perhaps going from microseconds to milliseconds, which when dealing with flash resources can be material) might be an issue.

Lastly, how much resources does each CVM take up? Wouldn’t it be nicer to distribute those valuable CPU and RAM resources to the workloads in the cluster that could use them? By introducing VSA’s I’m having to size my clusters to account for those resources, instead of leaving those resources to my various applications. Would it be fair to ask, if Nutanix had the opportunity during development to use kernel integration vs CVM’s, would they have remained with CVM’s being the best option?