Comments on: Cores Per Socket and vNUMA in VMware vSphere #vForumAU #vForumSG

By: What is vNUMA?

What is vNUMA? — Thu, 24 Sep 2015 22:05:51 +0000

[…] Sizing Many Cores per Socket or Single-Core Socket Mystery Does corespersocket Affect Performance? Cores Per Socket and vNUMA in VMware vSphere Performance Best Practices for VMware vSphere® […]

By: Marco Law

Marco Law — Fri, 25 Oct 2013 02:58:55 +0000

Hi Michael,
I also have some query on vNUMA.
If I have a box of ESX 2 socket 8 core with 512GB RAM. So each NUMA nude is around 250GB RAM.
If I have 3 VM, each will have 8 vCPU and 160GB RAM with memory reservation due to they are running in JVM, which recommend to have memory reservation. Do you think I should enable the vNUMA for this? As 2 of the VM can't share the same NUMA node memory (2VM already 320GB RAM > 1 NUMA node memory 250GB), so it mean it will across to remote node memory.

Another question what is the draw back for the vNUMA?
Thank you

Marco

By: Sten-TAM

Sten-TAM — Mon, 21 Oct 2013 16:21:27 +0000

Check bios setting on your Gen8, they are delivered with balanced power setting = not full performance.

Set it to optimal or os controled(you can then use vsphere cluster setting to get full performance.

By: Matt H

Matt H — Sun, 20 Oct 2013 05:47:09 +0000

In reply to vcdxnz001. Thanks for the GC note. No contention on the hosts (running < 50% even during these batch operations). Underlying infrastructure, HP C7000 / Gen8 blades, 2x8core 2.7GHz CPUs, FlexFabric, 3PAR T400 storage with ample spindles. Matt

By: vcdxnz001

vcdxnz001 — Sun, 20 Oct 2013 05:34:10 +0000

In reply to Matt H. Hi Mark, GC threads need to be < # vCPU\'s. I would recommend starting with 2 on a 4 vCPU VM. Also check the resource pool shares and CPU ready time. What is the physical hardware running underneath vSphere?

By: Matt H

Matt H — Sun, 20 Oct 2013 05:28:00 +0000

In reply to vcdxnz001.

Michael-

Thanks for the quick reply. The OIM VMs exist in a resource pool which reserves the entire capacity of the memory footprint so there is no need for memory reclamation. The version of Red Hat we are using is 6.4 but at Oracle Open World, we (not me) were told that would not be fully certified. I had my doubts. As for other tuning items, we told the middleware folks to limit the java garbage collection to 1 thread per vCPU core (in our case, 4) and they configured the java heap size (8 GB) to half the RAM in each VM (16 GB). As the only layer I am responsible for is the hypervisor, it is difficult for me to do more than what I am already doing. I may need to do some due diligence or risk the environment being removed from vSphere altogether.

Unfortunately, we do not employ vCenter Operations or Log Insight. We are a fairly new exadata shop, and recently lost our OCP-certified DBA. I will have to look into the DB connection pool issues to which you have referred. The main issue is our current DBA is a huge fan of Solaris Zones on Sun Fires and he was told this app needed to be implemented on VMware by our CTO who has since moved on.

Thanks again for your insight.

Matt

By: vcdxnz001

vcdxnz001 — Sun, 20 Oct 2013 05:08:57 +0000

In reply to Matt H. Hi Mark, The good news is that your products are fully supported running on VMware vSphere as Oracle Certifies down to the OS level, and provided your running a version of RedHat that Oracle Supports you'll have no trouble interacting with Oracle Support Engineers. VMware does not change the OS at all, so it does not invalidate your Oracle Support. Further Oracle has specific support statements for VMware for most of it's products and these can be found in My Oracle Support Online (Formally Metalink). For example Oracle DB ID is 249212.1. Also if there was ever any doubt you could log a call with the VMware Oracle Support team and they'd help you resolve it in the unlikely event you have any issues with Oracle Support directly. Although my interactions with Oracle Support have always been very good. The performance problem is pretty unlikely to only be caused due to the configuration of the vCPU's on the VM's. There is most likely some other factor limiting performance. It could well be the DB connection pool back to the Exadata from the OIM servers. I've seen this many times. Also there could be some Guest OS side things limiting performance. Without analyzing the environment in detail it's hard to say. Do you have any reservations in place on your OIM VM's? Have you done any tuning at the OS Level? Whatever is limiting the performance there will be something in the logs or visible in the OS / vSphere Host or Application Logs. Do you have vCenter Operations and Log Insight to help you with the performance analysis?

By: Matt H

Matt H — Sun, 20 Oct 2013 04:55:03 +0000

This is a very interesting recommendation. We have a large directory services implementation of Oracle Identity Manager that serves half-a-million students. At times we get bulk record update jobs (sent to databases on an exadata) that max out the CPU on our virtual machines. We have been looking at possible performance issues throughout the stack. The VMs were all 2 socket / 1 core CPUs and we were asked to double the CPU on each VM. Due to the fact these are all RHEL VMs, to save on licensing (the hosts are licensed for 2-socket unlimited RHEL), we chose to go 2 socket / 2 core instead of 4 socket / 1 core. Our middleware folks told us that the jobs ran the same amount of time despite the fact each VM had 2x the CPU power.

This makes me wonder if choosing multi-core sockets for our Red Hat virtual machines are part of the problem? Our Oracle DBAs tell us that the exadata isn't the bottleneck. Network and storage are not the bottleneck, either. Of course, running this Oracle application on vSphere-based RHEL VMs are making our UNIX admins cry foul since Oracle won't fully support the application unless it is running on Oracle products from top-to-bottom (physical Solaris, or OEL on Oracle VM).

Pretty frustrating.