One of the features many people may not be aware of that was released in vSphere 5 is Multiple-NIC vMotion. This is a feature that allows you to load balance a single or multiple vMotion transmissions over multiple physical NIC’s. This is of significant benefit when you’ve got VM’s and hosts with large amounts of memory, as vMotion migrations will complete significantly faster. So your Business Critical Applications with large amount of memory and CPU’s can now migrate without disruption even faster. Below I’ll briefly cover the good and great of this technology and also a gotcha that you need to be aware of.
The Good
I thought we’d start with the good news. With vSphere 5 you can now split single or multiple vMotion streams over multiple NIC’s. Up to 4 x 10Gb/s NIC’s or 16 x 1Gb/s NIC’s are supported. This magnifies even further the already impressive 30% improvement in vMotion performance vs vSphere 4.1.
The Great
It is super easy to set up multi-NIC vMotion. It’s all explained in KB 2007467. To briefly cover the set up.
- You set up multiple vmkernel port groups, each with a different NIC as primary, any other NIC’s as standby or unused, and a different IP address on the same subnet, .
- You then select the vMotion tick box on the vmkernel port.
That’s it.
Very simple. Now single vMotion’s and multiple concurrent vMotions will be load balanced over the NIC’s. There is absolutely not need to configure any complicated LACP or IP Hash load balancing to make this work, there is no need to use Load Based Teaming (Route based on physical NIC load). You can use this with standard switches, no need for distributed switch. It doesn’t even require Enterprise Plus licenses, but as the benefits are mostly with VM’s and hosts with lots of RAM you’re probably going to have Enterprise Plus anyway.
I tested performance of Multi-NIC vMotion with 2 x 10Gb/s NIC’s in my home lab and got almost 18Gb/s when using Jumbo Frames on vSphere 5. Hosts go into maintenance mode so fast you better not blink! I haven’t retested Multi-NIC vMotion again since upgrading to vSphere 5 U1 and the latest patches. I plan to test it when Update 2 or the next vSphere release comes out.
Here is the test results from my previous article. You can see the Multi-NIC vMotion Test at the bottom – vMotion 2 x 10G.
The Gotcha
There is a condition that may occur during long running vMotion operations that could cause all hosts ports configured for vMotion to be flooded with the vMotion traffic (on vSphere 5.0 prior to Update 2). The way I understand it this occurs when physical switches MAC tables start timing out the MAC’s (before the ARP timeout). The reason it occurs is because although the outbound traffic is split over multiple vmkernel ports and multiple NIC’s the ACK’s coming back from one MAC. So after a while the physical network may time out the other MAC’s as it’s not seeing any traffic from them. As the transmissions are still occurring the switches may start flooding every port that is configured for the vMotion VLAN. Because the problem is generated by MAC timeouts around the 5 minute mark you will be more likely to experience this problem with 1G vMotion NIC’s or with 10G vMotion NIC’s that have Network IO Control or QoS limits imposed, as your migrations will generally take longer.
To work around this problem you may be able to adjust the MAC timeout values on your switches, depending on the type of switches you’ve got. The default MAC timeout on Cisco switches is normally 5 minutes. On the Dell 8024 10G Base T switch I’ve got in my lab the Address Aging value defaults to 301 seconds and is adjustable. Be careful if you choose to adjust these values as there may be other consequences, any adjustments should be tested, and only applied to the switches connecting directly to your vSphere Hosts carrying the vMotion VLAN.
VMware is aware of this problem and is working on a fix has released a fix as part of vSphere 5.0 U2. The fix didn’t make it into ESXi 5 Patch 03 that was released on 13/07/2012 (07/12/2012 for those in the USA). I would hope that it makes it into the next vSphere 5 update release. I will have updated this article when now the problem is fixed, and let you know what patch or updates you need to apply. Until then I hope you are able to make use of Multi-NIC vMotion by applying the above workaround. At least configure it in your test environments and see how it goes.
Update (20130103): This issue is fixed on vSphere 5.0 U2. All you need to do is update your hosts to 5.0 U2 and this problem will be resolved. The workaround is no longer necessary.
I have just posted a workaround to this Gotcha in an article Workaround for Multi-NIC vMotion Unicast Flooding in vSphere 5. This workaround however is unsupported. So use at your own risk. It appears to work well on vSphere 5.0, 5.0 U1, and should work with 5.1 GA but hasn’t been tested.
Final Word
If you thought vMotion in vSphere 5 was already fast you ain’t seen nothing yet, till you’ve experienced Multi-NIC vMotion. Even with this slight gotcha it still has some benefits if you can apply the workaround in your environment. Especially with very large VM’s >96GB RAM, and large hosts >256GB RAM, it will significantly help your migration times.
Duncan Epping at Yellow Bricks has done a follow up article on this titled Clearing up a misunderstanding around CPU throttling with vMotion and Multi-NIC vMotion in vSphere 5. I would highly recommend that you read it. As noted in the comments on this article this should not be kicking in under normal circumstances and will only kick in if the vMotion would have otherwise failed. I’ll let you read Duncan’s article for the full story.
This article is also posted at the VMware Blog Site – Support Insider.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com, by Michael Webster +. Copyright © 2012 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
Great Post! I was wondering if you had seen any issue or had any additional information about clock speed throttling when doing large number of vMotions as mentioned by Eric Siebert http://de.twitter.com/ericsiebert/status/21051008…
I know in our environment with 10G interfaces when we put hosts in maintenance mode one of our monitoring applications shows huge spikes in response time to those VMs. Naturally it is safe to say that response time might be impacted. Who are we kidding, there are no pings dropped, but the impact on latency sensitive applications is certainly evident.
I haven't yet had an opportunity to dig into this deeper so thought I would see if you might already have additional information on the subject. Thanks again for the post!
Hi Josh, The technology you're refering to is SDPS (Stun During Page Send). It's only used when the page dirty rate of the VM's is faster then the ability to copy the pages to the destination host. SDPS only kicks in when a situation occurs that would mean a vMotion might otherwise fail. It's a failsafe mechanism. So if you have sufficient bandwidth and you're not migrating workloads with a page dirty rate that would exceed the 10G/s bandwidth then you shouldn't be seeing any issues with SDPS. You may want to consider allowing more bandwidth for vMotion depending on the impact to particular applications you're seeing, or reducing the concurrency of vMotion operations by modifying some advanced settings. Any changes should be tested.
However if you are using QoS or vNIC's in a UCS platform then you may have less vMotion bandwidth then necessary and then in some cases SDPS may kick in. But it is only slowing down the CPU of a VM for a few milliseconds at a time. This should not be sufficient to notice any significant response time impact. If you have very latency sensitive applications then they may require special attention and tuning.
Generally things just work, but for business critical applications and latency sensitive applications they do require a different approach and more care and attention. Let me know what you find when you dig deeper and if necessary get VMware involved. I'd be keen to see what you come up with and exactly what the situation is.
[…] was reading a nice article by Michael Webster on multi-nic vMotion. In the comment section Josh Attwell refers to a tweet by Eric Siebert around how CPUs are […]
Great article. Just wondering though; with this configuration I'm not sure how to configure this to work with a vDS using LBT, NIOC and SIOC.
Do you need a VMkernel port for each chunk of 10GbE you want to allow for vMotion?
If you wanted to guarantee 50% of the 4 uplinks to allow roughly 20Gbps vMotion traffic would you configure the environment somthing like this;
vDS1 – LBT, SIOC, NIOC
Management – 5 NIOC Shares, active uplinks 1,2,3,4
VM Networking – 20 NIOC Shares, active uplinks 1,2,3,4
iSCSI – 20 NIOC Shares, active uplinks 1,2,3,4
FT – 15 NIOC Shares, active uplinks 1,2,3,4
vMotion1 – 30 NIOC Shares, active uplinks 1,2,3,4
vMotion2 – 30 NIOC Shares, active uplinks 1,2,3,4
Hi Paul,
vMotion1 – Active Uplink 1, All others Standby or Unused
vMotion2 – Active Uplink 2, All others standby or Unused
All other port groups, except management, all uplinks active, using route based on physical NIC load. Best practice for management is active/passive with failback set to no, so that you've always got it on the same side of the switch fabric, and it doesn't flip flop around. This reduces the chances of false isolation events a bit more.
No need to configure LBT on the vMotion port groups, the other VM's will move around them. NIOC can still be used to control the quality of service. I generally just use the normal, low, high shares without specifying specific values. It's all relative. As long as the important traffic types get the bulk of the shares for when there is congestion. LBT and NIOC sort out most problems.
Hope this helps.
Thanks. That all makes sense. I really do envy your ability to test this stuff in that epic lab of yours.
Good question Paul, and great answer Michael! Thank you for the very helpful information. I have been looking for an answer to this exact question for our VMware environment.
Great article Mr. W. Keep up the good work & fantastic lab.
[…] The Good, The Great, and the Gotcha with Multi-NIC vMotion in vSphere 5 (longwhiteclouds.com) […]
We have already gone to great lengths to setup our environment using the methods described in KB1004048 using NIC teaming. Aside from the additional configuration, is there any negatives to using this method versus the method you described above? Thanks!
Yes, you will in most cases not actually be able to use the aggregate bandwidth of multiple uplinks as any individual stream will be limited to the bandwidth of a single uplink. With multi-nic vMotion you can effectively use both NIC's all the time. You are also now restricted to only using IP Hash as your load balancing algorithm, and all port groups on the vDS must be set to use it. For this reason multi-nic vMotion and IP Hash load balancing a mutually exclusive. This won't be a problem if a single link is sufficient for vMotion traffic. You will need to ensure you use Network IO Control to prevent any one traffic type flooding out the others. You can read about the problems with EtherChannel and LACP vs load balancing based on physical NIC load here – http://longwhiteclouds.com/2012/04/10/etherchanne…
[…] Unicast Flooding with Multi-NIC vMotion is targeted to be fixed in vSphere 5.1 U1 (and 5.0 U2) – see The Good, The Great, and the Gotcha with Multi-NIC vMotion in vSphere 5 […]
I'm wondering what the difference will be with a link failure between standby or Unused. If I have it set to standby and a link failure occurs during a vMotion, the vmkernel will get reassigned and the vMotion should complete successfully.
But what about if it's set to unused? What will happen to the vMotion that is using that path during a link failure, will it timeout and fail? Or will vMotion stop trying to use that VMK?
If on your vmk vMotion port the other uplinks are set to unused and the active uplink fails the vmk port will loose network access. But the remaining vmkernel ports configured for vMotion will continue the vMotion operations and provided there is still one surviving vMotion vmk port the vMotion operations will complete successfully. You can set the other uplinks to standby instead of unused. But given the redundancy at the vMotion vmk level there isn't necessarily a requirement for that. I think Multi-NIC vMotion will be far more heavily used once the unicast port flooding situation is addressed.
[…] my previous article “The Good, The Great and the Gotcha with Multi-NIC vMotion in vSphere 5” I discussed an issue that could cause unicast port flooding. One of my large financial […]
just great man! you might brought "The Gotcha" part as it's own article, it worth it. Great explanation of what is going on. thanx!
guys, just be very carefull not to do Etherchanell on two physically separate chassis (switches), unless they for sure support and provide multichassis-etherchanell.
What 10 Gb NICs do you have in your home lab, and where did you get them?
🙂
Intel 520-T2 and 540-T2 and just got them from Dell when I purchased my servers.
Hi Michael
Great Post! Though I’ve a question,
Say I keep my vMotion switch with two uplinks with both as active/active adapters(10Gb uplink each).
Now I’m putting a host in maintenance mode & I have say 50 VM’s for vMotion to other host, wouldn’t both the uplinks will still be used?
Maybe uplink1 for vmotion of first 4 VM’s and other uplink2 for vmotion of other 4VM’s at a time eventually vMotioning all the 50VM’s using both uplinks?
If that’s the case then I’m still utilizing both the uplinks for vMotion even when they are in active/active mode, then how setting up them as Multi-NIC vMotion helping me in multi VM’s scenario?
If you could explain me on this it’ll be appreciated.
Thanks in advance.
Hi Sam, The answer will depend on the teaming method, but assuming you've configured the portgroup with route based on virtual port ID the answer is no, it will only use one uplink. This is because you only have one VMKernel Port, it only has one mac address, and therefore will only use one uplink port. If you had the appropriate switch configuration you may be able to use LACP or Etherchannel on the port group with the one VMKernel Port, but this doesn't guarantee both uplinks are used, but there is a possibility they would be based on the hashing algorithm used. But this configuration potentially adds additional complexity to overall design and operations. It's much simpler to use the normal multi-nic vMotion configuration with two port groups, each with one VMKernel Port, and one uplink active. This will then allow for the hypervisor to load balance both NIC's and doesn't require any special switch configuration.
Thanks
I’ve two scenarios, if you could answer based on that:-
Scene 1
I’ve two hosts one with multi nic vmotion configuration and another host with just one nic.
Say Host 1 with “vmotion-1” and “vmotion-2” port groups and host 2 with port group named “vmotion”.
Will the vmotion will still work and utilize both nics of host 1 coz somewhere i remember I read the port group for vmotion name needs to be same across all the hosts?
Scene2
I’ve two hosts one with 2x10G adapters for vMotion vSwitch and another host with 1x10G & 1x1G adapter for vMotion vSwitch.
So can I still use both adapters on both the hosts with Mutli-Nic vMotion configuration even though one host has 2x10G adapters while other is a mix of 1G & 10G adapter.
Is it good to use 1G adapter on the other host in Multi-Nic Configuration or I should just not use that coz I guess 1x10G adapter will move machines faster then machine moving from 10G & 1G ?
Hi Sam, It's not a good idea to mix 10G and 1G adapters active on the same port group as you may get inconsistent performance. It's also not a good idea to have that sort of configuration for vMotion. Whether it works or not, well that's a different story. As for the case of one host with multi-nic vMotion on 10G and another host with just single nic vMotion, I see no reason why that wouldn't work or be supported. You'll only ever get a max of 10G throughput though for transmissions between those hosts. The best idea is to keep the configurations between hosts as consistent as possible and if you want to use multi-nic vMotion also use Network IO Control and use 10G NIC's if possible.