Comments on: HP Advisory – vSphere Hosts Disconnecting from Network

By: Allen Crawford

Allen Crawford — Wed, 10 Apr 2013 06:58:40 +0000

Yep, we're running the latest SPP from HP, though the firmware is included with the VMware driver and loaded at runtime. So the firmware you see during POST will not match the running firmware if you are using the 5.0.626 driver as it includes a newer version. HP is spot checking some servers as well for the "bad batch" issue but we're still in the middle of that process.

By: @vcdxnz001

@vcdxnz001 — Wed, 10 Apr 2013 06:54:30 +0000

In reply to Allen Crawford. Hi Allen, I assume you've upgraded the on board and add on NIC cards to the latest firmware as well? You might well be right, or it could just be a manufacturing fault with the batch of cards. I've seen it before where a series of components manufactured around the same time all had faults. I hope you're able to get this resolved to your satisfaction.

By: Allen Crawford

Allen Crawford — Wed, 10 Apr 2013 06:26:32 +0000

In reply to @vcdxnz001. Well, we've got those very same BIOS settings implemented already (though I need to verify "enhanced cooling" I suppose). Unfortunately, they are still not stable for us and considering the integrated NICs have the same issues, I am just struggling to believe it is a hardware issue instead of a poorly written driver.

By: @vcdxnz001

@vcdxnz001 — Wed, 10 Apr 2013 05:29:26 +0000

In reply to Allen Crawford. Hi Allen, I have a customer that did manage to get stability out of their NC522SFP's after the latest firmware and driver update. But in the end they still got the cards replaced. Other things I've learned to improve the situation include changing the server BIOS to Static High Performance or Maximum Performance, Enhanced Cooling, and turning off C States. These are normally recommended settings for vSphere servers, but the enhanced cooling really seemed to make a difference to the NC522SFP's as they were prone to getting very hot and more problems would then occur.

By: Allen Crawford

Allen Crawford — Wed, 10 Apr 2013 01:01:56 +0000

Good post, though that HP advisory is a bit of a joke. They still continue to reference an ancient version of the ESXi version (5.0.601) of the nx_nic driver/firmware combo. They are now up to version 5.0.626 on VMware's site and it is still not at all stable. We have the NC522SFP NICs that frequently just go "offline" for lack of a better word. Sometimes that is just really high latency, sometimes it is many dropped packets, and sometimes it is completely loss of connectivity (though we still have a physical link). Only a reboot resolves this. We've got an escalated case open with HP and VMware trying to make some progress, but I'm 100% of the opinion the issue is with the nx_nic driver, written by QLogic. Because the problem also occurs with the integrated NICs on our HP DL580 G7 servers (the NC375i). This is the fourth version of the driver we've used and they all have been awful. Running ESXi 5.0 U2 here. Not sure if it works better with 5.1 or not.

By: @vcdxnz001

@vcdxnz001 — Tue, 09 Apr 2013 18:11:38 +0000

Hi Jhonny, If their servers are already running in production and have done for a long time there is minimal risk and they shouldn't require a burn in test just to do an upgrade. But what I would suggest as part of any upgrade project is that you test the upgrade process and you test the new version of the software in your environment in a test lab or non-production environment, to make sure it meets your needs and that your design meets all of your requirements. It doesn't have to be long or laborious. It just has been to be so you understand the changes. It could be for as little as a couple of weeks. I've upgraded quite a few customers environments to 5.1 and they've had no issues. A little bit of QA can go a long way.

By: Jhonny Nemonic

Jhonny Nemonic — Tue, 09 Apr 2013 18:08:58 +0000

Hi IHAC with this problem, as a matter of a fact they are scare to move from 5 to 5.1, it's recommend to have a bur in test in this scenario? Or even thougth this could happen again?? I mean PSOD

By: @vcdxnz001

@vcdxnz001 — Tue, 09 Apr 2013 09:20:24 +0000

In reply to Stan. The QLogic CNA's have also had the same type of problems. It's a big concern when these two big manufacturers both have stability and reliability issues at the same time. My Broadcom and Intel NIC's have been flawless though.

By: Stan

Stan — Tue, 09 Apr 2013 09:14:27 +0000

I have been dealing the HP engineers on Emulex OneConnect based NICs since Apr 2012. Both HP and VMWare are not responsible in writing firmware and drivers for them, Emulex is responsible. And so far everything released up til Nov 2012 from Emulex had stability issues. Dec 2012 and the more recent Feb 12th release are more stable (at least on HP infrastructures).

Everyone who picked Emulex as supplier has been burnt by this. IBM, Dell and HP all use them. At least with HP, their G8 lines no longer force you to take on Emulex and you can actually choose to go back to Broadcomm.

The problem is worse if you use these Emulex NICs for IP-base storage.

By: @vcdxnz001

@vcdxnz001 — Mon, 08 Apr 2013 21:29:33 +0000

In reply to Jonas Nagel. Hi Jonas, VMware relies heavily on the OEM vendors of the hardware to qualify their products for use with VMware vSphere. The qualification tests may well include burn in testing. But this does not mean every server gets burned in before it is shipped. Also customer burn in tests don't need to take months. 48 hours would be normal. It's a reality that not all servers that come out of the factor are defect free or have the most up to date firmware or drivers. If you want to ensure your environment is reliable, some measure of QA testing of your hardware is important prior to putting it into production. Also hardware these days is largely software and anyone that's been in the software business or is a user of software knows that it has bugs. So why would modern hardware be any different? I've been burned too many times by hardware bugs not to do the necessary brief testing prior to production use. Even today (literally today) a batch of new servers for a customer would not perform vMotion due to firmware bugs with their CNA's. The bugs were only fixed in a fairly recent combination of firmware and drivers. So it's up to you if you follow this advice or not. But I would still recommend it. I would agree it shouldn't be required. But in my opinion it is.