25 Responses

  1. marco
    marco at |

    Do you know what effect the jumbo frames have on cisco switching buffers?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Marco, I haven't tested Jumbo Frames on Cisco switches for a while. Last time I tested Cisco equipment the performance was very good with Jumbo (but a bigger difference between Jumbo and Non-Jumbo). But this is very equipment dependent, there are many options with Cisco switches, and not all will perform the same way. Some line cards I know only have full buffers available if you distribute your connections to every 4th port, rendering the other ports unusable. But the assumption is that you won't run every port on the line card at full performance all of the time. A customer recently ran perf tests using JPerf on Windows 2008 R2 64bit with ESXi 4.1 on Nexus 2K into 5K and was getting 14Gb/s using LACP when Jumbo was enabled. Not sure the impact on buffers though.

      Reply
      1. marco
        marco at |

        I'm going to discuss this with my networking guy. We are currently using 2x 10Gb per ESX host, with one active other passive. Cisco 4900 switches. I can remember we turned off jumbo frames as per Duncan's blog about "No Jumbo frames on your Management Network!" and some buffer thing with the cisco's. I think It wass something when you enable jumboframes the cisco's would use less buffers for your normal traffic and will use a lot for the big sized packets. I will ask him to clarify and will get back about this.

        And now that I browsed back to Duncan's post, he corrected it into: Just received an email that all the cases where we thought vSphere HA issues were caused by Jumbo Frames being enabled were actually caused by the fact that it was not configured correctly end-to-end. Please validate Jumbo Frame configuration on all levels when configuring. (Physical Switches, vSwitch, Portgroup, VMkernel etc)

      2. @vcdxnz001
        @vcdxnz001 at |

        I was quite surprised when I first saw Duncan's post on that, as I've been running Jumbo for ages and had no problems with HA on vSphere 5. With your Cisco environment, do check things out carefully. If possible test the implementation in an isolated environment. You may find that you need to upgrade to the latest Cisco software release to get the best performance from Jumbo. I've not done any testing with Jumbo on 4900 series switches, so if you do test it I'd love to hear your results.

  2. Chris
    Chris at |

    Very interesting results. I do recall the 1GbE test where the gains were minimal and in some cases negative. Thanks for performing these tests.

    Reply
  3. Paul Kelly
    Paul Kelly at |

    I know you have some pretty extensive lab kit, would you believe I was going to pose this very question to you today? 😉

    Congratulations, you passed the mind reading test!

    Reply
  4. mikidutzaaMihai
    mikidutzaaMihai at |

    Do you have a graph with CPU usage also? I am curious what was the exact impact on CPU as well.

    Thanks

    Reply
  5. @vcdxnz001
    @vcdxnz001 at |

    I didn't keep the CPU graphs during the tests as my primary goal was throughput differences. I will repeat some tests and include VM CPU Usage. It was quite high, up to 70% if I recall correctly. Jumbo was similar CPU usage, but with higher throughput, yielding more CPU efficiency. Are you more interested in Host CPU or VM CPU?

    Reply
    1. mikidutzaaMihai
      mikidutzaaMihai at |

      Thanks for the reply. I was curious whether there was a significant difference in host CPU usage (i.e. if it's worth enabling for CPU efficiency reasons).

      Reply
  6. Jumbo Frames on vSphere 5 Update 1 « Long White Virtual Clouds

    […] previously posted an article regarding Jumbo Frames on vSphere 5 but was unable to test Jumbo Frames performance on Windows 2008 R2 because of a bug in the VMware […]

  7. Technology Short Take #21 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers

    […] other posts also made it into my list of “things to mention”: this post on jumbo frames on vSphere 5 (with more results from vSphere 5 Update 1) and this post on CA SSL certificates and vCenter […]

  8. Tim Patterson
    Tim Patterson at |

    If you are testing on a pure 10Gb network, how well would a MTU of 15500 (not a typo) compare against 1500 and 9000?

    Reply
    1. @vcdxnz001
      @vcdxnz001 at |

      Hi Tim, my switching equipment currently only goes up to 9216 or 10K, and vSphere only allows up to 9000. So it's not possible to test an MTU of that size. It may have an incremental benefit if an when it's ever su