In December 2014 I got an early Christmas present from Dell. They shipped me the latest 40G and 10G S series (Force 10) switches so that I could begin to test, validate and document the integration and reference architectures between Dell Networking and Nutanix. I’m starting with a L2 MLAG (Multi-chassis Link Aggregation Group)  configuration and I will work my way through to a full ECMP (Equal Cost Multi-path) L3 configuration including VMware NSX. This will be a journey and I’ll include the different options and their key considerations, and configurations in the eventual white papers that Nutanix publishes. I’ve had a few weeks to configure them and do some initial testing (after the Christmas Holiday break), so I thought I’d write about what I’ve found so far. On the NSX front, you’ll be interested to know that Nutanix already has customers running NSX and that the platforms work extremely well together, as they both scale out linearly and predictably. But we’ll leave NSX specific discussion till another day. This article will contain some highlights without steeling the thunder of the white papers I’m working on.

Traditionally datacenter networks were usually designed with 3 layers, Access – where servers connected, Aggregation or Distribution – where the access switches connected and also normally the L2 demarcation, and Core, where everything was bought together at L3. Spanning Tree Protocol (STP) is enabled and redundant links will be blocked to prevent network loops on the L2 segments. However with this design you are not able to utilize all the links, due to STP, and it can be complex to scale and achieve consistent latency between the different points in the network. However these problems can be solved by taking a leaf and spine architecture approach.

As you can see in the diagram above I have 2 Spine Switches – Dell S6000’s with 32 x 40GbE ports, and 4 Leaf Switches – Dell S4810 with 48 x 10GbE Ports and 4 x 40GbE Ports. The Spine and two of the Leaf Switches are running Dell FTOS (Force Ten OS) 9.6, while two of the Leaf switches are the S4810-ON (Open Networks) model and are running Cumulus Linux 2.5. Check out the details of the Dell Networking Force 10 S series switches.

The reason I like Cumulus Linux, is well, it’s just Linux, but with hardware accelerated switching and routing. If you know Linux (Debian is the distribution it’s based off), then it’s very easy to use. No need for additional training, fits into the same management frameworks, such as Puppet, Chef, Ansible and others. So it’s great to have the choice of this or FTOS on the Dell Networking switches. I found both Cumulus and FTOS easy to use, and the documentation is very good.

The port to port latency on the S6000’s is ~500ns, while the S4810’s are ~800ns (so about 1.3us Leaf to Spine), even when routing. With the overhead in the IP stack of each of the hypervisors and VM’s I’m seeing latency across the network between VM’s of < 90us end to end, this is using standard Intel 10GbE NIC’s, VMXNET3 vNIC and without using the latency sensitive settings or tuning within VMware vSphere 5.5.

The main benefit of using MLAG is that you don’t have to have a whole lot of links disabled to prevent network loops and have STP interfering with it, it’s also very easy to set up. You still have STP enabled to prevent loops during switch boot, but when things are up and running all of the links are available to pass traffic and you benefit from the combined bandwidth. Each switch is managed and updated independently, so doesn’t become a single point of failure, as it could if you’d chosen to stack the switches. Switch stacking might be ok if you needed multiple stacks anyway and the host links to the stack themselves were redundant (you can’t mix stacking and MLAG together). While this does add some slight management overhead, it gains the ability to update the switch firmware independently without causing any disruption. With the standard automation and management frameworks such as Puppet, Ansible, Chef, CFEngine etc, the management overhead is greatly reduced or eliminated in any case.

Here is a diagram of what my lab network looks like at a high level. My lab hosts are directly connected to the leaf switches.

In FTOS an MLAG is called a Virtual Link Trunk (VLT), and in Cumulus it’s called CLAG (Chassis Link Aggregation Group). The process to configure them is fairly similar at a high level.

Configure Out of Band Management Interfaces (after initial switch boot / install)
Create a port channel between the adjacent switches (Spine or Leaf)
Configure Spanning Tree (RSTP)
Create a Peer Link on top of the port channel between the adjacent switches so that inter-switch communications can take place to sync mac addresses etc. Set up a backup address in case the primary fails, this will prevent split brain scenarios.
Configure and enable VLT or CLAGD (examples will follow)
Configure and enable the other port channels, edge ports, VLAN’s, routing etc

FTOS Spine Example:

! Note: Peer Link is recommended to be static port channel not LACP.
!
lacp ungroup member-independent vlt
lacp ungroup member-independent port-channel 100
!
default vlan-id 4000
!
protocol spanning-tree rstp
no disable
bridge-priority 16384
!
vlt domain 1
peer-link port-channel 100
back-up destination xxx.xxx.xxx.xxx <- IP Address of Backup Destination
primary-priority 16384
peer-routing
peer-routing-timeout 1
!
interface Port-channel 10
description Cumulus Leaf-Link
no ip address
mtu 9216
portmode hybrid
switchport
lacp fast-switchover 
vlt-peer-lag port-channel 10
no shutdown
!
interface Port-channel 100
description Peer-Link
no ip address
mtu 9216
channel-member fortyGigE 0/120,124
no shutdown
!
interface fortyGigE 0/112
description Leaf1 – Port Channel 10
no ip address
mtu 9216
flowcontrol rx on tx off
!
port-channel-protocol LACP
port-channel 10 mode active
no shutdown
!
interface fortyGigE 0/116
description Leaf2 – Port Channel 10
no ip address
mtu 9216
flowcontrol rx on tx off
!
port-channel-protocol LACP
port-channel 10 mode active
no shutdown
!
interface fortyGigE 0/120
description Peer-Port 1 – Port Channel 100
no ip address
mtu 9216
flowcontrol rx on tx off
no shutdown
!
interface fortyGigE 0/124
description Peer-Port 2 – Port Channel 100
no ip address
mtu 9216
flowcontrol rx on tx off
no shutdown
!
interface Vlan 500
description Host VLAN
no ip address
mtu 9216
tagged Port-channel 1,10
no shutdown
!
interface Vlan 4000
mtu 9216
!untagged Port-channel 100
no shutdown
!

Cumulus Leaf Example with CLAGD:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5), ifup(8)
#
# Please see /usr/share/doc/python-ifupdown2/examples/ for examples
#
#
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0
address xxx.xxx.xxx.xxx/24
broadcast xxx.xxx.xxx.255
# Spine Link
auto spn1-2
iface spn1-2
bond-slaves swp49 swp50
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
# clag-id needs to be unique on each clag, like a vlt domain id on FTOS
mstpctl-portnetwork no
mtu 9216
# Peer Link to Other LeafSwitch
auto pl
iface pl
bond-slaves swp51 swp52
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
mstpctl-portnetwork no
mtu 9216
# CLAGD Peer Int Config
auto pl.4000
iface pl.4000
address xxx.xxx.xxx.xxx/30
clagd-enable yes
clagd-priority 8192
clagd-peer-ip 172.16.0.2
clagd-backup-ip 192.168.255.12
clagd-sys-mac 44:38:39:ff:00:01
# Switch Port Interface Configuration
auto swp1
iface swp1
mtu 9216

auto swp2
iface swp2
mtu 9216

auto swp3
iface swp3
mtu 9216

auto swp4
iface swp4
mtu 9216

auto swp49
iface swp49
mtu 9216

auto swp50
iface swp50
mtu 9216

auto swp51
iface swp51
mtu 9216

auto swp52
iface swp52
mtu 9216

# Bridge Configuration
#
auto br0
iface br0
bridge-vlan-aware yes
bridge-ports pl spn1-2 glob swp[1-4]
bridge-stp on
bridge-pvid 1
bridge-vids 500
mstpctl-portadminedge swp1=yes swp2=yes swp3=yes swp4=yes
bridge-mcsnoop 1
mtu 9216
# Bridge VLAN
# Host VLAN
auto br0.500
iface br0.500
address xxx.xxx.xxx.xxx/23
broadcast xxx.xxx.xxx.255
up ip route add 0.0.0.0/0 via 192.168.1.230
mtu 9216

The above are examples and not complete configs and you can’t just copy and paste it all and expect it to work in your environment. But it could be used as a starting point.

Bringing this all together. The Dell S6000 / S4810 combination allows you to create a scalable network design that provides predictable and consistent low latency and high throughput from end to end in the network. The configuration of the MLAG/CLAG/VLT is straight forward, and it provides management flexibility. You get to choose either FTOS or Open Networking such as Cumulus Linux as your switching software. With Cumulus Linux, it’s just like Linux, but wire speed non-blocking networking accelerated in hardware and fits into the normal management frameworks. For environments, such as Hyper-converged or Web-scale infrastructure, the network scales linearly, as do the systems that connect to it. Each time you grow, you get consistent, predictable and linear performance.

Here is an example of a high level diagram that might be appropriate for a small scale deployment. In this case the Dell S4810’s are used as both Leaf and Spine switches in a VLT configuration, with Dell N3048 or N3024 providing 1G connectivity. With this you could easily start with a single rack and scale to 8 racks. Each rack would have full 10G and 1G redundant connectivity. With say 24 Servers or Hyper-converged nodes per rack you would be able to support 192 servers across 8 racks. 40G QSFP+ ports are used for Peer Links, while the Leaf connects to the Spine using 40G to 10G break out cables.

Here is an example high level diagram of a medium density Nutanix Web-scale Converged Infrstructure deployment with Dell S4810 Leaf switches connected to S6000 Spine, which could be using FTOS VLT or Cumulus CLAG. As you can see this design is capable of scaling to 12 racks (576 nodes) with the S6000 Spine switches, and up to 52 racks and 2496 nodes, with the Dell Z9500 Spine switches. Dell Z9500 supports 128 x 40GbE ports. Both options have spare 40GbE ports still available for Boarder-Leaf connectivity (Cross Datacenter, or Internet routers etc).

If you wanted a higher density design you can combine S6000 in middle of rack or top of rack configuration for Leaf switches with S6000 or Z9500 Spine. This diagram provides a high level example of a high density configuration on a standard 48U rack. In this example the design would support 10 racks with 880 nodes with S6000 Spine, and 48 racks with 4224 nodes with a Z9500 Spine. With enough ports to accommodate the Boarder-Leaf nodes as well.

With densities in the datacenter increasing and the power consumption of the servers decreasing I can see an explosion of 40GbE Top of Rack (ToR) or Middle of Rack (MoR) Leaf switches. This would also provide an option to easily allow different bandwidth oversubscription models, 6:1, 4:1 etc as requirements change.

So far we’ve covered the Dell Networking switches by themselves, with FTOS and Cumulus, and then some examples combined with Nutanix. Dell is an OEM partner of Nutanix software and delivers the Dell XC Series Web-scale Converged Appliances. With Dell XC you have a number of hardware options for different use cases, and you can build a complete solution, including networking.

The following example shows Dell S4810 Leaf switches and Dell S6000 Spine switches with Dell XC series appliances. These appliances are 2U each and contain one node. Dell XC also has 1U options.

Final Word

Those of you who spend time in a modern datacenter will see I’ve drawn the diagrams with the switch ports facing forward, which is actually backwards. I did this to make it easy to draw, and because I like flashing lights. In real world environments the air flow of the switches would be reversed and the back of the switches, where the PSU is, would be facing the front of the rack, so that the cabling can be nice and tidy. In summary, where predictable, low latency, high throughput, linearly scalable networking is required, Leaf Spine architectures are becoming increasingly common (and are simpler than three tier IMHO). Web-scale converged or Hyper-converged Infrastructure benefits from the low latency, high throughput and linearly scalability the Dell Networking switches can provide. As does network virtualization such as VMware NSX. Dell Networking switches combined with Nutanix appliances, or Dell XC series appliances powered by Nutanix can deliver a unified and simplified high performance virtual infrastructure with greatly reduced complexity compared to a traditional three tier architecture. A great foundation for a private cloud or software defined datacenter.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.