(cas:72) Google Analyticator was unable to authenticate you with Google using the Auth Token you pasted into the input box on the previous step.

This could mean either you pasted the token wrong, or the time/date on your server is wrong, or an SSL issue preventing Google from Authenticating.

Try Deauthorizing & Resetting Google Analyticator.

Tech Info 400:Error fetching OAuth2 access token, message: 'invalid_grant'
Unique
Visitors
Powered By Google Analytics
Cumulus Linux – Long White Virtual Cloudsu by http://longwhiteclouds.com all things Nutanix, VMware, cloud and virtualizing business critical applications Sun, 29 Nov 2015 13:46:14 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.6 45024036 VMware Distributed vSwitch LACP Configuration with Dell Force10 and Cumulus Linux http://longwhiteclouds.com/2015/11/25/vmware-distributed-vswitch-lacp-configuration-with-dell-force10-and-cumulus-linux/ http://longwhiteclouds.com/2015/11/25/vmware-distributed-vswitch-lacp-configuration-with-dell-force10-and-cumulus-linux/#comments Tue, 24 Nov 2015 19:50:18 +0000 http://longwhiteclouds.com/?p=10973


In my article Etherchannel and IP Hash or Load Based Teaming? I argued that using port channels and Etherchannel or LACP is an overly complex configuration that doesn’t offer much in terms of predictable load balance across host NICs. There is a lot more switch side configuration when you want to use LACP or Etherchannel and […]

]]>


In my article Etherchannel and IP Hash or Load Based Teaming? I argued that using port channels and Etherchannel or LACP is an overly complex configuration that doesn’t offer much in terms of predictable load balance across host NICs. There is a lot more switch side configuration when you want to use LACP or Etherchannel and port-channels to your host NIC’s, for in most cases little benefit. In most common cases Load Based Teaming in vSphere is recommended with vSphere Distributed Switches.

However there are a number of situations where you really can’t avoid doing port-channels and LACP or Static Etherchannel load balancing to your hosts. One of those situations is where you wish to use a network virtualization product such as VMware NSX, and you wish to maximize performance and utilization of your host NIC’s. This article will focus on both the vSphere Host and physical switch side configuration to set up LACP on VMware vSphere 6.0 when using Dell Force10 Networks switches based on Cumulus Linux Network OS.

First we will start with the network configuration and then move onto the vSphere Host configuration. For those not familiar with Cumulus Linux you should check out the Cumulus Networks web site, as it makes networking easy enough even server administrators can do it, it’s just Linux. It’s suitable for small environments with limited administration staff, as everything can be easily automated using familiar automation tools and frameworks, for both server configuration to network configuration, and also for large scale environments that need to change and adapt rapidly. I’ve heard that Cumulus is popular with at least one of the large cloud / web-scale companies where they have thousands of switches.

Cumulus Linux offers comprehensive network functionality on a choice of hardware at a fraction of the cost of other network OS’s. In the examples presented here we use Dell Force10 switches to provide low latency, high performance and high reliability network fabric. Dell offers a choice of network OS on their Open Network compliant switches, including Cumulus Linux, Force10 OS, and BigSwitch. Dell provided the network switches for my lab environment and I use them for all the performance testing and validation that I perform on a day to day basis across business critical applications running in virtualized environments on Nutanix web-scale hyperconverged infrastructure.

Below I’ve provided two different ways of configuration for Cumulus Linux that will allow you to leverage LACP across a redundant switching fabric utilizing CLAG. This is similar to MLAG, or VLT in other switch configurations. The below configuration is for the leaf switches in a leaf and spine architecture, which is depicted in the image below. The first type of configuration leverages /etc/network/interfaces completely, and is a more manual configuration. The second way leverages mako templates, which are placed in /etc/network/interfaces.d, which automatically builds the configurations based on the template.

 

NZ Performance Lab - Networking

 

The first part of the configuration is to configure the links between the two top of rack or middle of rack leaf switches and the link to the spine.

LeafSwitch1 CLAG, Peer and Spine Config using /etc/network/interfaces:


# Spine Link
auto spn1-2
iface spn1-2 
  bond-slaves swp49 swp50
  bond-mode 802.3ad
  bond-miimon 100
  bond-use-carrier 1
  bond-min-links 1
  bond-lacp-rate 1
  bond-xmit_hash_policy layer3+4
  clag-id 1
  mstpctl-portnetwork no
  mtu 9216

# Peer Link
auto pl
iface pl 
  bond-slaves swp51 swp52
  bond-mode 802.3ad
  bond-miimon 100
  bond-use-carrier 1
  bond-min-links 1
  bond-lacp-rate 1
  bond-xmit_hash_policy layer3+4
  mstpctl-portnetwork no
  mtu 9216

# CLAGD Peer Config

auto pl.4000
iface pl.4000 
  address 172.16.0.1/30
  clagd-enable yes
  clagd-priority 8192
  clagd-peer-ip 172.16.0.2
  clagd-backup-ip 192.168.255.12#MGMT IP Address of Peer
  clagd-sys-mac 44:38:39:ff:00:01

LeafSwitch2 CLAG, Peer and Spine Config using /etc/network/interfaces:

auto spn1-2
iface spn1-2 
  bond-slaves swp49 swp50 
  bond-mode 802.3ad
  bond-miimon 100 
  bond-use-carrier 1 
  bond-min-links 1 
  bond-lacp-rate 1
  bond-xmit_hash_policy layer3+4 
  clag-id 1
  mstpctl-portnetwork no
  mtu 9216

# Peer Link
auto pl
iface pl 
  bond-slaves swp51 swp52
  bond-mode 802.3ad
  bond-miimon 100
  bond-lacp-rate 1
  bond-use-carrier 1
  bond-min-links 1
  bond-xmit_hash_policy layer3+4
  mstpctl-portnetwork no
  mtu 9216

# CLAGD Peer Config
auto pl.4000
iface pl.4000 
  address 172.16.0.2/30
  clagd-enable yes
  clagd-peer-ip 172.16.0.1
  clagd-backup-ip 192.168.255.11 #MGMT IP Address of Peer
  clagd-sys-mac 44:38:39:ff:00:01

Next we need to define the port channels also known as bonds in Linux networking, which will be used by the hosts connected to the switches redundantly. This ensures there is no single point of failure on the network fabric to hosts. A bond consists of one of more physical ports connected to the hosts. The CLAG ID must be unique across all bonds.

Here is an example of a host port channel configured in /etc/network/interfaces

# Switch Port Interface Configuration
# NXVMW Block Configuration
auto nxvmw-node1
iface nxvmw-node1
  bond-slaves swp1
  bond-mode 802.3ad
  bond-lacp-rate 1
  bond-min-links 1
  bond-lacp-bypass-allow 1
  bond-miimon 100
  bond-xmit_hash_policy layer3+4
  bridge-pvid 560
  mstpctl-portadminedge yes
  mstpctl-bpduguard yes
  clag-id 2
  mtu 9216

Now we have our uplinks to the spine and our downlinks to the hosts, we need our Bridge for all of the ports to connect to, in this case we will use VLAN-Aware Bridge, as we want to trunk VLAN’s to the host ports. A bridge in Linux networking for those not familiar is like a virtual switch that all physical and virtual ports connect to. In the case of Cumulus Linux the bridge is the ports that all of the hardware accelerated physical switch ports connect to. Individual ports to the hosts are created in bonds. Bridges can have multiple VLAN interfaces in addition to the physical switch port interfaces and bonds connected. You can use a glob to create a group of multiple similar interfaces in the specification as shown below. The example includes multiple VLAN’s, including some VLAN’s assigned with layer3 IP addresses.

# Bridge Configuration

auto br0
iface br0
   bridge-vlan-aware yes
   bridge-ports pl spn1-2 glob nxvmw-node1-4 glob nx3000-node1-4
   bridge-stp on
   bridge-pvid 1
   bridge-vids 100,560,570,580,590,600
   bridge-mcsnoop 1
   bridge-igmp-querier-src 192.168.245.211
   mtu 9216

# Bridge VLAN
auto br0.600
iface br0.600
   address 172.16.100.211/23
   broadcast 172.16.101.255
   mtu 9216

auto br0.100
iface br0.100
   address 192.168.245.211/24
   broadcast 192.168.245.255
   up ip route add 0.0.0.0/0 via 192.168.245.230
   mtu 9216

ifupdown2 supports Mako templates natively, so you can configure templates such as below and put them in /etc/network/interfaces.d to achieve the same thing. The Mako templates allow a simple form of automation and programability for interface creation and configuration on Cumulus Linux. Customers could have a script or automation tool that configures their virtual environment and automatically applies the changes to mako templates on their Cumulus switches and updates the physical environment all in one go.

Here is the Mako template to configure clagd – /etc/network/interfaces.d/clag_cfg, this creates the peer links and all the CLAG configuration necessary.

<%
#INPUTS - Peer Link
 lacp_mode = "fast" # or "slow" # Use for bond rate&lt;/pre&gt;
 peer_bond_name = "pl" # Name for the bond link
 peer_port_start = 51
 peer_port_end = 52
#INPUT - Clag Info
 clag_vlan_id = 4000
 clag_address = "172.16.0.2/30" #SP1 CLAG address
 clag_peer_address = "172.16.0.1" # CLAG peer link address
 clag_pair_unique_mac = "44:38:39:ff:ff:01" # Unique/reserved Per CLAG Pair MAC 
 clag_priority = 8192 # CLAG PRIORITY/Slave should be higher number, like 12288
 clag_args = [] # OUTPUT
# Will create peer_slaves =
# Optional flags can be added over here.bond member matching list.
 "glob swp[{0}-{1}]".format(peer_port_start, peer_port_end)

if clag_args:
  clag_args_str = " ".join(clag_args)else:
  clag_args_str = ""lacp_rate = 1 if "fast" in lacp_mode else 0 %>


auto ${peer_bond_name}
iface ${peer_bond_name}
  bond-slaves ${peer_slaves}
  bond-mode 802.3ad
  bond-miimon 100
  bond-use-carrier 1
  bond-min-links 1
  bond-lacp-rate ${lacp_rate}
  bond-xmit-hash-policy layer3+4
  mstpctl-portnetwork no
  mtu 9216

auto ${peer_bond_name}.${clag_vlan_id}
iface ${peer_bond_name}.${clag_vlan_id}
  address ${clag_address}
  clagd-enable yes
  clagd-priority ${clag_priority}
  clagd-peer-ip ${clag_peer_address}
  clagd-sys-mac ${clag_pair_unique_mac}
  mtu 9216

%if clag_args_str:
  clagd-args ${clag_args_str}
%endif

Here is a Mako template that could be used to configure the uplinks to the Spine switches – /etc/network/interfaces.d/spine_uplink_cfg

<%
# INPUT
# Tuple is of the form (name of bond, bond slaves, LACP RATE = 1/FAST or 0/SLOW)
deploy_uplink_bonds = [("spn1-2","swp49 swp50","1","1")]
%>

% for bond_name, bond_slaves, lacp_rate, clag_id in deploy_uplink_bonds:

auto ${bond_name}
iface ${bond_name}
  bond-slaves ${bond_slaves}
  bond-mode 802.3ad
  bond-miimon 100
  bond-lacp-rate ${lacp_rate}
  bond-use-carrier 1 
  bond-min-links 1 
  bond-xmit_hash_policy layer3+4 
  clag-id {clag_id}
  mstpctl-portnetwork no
  mtu 9216 
% endfor 

Here is a Mako template that could be used to configure the host port channels aka bonds – /etc/network/interfaces.d/downlink_bonds_cfg

<%
 # INPUT
 # Tuple is of the form (name of bond, bond slaves, LACP RATE = 1/FAST or 0/SLOW, clag_id) 
deploy_downlink_bonds = [("nxvmw-node1","swp1","1","2"),
                         ("nxvmw-node2","swp2","1","3"),
                         ("nxvmw-node3","swp47","1","4"),
                         ("nxvmw-node4","swp48","1","5")] %>
 
% for bond_name, bond_slaves, lacp_rate, clag_id in deploy_downlink_bonds: 

auto ${bond_name}
iface ${bond_name}
 bond-slaves ${bond_slaves} 
 bond-mode 802.3ad
 bond-miimon 100
 bond-lacp-rate ${lacp_rate} 
 bond-min-links 1
 bond-lacp-rate 1
 bond-lacp-bypass-allow 1
 bridge-pvid 560
 mstpctl-portadminedge yes 
 mstpctl-bpduguard yes
 clag-id {clag_id}
 bond-xmit-hash-policy layer3+4 
 mtu 9216
% endfor

In a future article I will cover a similar configuration with Dell Force10 switches and the Dell Force10 Network OS (FTOS), in addition to VMware NSX and VXLAN configuration.

To use LACP in vSphere you need to be on vSphere 5.1 or above, and to have configured your vSphere Distributed Switch for LACP. If you’re running vSphere 5.5 or above it’s recommended that you use Enhanced LACP, which offers additional load balancing methods. The load balancing method I’ve found most useful in my tests is depicted in the vDS configuration image below, which is based on a vDS version 6.0:

VMware vShphere 6.0 LACP Config 2015-11-25_06-36-53

With vSphere 6.0 vDS you can configure up to 64 port channels, known as Link Aggregation Groups (LAG’s) per Host and up to 32 ports per LAG. It is recommended you review the other relevant vSphere Configuration Maximums.

You will connect your hosts directly to a port on the LAG and not a standard vDS uplink. The LAG then appears as a single interface in the port group configuration. The below image shows the host 10GbE NIC’s connected to a LAG on a vDS.

VMware vSphere 6.0 LACP Host Uplink Config 2015-11-25_07-46-11

When you are configuring the vDS port groups that VM’s will connect to you need to ensure that only the LAG is present as the active interface. All other interfaces should be marked as unused. However during migration you can mark the LAG as active and another interface as stand by, until you’ve migrated all physical and virtual interfaces to the LAG. It is recommended you read the migrating to LAG guidance on the Enhanced LACP Configuration of the vDS. The below image shows an example of the vDS Portgroup configuration.

VMware vSphere 6.0 LACP Portgroup Config 2015-11-25_07-46-11

When using a LAG the Load Balancing method in the Portgroup configuration does not apply.

As far as performance goes I’ve successfully tested hosts maxed out with 2 x 10GbE Links doing full duplex traffic to wire speed, i.e. 4.8GBytes/s and latency as low as 70us, even when using VMware NSX and VXLAN overlay networks.

Final Word

When you need to use LACP and port channels / LAG’s for VMware vSphere, using Cumulus Linux on your physical network switches provides an easy way to get up and running quickly and provide configuration automation. The above configurations can be used and reused as much as you like. You can also download a free version of Cumulus Linux VX, which simulates the physical network configurations, before you apply it to physical switches.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.


]]>
http://longwhiteclouds.com/2015/11/25/vmware-distributed-vswitch-lacp-configuration-with-dell-force10-and-cumulus-linux/feed/ 2 10973
Configuring Scalable Low Latency L2 Leaf-Spine Network Fabrics with Dell Networking Switches http://longwhiteclouds.com/2015/03/26/configuring-scalable-low-latency-l2-leaf-spine-network-fabrics-with-dell-networking-switches/ http://longwhiteclouds.com/2015/03/26/configuring-scalable-low-latency-l2-leaf-spine-network-fabrics-with-dell-networking-switches/#comments Thu, 26 Mar 2015 08:36:19 +0000 http://longwhiteclouds.com/?p=10603


In December 2014 I got an early Christmas present from Dell. They shipped me the latest 40G and 10G S series (Force 10) switches so that I could begin to test, validate and document the integration and reference architectures between Dell Networking and Nutanix. I’m starting with a L2 MLAG (Multi-chassis Link Aggregation Group)  configuration […]

]]>


Logo-JPG-Dell_Force10_Dell-BlueIn December 2014 I got an early Christmas present from Dell. They shipped me the latest 40G and 10G S series (Force 10) switches so that I could begin to test, validate and document the integration and reference architectures between Dell Networking and Nutanix. I’m starting with a L2 MLAG (Multi-chassis Link Aggregation Group)  configuration and I will work my way through to a full ECMP (Equal Cost Multi-path) L3 configuration including VMware NSX. This will be a journey and I’ll include the different options and their key considerations, and configurations in the eventual white papers that Nutanix publishes. I’ve had a few weeks to configure them and do some initial testing (after the Christmas Holiday break), so I thought I’d write about what I’ve found so far. On the NSX front, you’ll be interested to know that Nutanix already has customers running NSX and that the platforms work extremely well together, as they both scale out linearly and predictably. But we’ll leave NSX specific discussion till another day. This article will contain some highlights without steeling the thunder of the white papers I’m working on.

Traditionally datacenter networks were usually designed with 3 layers, Access – where servers connected, Aggregation or Distribution – where the access switches connected and also normally the L2 demarcation, and Core, where everything was bought together at L3. Spanning Tree Protocol (STP) is enabled and redundant links will be blocked to prevent network loops on the L2 segments. However with this design you are not able to utilize all the links, due to STP, and it can be complex to scale and achieve consistent latency between the different points in the network. However these problems can be solved by taking a leaf and spine architecture approach.

As you can see in the diagram above I have 2 Spine Switches – Dell S6000’s with 32 x 40GbE ports, and 4 Leaf Switches – Dell S4810 with 48 x 10GbE Ports and 4 x 40GbE Ports. The Spine and two of the Leaf Switches are running Dell FTOS (Force Ten OS) 9.6, while two of the Leaf switches are the S4810-ON (Open Networks) model and are running Cumulus Linux 2.5. Check out the details of the Dell Networking Force 10 S series switches.

cumulus-networks-dell-sdn-bare-metal-white-box-switchesCumulusTurtleLogo

 

 

The reason I like Cumulus Linux, is well, it’s just Linux, but with hardware accelerated switching and routing. If you know Linux (Debian is the distribution it’s based off), then it’s very easy to use. No need for additional training, fits into the same management frameworks, such as Puppet, Chef, Ansible and others. So it’s great to have the choice of this or FTOS on the Dell Networking switches. I found both Cumulus and FTOS easy to use, and the documentation is very good.

The port to port latency on the S6000’s is ~500ns, while the S4810’s are ~800ns (so about 1.3us Leaf to Spine), even when routing. With the overhead in the IP stack of each of the hypervisors and VM’s I’m seeing latency across the network between VM’s of < 90us end to end, this is using standard Intel 10GbE NIC’s, VMXNET3 vNIC and without using the latency sensitive settings or tuning within VMware vSphere 5.5.

The main benefit of using MLAG is that you don’t have to have a whole lot of links disabled to prevent network loops and have STP interfering with it, it’s also very easy to set up. You still have STP enabled to prevent loops during switch boot, but when things are up and running all of the links are available to pass traffic and you benefit from the combined bandwidth. Each switch is managed and updated independently, so doesn’t become a single point of failure, as it could if you’d chosen to stack the switches. Switch stacking might be ok if you needed multiple stacks anyway and the host links to the stack themselves were redundant (you can’t mix stacking and MLAG together). While this does add some slight management overhead, it gains the ability to update the switch firmware independently without causing any disruption. With the standard automation and management frameworks such as Puppet, Ansible, Chef, CFEngine etc, the management overhead is greatly reduced or eliminated in any case.

Here is a diagram of what my lab network looks like at a high level. My lab hosts are directly connected to the leaf switches.

NZ Performance Lab - Networking

In FTOS an MLAG is called a Virtual Link Trunk (VLT), and in Cumulus it’s called CLAG (Chassis Link Aggregation Group). The process to configure them is fairly similar at a high level.

Configure Out of Band Management Interfaces (after initial switch boot / install)
Create a port channel between the adjacent switches (Spine or Leaf)
Configure Spanning Tree (RSTP)
Create a Peer Link on top of the port channel between the adjacent switches so that inter-switch communications can take place to sync mac addresses etc. Set up a backup address in case the primary fails, this will prevent split brain scenarios.
Configure and enable VLT or CLAGD (examples will follow)
Configure and enable the other port channels, edge ports, VLAN’s, routing etc

FTOS Spine Example:

! Note: Peer Link is recommended to be static port channel not LACP.
!
lacp ungroup member-independent vlt
lacp ungroup member-independent port-channel 100
!
default vlan-id 4000
!
protocol spanning-tree rstp
no disable
bridge-priority 16384
!
vlt domain 1

peer-link port-channel 100
back-up destination xxx.xxx.xxx.xxx <- IP Address of Backup Destination
primary-priority 16384
peer-routing
peer-routing-timeout 1
!
interface Port-channel 10
description Cumulus Leaf-Link
no ip address

mtu 9216
portmode hybrid
switchport
lacp fast-switchover 
vlt-peer-lag port-channel 10
no shutdown
!
interface Port-channel 100
description Peer-Link
no ip address
mtu 9216
channel-member fortyGigE 0/120,124
no shutdown
!
interface fortyGigE 0/112
description Leaf1 – Port Channel 10
no ip address
mtu 9216
flowcontrol rx on tx off
!
port-channel-protocol LACP
port-channel 10 mode active
no shutdown
!
interface fortyGigE 0/116
description Leaf2 – Port Channel 10
no ip address
mtu 9216
flowcontrol rx on tx off
!
port-channel-protocol LACP
port-channel 10 mode active
no shutdown
!
interface fortyGigE 0/120
description Peer-Port 1 – Port Channel 100
no ip address
mtu 9216
flowcontrol rx on tx off
no shutdown
!
interface fortyGigE 0/124
description Peer-Port 2 – Port Channel 100
no ip address
mtu 9216
flowcontrol rx on tx off
no shutdown
!
interface Vlan 500
description Host VLAN
no ip address
mtu 9216
tagged Port-channel 1,10
no shutdown
!
interface Vlan 4000
mtu 9216
!untagged Port-channel 100
no shutdown
!

Cumulus Leaf Example with CLAGD:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5), ifup(8)
#
# Please see /usr/share/doc/python-ifupdown2/examples/ for examples
#
#
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0
address xxx.xxx.xxx.xxx/24
broadcast xxx.xxx.xxx.255
# Spine Link
auto spn1-2
iface spn1-2
bond-slaves swp49 swp50
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
# clag-id needs to be unique on each clag, like a vlt domain id on FTOS
mstpctl-portnetwork no
mtu 9216
# Peer Link to Other LeafSwitch
auto pl
iface pl
bond-slaves swp51 swp52
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
mstpctl-portnetwork no
mtu 9216
# CLAGD Peer Int Config
auto pl.4000
iface pl.4000
address xxx.xxx.xxx.xxx/30
clagd-enable yes
clagd-priority 8192
clagd-peer-ip 172.16.0.2
clagd-backup-ip 192.168.255.12
clagd-sys-mac 44:38:39:ff:00:01
# Switch Port Interface Configuration
auto swp1
iface swp1
mtu 9216

auto swp2
iface swp2
mtu 9216

auto swp3
iface swp3
mtu 9216

auto swp4
iface swp4
mtu 9216

auto swp49
iface swp49
mtu 9216

auto swp50
iface swp50
mtu 9216

auto swp51
iface swp51
mtu 9216

auto swp52
iface swp52
mtu 9216

# Bridge Configuration
#
auto br0
iface br0
bridge-vlan-aware yes
bridge-ports pl spn1-2 glob swp[1-4]
bridge-stp on
bridge-pvid 1
bridge-vids 500
mstpctl-portadminedge swp1=yes swp2=yes swp3=yes swp4=yes
bridge-mcsnoop 1
mtu 9216
# Bridge VLAN
# Host VLAN
auto br0.500
iface br0.500
address xxx.xxx.xxx.xxx/23
broadcast xxx.xxx.xxx.255
up ip route add 0.0.0.0/0 via 192.168.1.230
mtu 9216

The above are examples and not complete configs and you can’t just copy and paste it all and expect it to work in your environment. But it could be used as a starting point.

Bringing this all together. The Dell S6000 / S4810 combination allows you to create a scalable network design that provides predictable and consistent low latency and high throughput from end to end in the network. The configuration of the MLAG/CLAG/VLT is straight forward, and it provides management flexibility. You get to choose either FTOS or Open Networking such as Cumulus Linux as your switching software. With Cumulus Linux, it’s just like Linux, but wire speed non-blocking networking accelerated in hardware and fits into the normal management frameworks. For environments, such as Hyper-converged or Web-scale infrastructure, the network scales linearly, as do the systems that connect to it. Each time you grow, you get consistent, predictable and linear performance.

Here is an example of a high level diagram that might be appropriate for a small scale deployment. In this case the Dell S4810’s are used as both Leaf and Spine switches in a VLT configuration, with Dell N3048 or N3024 providing 1G connectivity. With this you could easily start with a single rack and scale to 8 racks. Each rack would have full 10G and 1G redundant connectivity. With say 24 Servers or Hyper-converged nodes per rack you would be able to support 192 servers across 8 racks. 40G QSFP+ ports are used for Peer Links, while the Leaf connects to the Spine using 40G to 10G break out cables.

NX3000 Small Scale Networking

 

Here is an example high level diagram of a medium density Nutanix Web-scale Converged Infrstructure deployment with Dell S4810 Leaf switches connected to S6000 Spine, which could be using FTOS VLT or Cumulus CLAG. As you can see this design is capable of scaling to 12 racks (576 nodes) with the S6000 Spine switches, and up to 52 racks and 2496 nodes, with the Dell Z9500 Spine switches. Dell Z9500 supports 128 x 40GbE ports. Both options have spare 40GbE ports still available for Boarder-Leaf connectivity (Cross Datacenter, or Internet routers etc).

NX3060 Medium Density

If you wanted a higher density design you can combine S6000 in middle of rack or top of rack configuration for Leaf switches with S6000 or Z9500 Spine. This diagram provides a high level example of a high density configuration on a standard 48U rack. In this example the design would support 10 racks with 880 nodes with S6000 Spine, and 48 racks with 4224 nodes with a Z9500 Spine. With enough ports to accommodate the Boarder-Leaf nodes as well.

NX3000 High Density

With densities in the datacenter increasing and the power consumption of the servers decreasing I can see an explosion of 40GbE Top of Rack (ToR) or Middle of Rack (MoR) Leaf switches. This would also provide an option to easily allow different bandwidth oversubscription models, 6:1, 4:1 etc as requirements change.

So far we’ve covered the Dell Networking switches by themselves, with FTOS and Cumulus, and then some examples combined with Nutanix. Dell is an OEM partner of Nutanix software and delivers the Dell XC Series Web-scale Converged Appliances. With Dell XC you have a number of hardware options for different use cases, and you can build a complete solution, including networking.

The following example shows Dell S4810 Leaf switches and Dell S6000 Spine switches with Dell XC series appliances. These appliances are 2U each and contain one node. Dell XC also has 1U options.

Dell XC with Dell Networking

 

Final Word

Those of you who spend time in a modern datacenter will see I’ve drawn the diagrams with the switch ports facing forward, which is actually backwards. I did this to make it easy to draw, and because I like flashing lights. In real world environments the air flow of the switches would be reversed and the back of the switches, where the PSU is, would be facing the front of the rack, so that the cabling can be nice and tidy. In summary, where predictable, low latency, high throughput, linearly scalable networking is required, Leaf Spine architectures are becoming increasingly common (and are simpler than three tier IMHO). Web-scale converged or Hyper-converged Infrastructure benefits from the low latency, high throughput and linearly scalability the Dell Networking switches can provide. As does network virtualization such as VMware NSX. Dell Networking switches combined with Nutanix appliances, or Dell XC series appliances powered by Nutanix can deliver a unified and simplified high performance virtual infrastructure with greatly reduced complexity compared to a traditional three tier architecture. A great foundation for a private cloud or software defined datacenter.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.


]]>
http://longwhiteclouds.com/2015/03/26/configuring-scalable-low-latency-l2-leaf-spine-network-fabrics-with-dell-networking-switches/feed/ 16 10603