Cumulus Linux is the answer to companies that want to run software defined networking on a range of open networks industry standard switches, without necessarily being locked into one physical switch hardware vendor. But unlike network virtualization solutions such as NSX, Cumulus Linux is the Network OS (NOS) for the physical switches, rather than a virtualization layer on top. Cumulus is part of the NSX ecosystem and integrated into NSX, so essentially you can use Cumulus to run on the physical switches and integrate it to NSX to provide the network virtualization (termination and VXLAN switching/routing in hardware also supported on some switches). Cumulus is Linux for network switches, so it’s easy to manage, and very easy to automate. I happen to be working on a project now to build the best practices for Cumulus Linux with Nutanix and VMware vSphere. So I needed an easy way to get Cumulus installed on my lab switches, from my MacBook Pro, which is what the remainder of this article is about.

You can choose open network (ON) switches from a variety of vendors that are on the Cumulus HCL. In my case I chose Dell Force10 S4810-ON‘s. The -ON is an important part, as that is the Open Network variety. The Dell Force10 Switches are enterprise class low latency switches. The -ON switches come with the Open Network Install Environment included, so that you can install Cumulus Linux. This is the default boot environment for the switches and starts automatically.

There are 6 ways you can install Cumulus Linux NOS on your Open Network switches as follows:

1. Passed from the boot loader.
2. Search locally attached storage devices for one of the ONIE default installer filenames (for example,
   USB).
3. Exact the URLs from DHCPv4.
4. Inexact URLs based on DHCPv4 responses.
5. Query to IPv6 link-local neighbors using HTTP for an installer.
6. TFTP waterfall — from DHCPv4 option 66

In my case I had DHCP configured for my management network so I just needed an easy way to start up a web server so the switches could discover the Cumulus Linux firmware and download and install it. As a tip, if you don’t have a DHCP server already you can configure your MacBook Pro for Internet sharing, which then starts a DHCP server. Setting up the web server was a lot easier than I thought it would be. It turns out there is a very easy way to start up a temporary HTTP server on a MacBook Pro from any directory through the terminal. I stumbled across an article titled Start a Simple Web Server from Any Directory on Your Mac. All I had to do was change to the directory containing the Cumulus Linux package, which I had renamed to work with the ONIE process. But there was one key element missing from the article that I needed in order to get it to work.

If you attempt to run the web server exactly as mentioned in the Life Hacker article, such as python -m SimpleHTTPServer 80, you will get the following:

michael2012mbp:Cumulus michaelwebster$ python -m SimpleHTTPServer 80

Traceback (most recent call last):

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py”, line 162, in _run_module_as_main “__main__”, fname, loader, pkg_name)

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py”, line 72, in _run_code exec code in run_globals

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SimpleHTTPServer.py”, line 224, in <module> test()

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SimpleHTTPServer.py”, line 220, in test BaseHTTPServer.test(HandlerClass, ServerClass)

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py”, line 595, in test httpd = ServerClass(server_address, HandlerClass)

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py”, line 419, in __init__ self.server_bind()

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py”, line 108, in server_bind SocketServer.TCPServer.server_bind(self)

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py”, line 430, in server_bind self.socket.bind(self.server_address)

  File “/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py”, line 224, in meth return getattr(self._sock,name)(*args)

socket.error: [Errno 13] Permission denied

This was easily fixed by using the sudo command, entering the admin password, and running the operation as root. Which resulted in the following:

michael2012mbp:Cumulus michaelwebster$ sudo python -m SimpleHTTPServer 80

WARNING: Improper use of the sudo command could lead to data loss or the deletion of important system files. Please double-check your typing when using sudo. Type “man sudo” for more information.

To proceed, enter your password, or type Ctrl-C to abort.

Password:

Serving HTTP on 0.0.0.0 port 80 …

Then it was just a matter of kicking off the discovery process on my S4810 switches, which I did by rebooting them via the console cable (which happens to be connected via USB to a VDI desktop running on one of my Nutanix hosts, and it even still works with vMotion).

Once the switches restarted they found my MacBook on the network and found the web server and began to search for the firmware. ONIE goes through a standard process to identify and download the correct firmware by using the following naming conventions from the most specific to the lest specific:

1. onie-installer-<arch>-<vendor>_<machine>-r<machine_revision>
2. onie-installer-<arch>-<vendor>_<machine>
3. onie-installer-<vendor>_<machine>
4. onie-installer-<arch>
5. onie-installer

In my case it looked like this from the web server on my MacBook Pro:

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] “GET /onie-installer-powerpc-dni_7448-r0 HTTP/1.1” 404 –

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] code 404, message File not found

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] “GET /onie-installer-powerpc-dni_7448 HTTP/1.1” 404 –

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] code 404, message File not found

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] “GET /onie-installer-dni_7448 HTTP/1.1” 404 –

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] code 404, message File not found

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] “GET /onie-installer-powerpc HTTP/1.1” 404 –

xxx.xxx.xxx.11 – – [13/Nov/2014 19:31:34] “GET /onie-installer HTTP/1.1” 200 –

As you can guess from the above I had named the Cumulus Linux package onie-installer. I could have named it onie-installer-powerpc or any one of the other specific naming conventions. If this was a large scale environment it would be best to set up the distribution point using specific names for each model of system in the environment. Although the Dell S4810’s are PowerPC based some switches are x86_64 based, such as the Dell S6000-ON 40GbE switches.

Final Word

The process I used to get Cumulus Linux installed on my lab switches is probably fine for small scale environments and PoC’s. For large scale you will want something more robust and automated. One of the great things about Cumulus being Linux for switches is that it can fit into you existing Linux management and automation frameworks, such as Puppet, Chef and CFEngine. You can completely automate the configuration and management of a large scale switching environment, which when combined with network virtualization by NSX can become incredibly agile and flexible for any type of application workload. I’m working with Cumulus and I will be documenting the official best practices for Cumulus with Nutanix and it will be published on the Nutanix web site once we’re done. Along the way I will bring you any interesting things that I find.

—

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2014 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.