| Unique Visitors |
In my article Etherchannel and IP Hash or Load Based Teaming? I argued that using port channels and Etherchannel or LACP is an overly complex configuration that doesn’t offer much in terms of predictable load balance across host NICs. There is a lot more switch side configuration when you want to use LACP or Etherchannel and port-channels to your host NIC’s, for in most cases little benefit. In most common cases Load Based Teaming in vSphere is recommended with vSphere Distributed Switches.
However there are a number of situations where you really can’t avoid doing port-channels and LACP or Static Etherchannel load balancing to your hosts. One of those situations is where you wish to use a network virtualization product such as VMware NSX, and you wish to maximize performance and utilization of your host NIC’s. This article will focus on both the vSphere Host and physical switch side configuration to set up LACP on VMware vSphere 6.0 when using Dell Force10 Networks switches based on Cumulus Linux Network OS.
First we will start with the network configuration and then move onto the vSphere Host configuration. For those not familiar with Cumulus Linux you should check out the Cumulus Networks web site, as it makes networking easy enough even server administrators can do it, it’s just Linux. It’s suitable for small environments with limited administration staff, as everything can be easily automated using familiar automation tools and frameworks, for both server configuration to network configuration, and also for large scale environments that need to change and adapt rapidly. I’ve heard that Cumulus is popular with at least one of the large cloud / web-scale companies where they have thousands of switches.
Cumulus Linux offers comprehensive network functionality on a choice of hardware at a fraction of the cost of other network OS’s. In the examples presented here we use Dell Force10 switches to provide low latency, high performance and high reliability network fabric. Dell offers a choice of network OS on their Open Network compliant switches, including Cumulus Linux, Force10 OS, and BigSwitch. Dell provided the network switches for my lab environment and I use them for all the performance testing and validation that I perform on a day to day basis across business critical applications running in virtualized environments on Nutanix web-scale hyperconverged infrastructure.
Below I’ve provided two different ways of configuration for Cumulus Linux that will allow you to leverage LACP across a redundant switching fabric utilizing CLAG. This is similar to MLAG, or VLT in other switch configurations. The below configuration is for the leaf switches in a leaf and spine architecture, which is depicted in the image below. The first type of configuration leverages /etc/network/interfaces completely, and is a more manual configuration. The second way leverages mako templates, which are placed in /etc/network/interfaces.d, which automatically builds the configurations based on the template.
The first part of the configuration is to configure the links between the two top of rack or middle of rack leaf switches and the link to the spine.
LeafSwitch1 CLAG, Peer and Spine Config using /etc/network/interfaces:
# Spine Link auto spn1-2 iface spn1-2 bond-slaves swp49 swp50 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-min-links 1 bond-lacp-rate 1 bond-xmit_hash_policy layer3+4 clag-id 1 mstpctl-portnetwork no mtu 9216 # Peer Link auto pl iface pl bond-slaves swp51 swp52 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-min-links 1 bond-lacp-rate 1 bond-xmit_hash_policy layer3+4 mstpctl-portnetwork no mtu 9216 # CLAGD Peer Config auto pl.4000 iface pl.4000 address 172.16.0.1/30 clagd-enable yes clagd-priority 8192 clagd-peer-ip 172.16.0.2 clagd-backup-ip 192.168.255.12#MGMT IP Address of Peer clagd-sys-mac 44:38:39:ff:00:01
LeafSwitch2 CLAG, Peer and Spine Config using /etc/network/interfaces:
auto spn1-2 iface spn1-2 bond-slaves swp49 swp50 bond-mode 802.3ad bond-miimon 100 bond-use-carrier 1 bond-min-links 1 bond-lacp-rate 1 bond-xmit_hash_policy layer3+4 clag-id 1 mstpctl-portnetwork no mtu 9216 # Peer Link auto pl iface pl bond-slaves swp51 swp52 bond-mode 802.3ad bond-miimon 100 bond-lacp-rate 1 bond-use-carrier 1 bond-min-links 1 bond-xmit_hash_policy layer3+4 mstpctl-portnetwork no mtu 9216 # CLAGD Peer Config auto pl.4000 iface pl.4000 address 172.16.0.2/30 clagd-enable yes clagd-peer-ip 172.16.0.1 clagd-backup-ip 192.168.255.11 #MGMT IP Address of Peer clagd-sys-mac 44:38:39:ff:00:01
Next we need to define the port channels also known as bonds in Linux networking, which will be used by the hosts connected to the switches redundantly. This ensures there is no single point of failure on the network fabric to hosts. A bond consists of one of more physical ports connected to the hosts. The CLAG ID must be unique across all bonds.
Here is an example of a host port channel configured in /etc/network/interfaces
# Switch Port Interface Configuration # NXVMW Block Configuration auto nxvmw-node1 iface nxvmw-node1 bond-slaves swp1 bond-mode 802.3ad bond-lacp-rate 1 bond-min-links 1 bond-lacp-bypass-allow 1 bond-miimon 100 bond-xmit_hash_policy layer3+4 bridge-pvid 560 mstpctl-portadminedge yes mstpctl-bpduguard yes clag-id 2 mtu 9216
Now we have our uplinks to the spine and our downlinks to the hosts, we need our Bridge for all of the ports to connect to, in this case we will use VLAN-Aware Bridge, as we want to trunk VLAN’s to the host ports. A bridge in Linux networking for those not familiar is like a virtual switch that all physical and virtual ports connect to. In the case of Cumulus Linux the bridge is the ports that all of the hardware accelerated physical switch ports connect to. Individual ports to the hosts are created in bonds. Bridges can have multiple VLAN interfaces in addition to the physical switch port interfaces and bonds connected. You can use a glob to create a group of multiple similar interfaces in the specification as shown below. The example includes multiple VLAN’s, including some VLAN’s assigned with layer3 IP addresses.
# Bridge Configuration auto br0 iface br0 bridge-vlan-aware yes bridge-ports pl spn1-2 glob nxvmw-node1-4 glob nx3000-node1-4 bridge-stp on bridge-pvid 1 bridge-vids 100,560,570,580,590,600 bridge-mcsnoop 1 bridge-igmp-querier-src 192.168.245.211 mtu 9216 # Bridge VLAN auto br0.600 iface br0.600 address 172.16.100.211/23 broadcast 172.16.101.255 mtu 9216 auto br0.100 iface br0.100 address 192.168.245.211/24 broadcast 192.168.245.255 up ip route add 0.0.0.0/0 via 192.168.245.230 mtu 9216
ifupdown2 supports Mako templates natively, so you can configure templates such as below and put them in /etc/network/interfaces.d to achieve the same thing. The Mako templates allow a simple form of automation and programability for interface creation and configuration on Cumulus Linux. Customers could have a script or automation tool that configures their virtual environment and automatically applies the changes to mako templates on their Cumulus switches and updates the physical environment all in one go.
Here is the Mako template to configure clagd – /etc/network/interfaces.d/clag_cfg, this creates the peer links and all the CLAG configuration necessary.
<%
#INPUTS - Peer Link
lacp_mode = "fast" # or "slow" # Use for bond rate</pre>
peer_bond_name = "pl" # Name for the bond link
peer_port_start = 51
peer_port_end = 52
#INPUT - Clag Info
clag_vlan_id = 4000
clag_address = "172.16.0.2/30" #SP1 CLAG address
clag_peer_address = "172.16.0.1" # CLAG peer link address
clag_pair_unique_mac = "44:38:39:ff:ff:01" # Unique/reserved Per CLAG Pair MAC
clag_priority = 8192 # CLAG PRIORITY/Slave should be higher number, like 12288
clag_args = [] # OUTPUT
# Will create peer_slaves =
# Optional flags can be added over here.bond member matching list.
"glob swp[{0}-{1}]".format(peer_port_start, peer_port_end)
if clag_args:
clag_args_str = " ".join(clag_args)else:
clag_args_str = ""lacp_rate = 1 if "fast" in lacp_mode else 0 %>
auto ${peer_bond_name}
iface ${peer_bond_name}
bond-slaves ${peer_slaves}
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-lacp-rate ${lacp_rate}
bond-xmit-hash-policy layer3+4
mstpctl-portnetwork no
mtu 9216
auto ${peer_bond_name}.${clag_vlan_id}
iface ${peer_bond_name}.${clag_vlan_id}
address ${clag_address}
clagd-enable yes
clagd-priority ${clag_priority}
clagd-peer-ip ${clag_peer_address}
clagd-sys-mac ${clag_pair_unique_mac}
mtu 9216
%if clag_args_str:
clagd-args ${clag_args_str}
%endif
Here is a Mako template that could be used to configure the uplinks to the Spine switches – /etc/network/interfaces.d/spine_uplink_cfg
<%
# INPUT
# Tuple is of the form (name of bond, bond slaves, LACP RATE = 1/FAST or 0/SLOW)
deploy_uplink_bonds = [("spn1-2","swp49 swp50","1","1")]
%>
% for bond_name, bond_slaves, lacp_rate, clag_id in deploy_uplink_bonds:
auto ${bond_name}
iface ${bond_name}
bond-slaves ${bond_slaves}
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate ${lacp_rate}
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id {clag_id}
mstpctl-portnetwork no
mtu 9216
% endfor
Here is a Mako template that could be used to configure the host port channels aka bonds – /etc/network/interfaces.d/downlink_bonds_cfg
<%
# INPUT
# Tuple is of the form (name of bond, bond slaves, LACP RATE = 1/FAST or 0/SLOW, clag_id)
deploy_downlink_bonds = [("nxvmw-node1","swp1","1","2"),
("nxvmw-node2","swp2","1","3"),
("nxvmw-node3","swp47","1","4"),
("nxvmw-node4","swp48","1","5")] %>
% for bond_name, bond_slaves, lacp_rate, clag_id in deploy_downlink_bonds:
auto ${bond_name}
iface ${bond_name}
bond-slaves ${bond_slaves}
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate ${lacp_rate}
bond-min-links 1
bond-lacp-rate 1
bond-lacp-bypass-allow 1
bridge-pvid 560
mstpctl-portadminedge yes
mstpctl-bpduguard yes
clag-id {clag_id}
bond-xmit-hash-policy layer3+4
mtu 9216
% endfor
In a future article I will cover a similar configuration with Dell Force10 switches and the Dell Force10 Network OS (FTOS), in addition to VMware NSX and VXLAN configuration.
To use LACP in vSphere you need to be on vSphere 5.1 or above, and to have configured your vSphere Distributed Switch for LACP. If you’re running vSphere 5.5 or above it’s recommended that you use Enhanced LACP, which offers additional load balancing methods. The load balancing method I’ve found most useful in my tests is depicted in the vDS configuration image below, which is based on a vDS version 6.0:

With vSphere 6.0 vDS you can configure up to 64 port channels, known as Link Aggregation Groups (LAG’s) per Host and up to 32 ports per LAG. It is recommended you review the other relevant vSphere Configuration Maximums.
You will connect your hosts directly to a port on the LAG and not a standard vDS uplink. The LAG then appears as a single interface in the port group configuration. The below image shows the host 10GbE NIC’s connected to a LAG on a vDS.

When you are configuring the vDS port groups that VM’s will connect to you need to ensure that only the LAG is present as the active interface. All other interfaces should be marked as unused. However during migration you can mark the LAG as active and another interface as stand by, until you’ve migrated all physical and virtual interfaces to the LAG. It is recommended you read the migrating to LAG guidance on the Enhanced LACP Configuration of the vDS. The below image shows an example of the vDS Portgroup configuration.

When using a LAG the Load Balancing method in the Portgroup configuration does not apply.
As far as performance goes I’ve successfully tested hosts maxed out with 2 x 10GbE Links doing full duplex traffic to wire speed, i.e. 4.8GBytes/s and latency as low as 70us, even when using VMware NSX and VXLAN overlay networks.
Final Word
When you need to use LACP and port channels / LAG’s for VMware vSphere, using Cumulus Linux on your physical network switches provides an easy way to get up and running quickly and provide configuration automation. The above configurations can be used and reused as much as you like. You can also download a free version of Cumulus Linux VX, which simulates the physical network configurations, before you apply it to physical switches.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.
In December 2014 I got an early Christmas present from Dell. They shipped me the latest 40G and 10G S series (Force 10) switches so that I could begin to test, validate and document the integration and reference architectures between Dell Networking and Nutanix. I’m starting with a L2 MLAG (Multi-chassis Link Aggregation Group) configuration and I will work my way through to a full ECMP (Equal Cost Multi-path) L3 configuration including VMware NSX. This will be a journey and I’ll include the different options and their key considerations, and configurations in the eventual white papers that Nutanix publishes. I’ve had a few weeks to configure them and do some initial testing (after the Christmas Holiday break), so I thought I’d write about what I’ve found so far. On the NSX front, you’ll be interested to know that Nutanix already has customers running NSX and that the platforms work extremely well together, as they both scale out linearly and predictably. But we’ll leave NSX specific discussion till another day. This article will contain some highlights without steeling the thunder of the white papers I’m working on.
Traditionally datacenter networks were usually designed with 3 layers, Access – where servers connected, Aggregation or Distribution – where the access switches connected and also normally the L2 demarcation, and Core, where everything was bought together at L3. Spanning Tree Protocol (STP) is enabled and redundant links will be blocked to prevent network loops on the L2 segments. However with this design you are not able to utilize all the links, due to STP, and it can be complex to scale and achieve consistent latency between the different points in the network. However these problems can be solved by taking a leaf and spine architecture approach.
As you can see in the diagram above I have 2 Spine Switches – Dell S6000’s with 32 x 40GbE ports, and 4 Leaf Switches – Dell S4810 with 48 x 10GbE Ports and 4 x 40GbE Ports. The Spine and two of the Leaf Switches are running Dell FTOS (Force Ten OS) 9.6, while two of the Leaf switches are the S4810-ON (Open Networks) model and are running Cumulus Linux 2.5. Check out the details of the Dell Networking Force 10 S series switches.
The reason I like Cumulus Linux, is well, it’s just Linux, but with hardware accelerated switching and routing. If you know Linux (Debian is the distribution it’s based off), then it’s very easy to use. No need for additional training, fits into the same management frameworks, such as Puppet, Chef, Ansible and others. So it’s great to have the choice of this or FTOS on the Dell Networking switches. I found both Cumulus and FTOS easy to use, and the documentation is very good.
The port to port latency on the S6000’s is ~500ns, while the S4810’s are ~800ns (so about 1.3us Leaf to Spine), even when routing. With the overhead in the IP stack of each of the hypervisors and VM’s I’m seeing latency across the network between VM’s of < 90us end to end, this is using standard Intel 10GbE NIC’s, VMXNET3 vNIC and without using the latency sensitive settings or tuning within VMware vSphere 5.5.
The main benefit of using MLAG is that you don’t have to have a whole lot of links disabled to prevent network loops and have STP interfering with it, it’s also very easy to set up. You still have STP enabled to prevent loops during switch boot, but when things are up and running all of the links are available to pass traffic and you benefit from the combined bandwidth. Each switch is managed and updated independently, so doesn’t become a single point of failure, as it could if you’d chosen to stack the switches. Switch stacking might be ok if you needed multiple stacks anyway and the host links to the stack themselves were redundant (you can’t mix stacking and MLAG together). While this does add some slight management overhead, it gains the ability to update the switch firmware independently without causing any disruption. With the standard automation and management frameworks such as Puppet, Ansible, Chef, CFEngine etc, the management overhead is greatly reduced or eliminated in any case.
Here is a diagram of what my lab network looks like at a high level. My lab hosts are directly connected to the leaf switches.

In FTOS an MLAG is called a Virtual Link Trunk (VLT), and in Cumulus it’s called CLAG (Chassis Link Aggregation Group). The process to configure them is fairly similar at a high level.
Configure Out of Band Management Interfaces (after initial switch boot / install)
Create a port channel between the adjacent switches (Spine or Leaf)
Configure Spanning Tree (RSTP)
Create a Peer Link on top of the port channel between the adjacent switches so that inter-switch communications can take place to sync mac addresses etc. Set up a backup address in case the primary fails, this will prevent split brain scenarios.
Configure and enable VLT or CLAGD (examples will follow)
Configure and enable the other port channels, edge ports, VLAN’s, routing etc
FTOS Spine Example:
! Note: Peer Link is recommended to be static port channel not LACP.
!
lacp ungroup member-independent vlt
lacp ungroup member-independent port-channel 100
!
default vlan-id 4000
!
protocol spanning-tree rstp
no disable
bridge-priority 16384
!
vlt domain 1
peer-link port-channel 100
back-up destination xxx.xxx.xxx.xxx <- IP Address of Backup Destination
primary-priority 16384
peer-routing
peer-routing-timeout 1
!
interface Port-channel 10
description Cumulus Leaf-Link
no ip address
mtu 9216
portmode hybrid
switchport
lacp fast-switchover
vlt-peer-lag port-channel 10
no shutdown
!
interface Port-channel 100
description Peer-Link
no ip address
mtu 9216
channel-member fortyGigE 0/120,124
no shutdown
!
interface fortyGigE 0/112
description Leaf1 – Port Channel 10
no ip address
mtu 9216
flowcontrol rx on tx off
!
port-channel-protocol LACP
port-channel 10 mode active
no shutdown
!
interface fortyGigE 0/116
description Leaf2 – Port Channel 10
no ip address
mtu 9216
flowcontrol rx on tx off
!
port-channel-protocol LACP
port-channel 10 mode active
no shutdown
!
interface fortyGigE 0/120
description Peer-Port 1 – Port Channel 100
no ip address
mtu 9216
flowcontrol rx on tx off
no shutdown
!
interface fortyGigE 0/124
description Peer-Port 2 – Port Channel 100
no ip address
mtu 9216
flowcontrol rx on tx off
no shutdown
!
interface Vlan 500
description Host VLAN
no ip address
mtu 9216
tagged Port-channel 1,10
no shutdown
!
interface Vlan 4000
mtu 9216
!untagged Port-channel 100
no shutdown
!
Cumulus Leaf Example with CLAGD:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5), ifup(8)
#
# Please see /usr/share/doc/python-ifupdown2/examples/ for examples
#
#
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0
address xxx.xxx.xxx.xxx/24
broadcast xxx.xxx.xxx.255
# Spine Link
auto spn1-2
iface spn1-2
bond-slaves swp49 swp50
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
clag-id 1
# clag-id needs to be unique on each clag, like a vlt domain id on FTOS
mstpctl-portnetwork no
mtu 9216
# Peer Link to Other LeafSwitch
auto pl
iface pl
bond-slaves swp51 swp52
bond-mode 802.3ad
bond-miimon 100
bond-use-carrier 1
bond-min-links 1
bond-xmit_hash_policy layer3+4
mstpctl-portnetwork no
mtu 9216
# CLAGD Peer Int Config
auto pl.4000
iface pl.4000
address xxx.xxx.xxx.xxx/30
clagd-enable yes
clagd-priority 8192
clagd-peer-ip 172.16.0.2
clagd-backup-ip 192.168.255.12
clagd-sys-mac 44:38:39:ff:00:01
# Switch Port Interface Configuration
auto swp1
iface swp1
mtu 9216
auto swp2
iface swp2
mtu 9216
auto swp3
iface swp3
mtu 9216
auto swp4
iface swp4
mtu 9216
auto swp49
iface swp49
mtu 9216
auto swp50
iface swp50
mtu 9216
auto swp51
iface swp51
mtu 9216
auto swp52
iface swp52
mtu 9216
# Bridge Configuration
#
auto br0
iface br0
bridge-vlan-aware yes
bridge-ports pl spn1-2 glob swp[1-4]
bridge-stp on
bridge-pvid 1
bridge-vids 500
mstpctl-portadminedge swp1=yes swp2=yes swp3=yes swp4=yes
bridge-mcsnoop 1
mtu 9216
# Bridge VLAN
# Host VLAN
auto br0.500
iface br0.500
address xxx.xxx.xxx.xxx/23
broadcast xxx.xxx.xxx.255
up ip route add 0.0.0.0/0 via 192.168.1.230
mtu 9216
The above are examples and not complete configs and you can’t just copy and paste it all and expect it to work in your environment. But it could be used as a starting point.
Bringing this all together. The Dell S6000 / S4810 combination allows you to create a scalable network design that provides predictable and consistent low latency and high throughput from end to end in the network. The configuration of the MLAG/CLAG/VLT is straight forward, and it provides management flexibility. You get to choose either FTOS or Open Networking such as Cumulus Linux as your switching software. With Cumulus Linux, it’s just like Linux, but wire speed non-blocking networking accelerated in hardware and fits into the normal management frameworks. For environments, such as Hyper-converged or Web-scale infrastructure, the network scales linearly, as do the systems that connect to it. Each time you grow, you get consistent, predictable and linear performance.
Here is an example of a high level diagram that might be appropriate for a small scale deployment. In this case the Dell S4810’s are used as both Leaf and Spine switches in a VLT configuration, with Dell N3048 or N3024 providing 1G connectivity. With this you could easily start with a single rack and scale to 8 racks. Each rack would have full 10G and 1G redundant connectivity. With say 24 Servers or Hyper-converged nodes per rack you would be able to support 192 servers across 8 racks. 40G QSFP+ ports are used for Peer Links, while the Leaf connects to the Spine using 40G to 10G break out cables.

Here is an example high level diagram of a medium density Nutanix Web-scale Converged Infrstructure deployment with Dell S4810 Leaf switches connected to S6000 Spine, which could be using FTOS VLT or Cumulus CLAG. As you can see this design is capable of scaling to 12 racks (576 nodes) with the S6000 Spine switches, and up to 52 racks and 2496 nodes, with the Dell Z9500 Spine switches. Dell Z9500 supports 128 x 40GbE ports. Both options have spare 40GbE ports still available for Boarder-Leaf connectivity (Cross Datacenter, or Internet routers etc).
If you wanted a higher density design you can combine S6000 in middle of rack or top of rack configuration for Leaf switches with S6000 or Z9500 Spine. This diagram provides a high level example of a high density configuration on a standard 48U rack. In this example the design would support 10 racks with 880 nodes with S6000 Spine, and 48 racks with 4224 nodes with a Z9500 Spine. With enough ports to accommodate the Boarder-Leaf nodes as well.
With densities in the datacenter increasing and the power consumption of the servers decreasing I can see an explosion of 40GbE Top of Rack (ToR) or Middle of Rack (MoR) Leaf switches. This would also provide an option to easily allow different bandwidth oversubscription models, 6:1, 4:1 etc as requirements change.
So far we’ve covered the Dell Networking switches by themselves, with FTOS and Cumulus, and then some examples combined with Nutanix. Dell is an OEM partner of Nutanix software and delivers the Dell XC Series Web-scale Converged Appliances. With Dell XC you have a number of hardware options for different use cases, and you can build a complete solution, including networking.
The following example shows Dell S4810 Leaf switches and Dell S6000 Spine switches with Dell XC series appliances. These appliances are 2U each and contain one node. Dell XC also has 1U options.
Final Word
Those of you who spend time in a modern datacenter will see I’ve drawn the diagrams with the switch ports facing forward, which is actually backwards. I did this to make it easy to draw, and because I like flashing lights. In real world environments the air flow of the switches would be reversed and the back of the switches, where the PSU is, would be facing the front of the rack, so that the cabling can be nice and tidy. In summary, where predictable, low latency, high throughput, linearly scalable networking is required, Leaf Spine architectures are becoming increasingly common (and are simpler than three tier IMHO). Web-scale converged or Hyper-converged Infrastructure benefits from the low latency, high throughput and linearly scalability the Dell Networking switches can provide. As does network virtualization such as VMware NSX. Dell Networking switches combined with Nutanix appliances, or Dell XC series appliances powered by Nutanix can deliver a unified and simplified high performance virtual infrastructure with greatly reduced complexity compared to a traditional three tier architecture. A great foundation for a private cloud or software defined datacenter.
—
This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2015 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.