We recently changed over from Vodafone VDSL to Chorus UltraFast Broadband (UFB) Fibre Max with Vodafone (See Vodafone Broadband Plans). The new Fibre connection is amazingly fast, but we soon discovered some limitations with the Vodafone router. I found out that the Vodafone UltraHub doesn’t support static routing. This meant that most of our home network couldn’t access the Internet. This article will show you how I worked around this limitation and created a Home Fibre Router / Firewall in the process. Warning: Sorry to anyone reading this in Australia who has NBN, feel free to migrate to New Zealand (Or Singapore).

You might be wondering why the featured image on this article is the Palo Alto Networks Logo and one of the 3020 Firewalls. When I ran into this problem of the Vodafone UltraHub not supporting static routing I contacted a friend at PAN to see if there was anything they had that might be able to help. I was offered a trail of a Virtual Edition Palo Alto Firewall (VM-300 – 4 vCPU, 9GB RAM), which I am keen to try out. I’ll definitely be writing about it when I get hold of the trail, and letting you know how it compares to a roll your own home solution. Thanks Palo Alto for being willing to work with me on a trial. All going well I may well switch over to it, as it’ll make connecting to other people around the world, who also have PAN’s much easier.

I’ve previously written about how we converted our Home Area Network (HAN) to a Leaf Spine Architecture, and how to create a low latency L2 network with Dell Force10 and Cumulus Linux. So we have a fairly complex set up here and a lot of servers and other devices and lots of VM’s on different VLAN’s and subnets. So when our Fibre connection came along it was a bit frustrating that most of the network couldn’t access the Internet. To complicate matters we also have a Vodafone SureSignal so we can get mobile service as we are in a coverage dead spot. The SureSignal requires special configuration, we’ll get to that later.

We have OSPF as our main home routing protocol, and I have the spine now on 2 x Dell S6000 32x40GbE switches, with the main leafs being 2 x S4048’s. We have our 1G boarder leaf’s running 2 x Dell N2048’s. All of the 1G segments, Wifi access points, and Internet access is off the boarder leaf.

I knew I needed something that could talk OSPF to the rest of our switches for all the VLANs and subnets as that would make it easy to get the routing correct and minimize the number of static routes. It needed to be able to support NAT and be stateful (and allow IPSEC) so it could handle the Vodafone SureSignal. It had to be able to talk to the Chorus GPON ONT directly, as any other router / modem would. I also had to minimize downtime in the process (lack of Internet access is a P1 at home and I have to apply for a maintenance window in advance unless I want to hear a lot of screaming). I decided to try my hand at setting up a Linux VM and seeing if I could make that work in the interim until I could find a more robust solution.

The speed through our original Vodafone UltraHub wasn’t too bad, as you can see below.

I’ll explain the physical network set up as well as the VM configuration as we go. First let’s look at the physical network and how I connected the ONT (the box that sits inside the house) to the environment without having a dedicated modem/router. This ended up being easier than I had thought.

Regardless of who your ISP is, one of the physical network providers provides the actual connection. In Auckland that is Chorus. They use an IPoE connection, which means basically it’s DHCP on an Ethernet link. This is much simpler than using PPPoE or other types of connection. Some Fibre connections may still need PPPoE, so please check with Chorus or your provider / ISP as you may need some additional things to get a Linux based router working with PPPoE. For the example I’ll give we just used an Interface with DHCP enabled. The physical connection comes in on a tagged VLAN. In our case that was VLAN 10.

The original Vodafone UltraHub had been configured to allow tagged packets from VLAN 10 get a DHCP IP address. All I needed to do to allow our physical home network access then was to create a VLAN 10, allow tagged packets to the relevant ports (making sure all the switches would pass it to each other), and then create a port group on my VM hosts with VLAN 10 tagged for the VM to connect to for the external network. Apart from having to configure VLAN 10 on quite a few switches and port channels it was surprisingly easy to get working. Don’t make the mistake I made at one point and set one of the switch ports to be untagged on VLAN 10 and connected that port to the ONT, that won’t work, as the ONT is sending tagged packets.

Once the physical network connectivity was established I created my Router VM. You can use any Linux distribution you like, in my case I chose CentOS. I configured the VM with 2 vCPU and 2GB RAM, with a 100GB Disk. I gave it 2 NIC’s, one on the Internal Network (zone=internal) and one on the External Network (zone=external). I did a minimal install of the OS and then added in Epel-Release, Quagga (dynamic routing), Traceroute, IPTraf, TCPDump, Links, DDClient (to update DynDNS), NSLookup (Bind-Utils), WireShark (wireshark-gnome), Net-Tools. The minimal install has SELinux, FirewallD, SSH and the basics that you need.

Here is the config I used on the Linux VM:

# cat /etc/sysctl.conf

# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Optimize Memory Management Settings 
vm.overcommit_memory = 1 
vm.dirty_background_ratio = 5 
vm.dirty_ratio = 15 
vm.dirty_expire_centisecs = 500 
vm.dirty_writeback_centisecs = 100 
vm.swappiness = 0

# Network MTU Probing - Michael Webster
net.ipv4.tcp_mtu_probing=1

# Optimize Network Stack and Memory Buffers 
# Increase TCP max buffer size setable using setsockopt()
net.core.rmem_max = 536870912 
net.core.wmem_max = 536870912 

# Increase Linux autotuning TCP buffer limit
net.ipv4.tcp_rmem = 4096 87380 536870912
net.ipv4.tcp_wmem = 4096 65536 536870912

# Increaes max backlog of packets and congestion control method
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_congestion_control=htcp

# advanced network stack tuning
net.core.somaxconn = 65535
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 65535


fs.aio-max-nr = 1048576
fs.file-max = 6815744

# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos_router/root rd.lvm.lv=centos_router/swap rhgb quiet elevator=noop iommu=soft apm=off numa=off transparent_hugepage=never vmw_pvscsi.cmd_per_lun=256 vmw_pvscsi.ring_pages=32"
GRUB_DISABLE_RECOVERY="true"

# grub2-mkconfig -o /boot/grub2/grub.cfg

# cat /etc/quagga/ospfd.conf 
!etc/quagga/ospfd.conf
hostname router
log file /var/log/quagga/ospfd.log
interface ens192
 description Internal
 ip ospf network broadcast
!
router ospf
 ospf router-id 192.168.xxx.xxx
 network 192.168.xxx.xxx/24 area 0
 default-information originate always
access-list localhost permit 127.0.0.1/32
access-list localhost deny any
line vty
 access-class localhost

Then I configured the firewall using firewall-cmd

# Put the correct interfaces into the correct zones
firewall-cmd --zone=internal --change-interface=ens192
firewall-cmd --zone=internal --change-interface=ens224

# Allow ospf to work for the Internal network
firewall-cmd --add-protocol=ospf --permanent --zone=internal

# Set internal target to allow traffic from Internal network
firewall-cmd --permanent --zone=internal --set-target=ACCEPT 

#Forward email and https traffic to internal system
firewall-cmd --permanent --zone=external --add-forward-port=port=25:proto=tcp:toport=25:toaddr=xxx.xxx.xxx.xxx
firewall-cmd --permanent --zone=external --add-forward-port=port=443:proto=tcp:toport=443:toaddr=xxx.xxx.xxx.xxx

#reload firewall config and display zone settings

firewall-cmd --reload

firewall-cmd --zone=external --list-all
firewall-cmd --zone=internal --list-all

It was at this stage that I ran into trouble with OSPF not wanting to work properly on the Linux VM. None of the neighbours would respond correctly. After some Googling it looked like the most obvious issue was going to be the MTU and sure enough the logs supported that as the physical switches were trying to send packets of 9198 bytes and the Linux VM could only accept 9000. Due to the various different switches in the environment I found I had to configure multiple different MTU settings to get everything to converge. On some switches it was MTU 9018 and on some others it was 9000. After getting the MTU’s correct across the different switches everything started working.

After switching over to the Linux Firewall the speed of the connection was even better. However the Vodafone SureSignal would not work. I stumbled across a configuration article for complex networks on the Vodafone NZ web site. This advised to open up and forward port 123, 500 and 4500 UDP from the Firewall to the SureSignal and ensure the SureSignal can resolve DNS correctly. In spite of doing this, and calling Vodafone NZ support it was not possible to get the SureSignal to function. In the end I decided to configure the Vodafone UltraHub and SureSignal on a separate VLAN so they could directly communicate and then put the Linux Firewall VM on that same VLAN so the rest of the network could have Internet access. This resulted in some Double NAT, but everything is now working.

Final Word

I hope in the future we don’t actually need a SureSignal and we can get proper mobile service, maybe when the new Vodafone NZ 5G network is launched in December 2019. Then we’ll be able to directly connect our Linux Firewall VM to the ONT and keep it simple (and fast!). I’m also hoping that when I try out the Palo Alto Networks VM-300 that I can get around the annoying issues I had with the SureSignal and that it works directly via the PAN to the Fibre ONT. Time will tell, and I’ll write about how the evaluation of that goes when it’s completed.

I’ve also been in contact with Chorus regarding their 10G Trial. I have the equipment to utilize this and enough devices at home and plenty of requirements to upload and download large data sets to the USA and elsewhere. The 10G service is due to go live in 2020 anyway, when it does, I’ll definitely be looking to get on it, even if I don’t make it onto the trial.

If you don’t have a SureSignal then you won’t have any of the problems I ran into and you’ll be able to connect your ONT directly to your Linux host or VM to act as your Internet firewall using the above info. Alternatively you could check out a Palo Alto Networks VM-300 (or one of their physical firewalls) if you run a corporate environment and want something robust and simple to use that also offers context aware security. I hope this is helpful in some way, let me know in the comments below.

This post first appeared on the Long White Virtual Clouds blog at longwhiteclouds.com. By Michael Webster +. Copyright © 2012 – 2019 – IT Solutions 2000 Ltd and Michael Webster +. All rights reserved. Not to be reproduced for commercial purposes without written permission.