Running a multi-gig Home Network in 2024
Table of Contents
Introduction
Why a multi-gig home network? What is a multi-gig network anyway? The second question is easy, it’s a network that runs at a speed higher than 1Gbps (usually one of or a combination of 2.5/5/10Gbps), which until a few years ago was almost exclusively reserved for the enterprise space due to hardware costs. But it isn’t anymore, there are quite a few relatively affordable options nowadays. The first question is a bit more complex, but the short answer is: because I can. The long answer is: because I enjoy tinkering with networking (to some extent), and because I want to have a fast network at home. My Internet Service Provider (ISP), Free (yes, really) already gives me lots of bandwith - 5Gbps down and 700Mbps up, so why not have a network that can handle as much as possible from that? I have a few bandwith intensive workloads at home - streaming from my NAS, video game streaming from the cloud / my gaming PC to my TV, remote storage for my Nomad cluster from my NAS, game downloads, backups, etc.
I embarked on this journey in 2022, but it took me some time to finish implementing everything and find the time to write about it, so I’m documenting it from here in 2024 (it didn’t actually take me 2 years to set up everything, I just did things sequentially with massive gaps in between). I’ll try to remember as much as possible, but I’ll also try to keep it up to date with the latest information I have.
Hardware
Router
One of the most important parts, the centerpriece of the network. I wanted to use a router with some very specific advanced features, which my ISP provided box doesn’t have (even if it has a lot of features for an ISP-provided one, such as WireGuard and OpenVPN, ad-blocking DNS, etc.), hence the need for a separate, proper one. Unfortunately there’s no way to entirely remove my ISP’s router (because it has an integrated 10G-EPON module to be able to actually use the fibre link to my home), but I can put it in bridge mode and use my own router behind it for all the logic and network management.
Requirements
- Multiple (at least 3) multi-gig Ethernet ports (ideally 10Gbps) - due to the physical constraints in my network cupboard, and the need for the ISP’s router in bridge mode, there simply isn’t the space for a switch there too, hence the need for a router with multiple multi-gig ports (at least one for the “WAN” link to the ISP router, and two others for the two rooms with lots of equipment). As for why Ethernet, the place I live in is wired for Ethernet in every room, following a standard that should give me up to 10Gbps, so I want to take advantage of that, even if I know that it’s not perfect and consumes lots of power/heat compare to SFP+.
- VLAN support (basic stuff)
- VPN support, both as a server but also, and more importantly, as a client with something like
ipsetsto direct only certain traffic through VPNs (to avoid a few specific geoblocks) - (relatively) affordable
- (relatively) low power consumption and (relatively) low noise and (relatively) small size - it’s going to live in a fully closed network cupboard, so it can’t be too loud or too hot, and there isn’t a ton of space either
- Wi-Fi optional to not needed, because I have a separate Unifi WiFi access point that I’m happy with, and in any case the network cupboard being weirdly located would result in poor coverage
- software support - I don’t want a router from a vendor that will stop updating it within a year or two, and then being stuck with outdated/insecure device at the core of my network. I was looking for something that has a good community around it, and/or a vendor that has a good track record of updating their devices for a long time
Options
It took quite some digging to find a router that fits all these requirements - most that did from the hardware side (multiple multi-gig ports) are “gamer routers” from the likes of Asus, Netgear and TP-Link (such as the Asus GT-AXE16000 or Netgear Nighthawk RS700S). Not only do they have Wi-Fi onboard which I don’t need, Netgear and TP-Link have… let’s say far from steallar reputation in terms of their software’s security and quickness/length of updates. Asus is a bit better because they use ASUSWRT (based on the Tomato network firmware), which is open source, and there’s a third party/community Asuswrt-Merlin project which builds on top of that, but it’s still not ideal. Their router that seemed to most closely match my requirements, the Asus GT-AXE16000, wasn’t supported by Asuswrt-Merlin and it was unclear when that support would land (and although I love contributing to open source, networking firmwares are way above my capabilities).
On the other side were the “prosumer” routers, such as those from Ubuquiti and Mikrotik. My previous network used to run on an Edgerouter-X, which was a great little router, so I was familiar with Ubiquiti’s software and hardware. However their product strategy (deprecating the Edge* line and focusing on the Unifi line, which is UI-managed and has less flexibility and features), as well as the overall quality of their software, and importantly in this case, the size of their routers (the Unifi Dream Machines are huge) made me look elsewhere. Which is a shame, because I plan on using their Access Points for WiFi, and it would have been great to be able to manage the whole network from a single central point. Mikrotik was a bit of a wildcard, I had never used their products, but they seemed to have a good reputation in the networking community, and their routers were relatively affordable. However, none had more than one multi-gig (and it was only 2.5Gbps) Ethernet port (like most vendors, and for very good reasons, they prioritise SFP+ ports for 1Gbps+).
By chance, I stumbled upon an article by ServeTheHome about an inexpensive 2.5Gbps mini PC that can serve as a router from Topton/Qotom (apparently appears under different brand names around the internet), based on a decent Intel CPU and with 4x Intel 2.5Gbps Ethernet ports. After some digging around I found a 6x 2.5Gbps port version, still using Intel NICs (Intel i226-V), which is important for compatibility with various OSes (the seller proposed shipping it with pfSense or OPNsense, but I guessed any *nix OS would be fine based on the specs and hardware involved). A big added benefit is the fanless design. The description of the device was exactly what I was looking for, and the price was very reasonable (~300 euros with shipping), so I decided to give it a try. I ordered it from AliExpress, and it arrived in a few weeks, much faster than planned.
Hardware specs:
- Intel Celeron N5105 (4 cores, 4 threads, 2.0GHz base, 2.9GHz turbo) - decent low-powered CPU, more than enough for a home router, even with multiple 2.5Gbps ports, VPNs, etc.
- 8GB DDR4 RAM - more than enough, in normal use it uses less than 1.5GB
- 128GB SSD - more than enough, the whole OS + config + everything takes around 5GB
- 6x Intel i225-V 2.5Gbps Ethernet ports - currently I only use 4 of them, but it provides some flexibility for the future (like connecting a backup 4G modem, or after I upgrade my ISP’s router to one that has more than one 2.5Gbps port, make an aggregate of two ports and load balance between the two for 2x2.5Gbps Internet)
The exact model I got doesn’t seem to be in stock anymore, but there are all sorts of similar ones with newer processors and various combinations of ports available on Amazon, Ebay, AliExpress.
Operating System
Since I went with custom hardware, I had to also pick an appropraite OS. I initially tried OPNSense, but the configuration by ClickOps wasn’t my thing, especially for my scenario where I have 3 ports that need to be bridged and configured the same; it was a lot of clicking around. I hesitated rolling a regular Linux distro such as Debian, but in the end decided to go for a more specialized one, VyOS. It is based on Vyatta, same as EdgeOS from Ubiquiti which I already knew from my previous router, the venerable ER-X. It’s entirely CLI-based - even better, a declarative CLI, like real networking software, and has a good community and documentation. All the configuration is in a single file (effectively) with some Git-like commit/compare/rollback/etc. features, including an extremely powerful commit-confirm which will commit configuration changes, and reboot if you don’t explicitly confirm them; this is brilliant for network equipment where a misconfiguration can easily cut off your own admin access. The biggest downside is that Long Term Support Releases are for paying customers/contributors only; for everyone else, you either have to compile your own, or use the rolling release, which is what I went with. I’m not too worried about it, because VyOS makes it very easy to backup the configuration (it’s just a text file automatically backed up to my NAS on each configuration commit), so if anything goes wrong I can just reflash the device and restore the configuration.
VyOS’ documentation is pretty good on all the basics, all options and features, and importantly for people getting started, there are configuration blueprints which show you how to achieve complete things (like a site to site VPN). There’s also a good community of people writing about it, like this blog series by Level Zero Networking, and many EdgeOS-related blog posts, docs, forum posts, etc. also apply or are easy to translate.
Switches
I needed 3 switches - one for the living room (for the TV, console, media box, etc.), and two chained ones for the office - one covering my homelab, (3 Nomad mini PCs + another mini PC), and the other for the rest of the office (NAS, work and personal laptops, Raspberry Pis running Home Assistant, assorted projects, another 3 Nomad nodes, etc.) in two different corners of the room.
Requirements
- A few multi-gig Ethernet ports (at least 2.5Gbps, but also at least one 10Gbps port for my NAS so that both my Nomad cluster and other uses can saturage their links) for the uplink and a few devices that can take advantage of it, but mostly 1Gbps ports; PoE isn’t needed because I have exactly one device that can use it, an Access Point, and it’s powered by a PoE injector
- “managed” (mostly VLAN support)
- (relatively) affordable
- (relatively) small and (relatively) low noise - 1 of them would be in the living room, and slightly visible, so a full rackmount switch would be out of place
Options
For switches, there were much more options available on the market, with quite a lot of flexibility in terms of port combinations. After going over:
- Mikrotik - CRS312-4C+8XG-RM with 4x combo 10G Ethernet/SFP+ ports, 8 1/2.5/5/10G Ethernet ports
- Netgear:
- Zyxel XGS1250-12 with 1xSFP+, 3x1/2.5/5/10G, 8x 1G Ethernet ports.
I settled on the Zyxel, it was by far the cheapest option (~190 euros vs ~270 for the Netgear GS110EMX), and it provides a very good combination of ports for my needs.
I use the SFP+ port on one of them for 10Gbps to my NAS, a 10Gbps Ethernet cable to chain two of the switches (thus getting 10Gbps between my Nomad homelab and the NAS serving its persistent volumes), and 2.5Gbps uplinks to the router. The web UI they come with is a bit weird and it took me reading the docs twice to understand how to properly configure the VLANs, especially trunks (having a port that allows multiple vlans), but it does the trick. Also, there’s an OpenWRT firmware for those switches with a lot of extra functionality if someone needs more than the stock firmware provides, or if they want to convert the non-managed switches (which are cheaper) to a managed version.
I use VLANs extensively to separate the IoT devices from the rest of the network, and to have a separate LAB network for my Nomad homelab Nomad.
Wireless Access Point
I’m currently using an older Unifi Access Point (AC-PRO) that only does WiFi 5 (formerly known as ac), doesn’t support WiFi6 nor WiFi 6E, and only has an 1Gbps uplink (technically two of them, but the second one can only serve as a backup) so it can’t fully benefit from my fast wired network. I plan on upgrading to a new model that can make use of a multi-gig uplink, and to prepare for the future WiFi 6, 6E and 7 devices on my network. WiFi frequencies are quite congested where I live (and I don’t help with 3 2.4GHz networks, 2 Zigbee networks and a ton of Bluetooth devices over the same 2.4Ghz band), so moving to WiFi 6/6E/7 should have a noticeable impact. Hopefully.
Overall I’m happy with Ubiquiti’s Unifi line for Access Points - they work pretty well and the UI is perfectly fine for configuring them (unlike for a router), so I’ll probably get a Unifi U7 Pro or a U7 Pro Max that has a 2.5Gbps uplink, which is perfect for my setup.
In terms of networks, I have 3 separate SSIDs (WiFi networks, mapped to VLANs) - one for Internet of Things devices to keep them separate and only give them internet access, one for guests, and one for everything else.
Next Steps
Overal, I’m very happy with the current setup. It works very well, is super reliable, allows me to do everything I want to do (including, importantly, conditional VPN for certain traffic), and is relatively easy to manage (no ongoing maintenance other than a VyOS update very few weeks).
Hardware
Outside of the upgrade of the WiFi Access Point, one day I might upgrade the router to one with a better uplink (or figure out a way to add one to the current router, e.g. with a USB3 adaptor) to make use of more of my ISP’s bandwith, but that’s not a priority right now.
Observability
An important part of any network is observability - knowing what happens in it, bandwith actually used (to see if there are any bottlenecks that merit an upgrade), what devices are connected, how much they use (e.g. my washing machine downloading gigabytes from the Internet would not be normal), etc. I’ll use Grafana Alloy with Grafana Cloud-hosted Prometheus, monitoring the various devices with the old and battle tested SNMP.