Upgrading my home network to UniFi

One morning, after waking up and dragging myself to my desk, I logged into my computer only to find out that I couldn’t connect to the Internet. Annoyed, I opened my router’s interface in my browser to go diagnose the issue. I began to ask myself if my ISP had broken something upstream again, or maybe if I had accidentally uninstalled my network drivers again. However, the connection to my router timed out!

At this point, I began to realize this would be a far more annoying problem to deal with today. This meant that there was either A) total failure of my downstairs networking equipment or B) the WiFi access point had failed. To rule this out, I went to my website from my phone, which was hosted on my home server downstairs. Unfortunately, it loaded fine…

That meant it was the fault of my WiFi! For context, I use a mesh WiFi network, because running network cables through my house would be a ton of work. Specifically, I use an older revision of the NETGEAR Orbi. There’s an access point downstairs where the router is, and one upstairs at my desk that I connect to via Ethernet.

I went to the upstairs access point’s web interface, but everything seemed fine. Next, I looked at the downstairs access point from my phone, and I then realized what had happened. The two access points were disagreeing on their connection state:

I tried moving the access point, factory resetting it, replacing the cables, but nothing worked. At this point, I considered the hardware dead. For a brand with the marketing line “no more dead zones or dropped conections”, I would say that my entire upstairs floor became both of those.

I could still reach the downstairs access point, but it was comparatively slow, reducing my network speed from 300 Mbps to about 20. I coordinated a plan with my dad, and we decided that we would replace the WiFi setup entirely.

I looked through a lot of consumer options, but every brand had its own downsides, and I wasn’t sure what to go with. Jokingly, I suggested going with UniFi, as I had always been interested in UniFi but never had the justification to replace my current hardware. To my surprise, he agreed to my idea, so I started researching. Here goes a huge rabbithole!

World’s dumbest woman vs. networking equipment

I am going to start this post with a clear statement: I am very bad at networking. On the very first server I ran, two of my friends had root access in case I broke WireGuard again. There was also a time I tried adding a second static IP to my server and I brought it down for three hours.

I don’t have a lot of freedom to play around with my homelab’s network for a few reasons:

• I am unemployed.
• My homelab network is actually just my home network. There’s pretty much no isolation from the rest of my network, so if I break something in a bad way, I break my residential connection and my family members get mad at me.
• I made the terrible mistake of hosting “important” services on a home server. Whoops!

However, beyond that, I vaguely know how these things work. I was focused on replacing my access point(s), but the entire selling point of UniFi is that it’s unified, so I also would want their router and at least one switch.

Usually, I would be annoyed about having to replace every single component in my tech stack to have a unified ecosystem (cough), but I was actually pretty happy about having a reason to replace everything. The current switch I have downstairs is extremely old and slow, and the current router/firewall I have is a early-2000s Dell OptiPlex running pfSense.

It took a bit of research and going back and forth to decide, but I eventually settled on the following:

• The UniFi Express 7 as my “Cloud Gateway”, which is the fancy UniFi branding for the central device. This acts as my router, firewall, and main hub for controlling everything.
• Two Lite 8 PoEs for switching. I literally just went to the “Utility” section of their site and sorted by the minimum price. I considered the Flex, but paying $10 more for 3 more ports was worth it.
• A U7 Pro for the upstairs access point. Did I need WiFi 7? No. Did I get it anyways? Yes.

My dad was currently on vacation with my brother, so he ordered it for me to pick up at a local Micro Center. My dad being on vacation was a net positive, as he was with a person who also owned UniFi, and so he was able to learn more about UniFi from direct experience. If you’re reading this, thanks John.

I picked up the order from Micro Center that evening. As I walked up a set of stairs, my mom said I was carrying the bag like a newborn baby, so she decided that it would now be named Marcus. Sure, whatever, welcome to my household Marcus.

Initial setup

I originally was going to wait to replace everything until my dad got home, but the temptation of shiny toys on my living room counter combined with me running dnf up at 6 MiB/s got to me. I unpacked everything and got the cables setup, and then I went downstairs to assess everything.

The current equipment resides in a dilapidated corner of my garage. Everything rests on a single shelf, and the wiring was so bad that I can’t post it on this page without a content warning. A single power strip provided power for every device, with a cacophony of Ethernet cables presumably running into a switch I couldn’t easily see.

This, obviously, had to go. However, a lot of this hardware was things that I did not want to touch. Removing and cleaning up everything without any help would’ve taken hours, which I didn’t want to do just yet. Instead, I decided on a temporary compromise, which I would clean up with my dad after he got home next week.

While the power strip on the shelf was completely full, there was another power strip on the opposite side of the garage. I was able to bring the power strip just over to the shelf, and simply placed the UniFi equipment next to the shelf, powering it from the second strip. This was a complete mess, but it worked!

Instead of being a responsible sysadmin and telling anyone about downtime, I just unplugged the Ethernet cable from my ISP’s equipment and watched as everything started to blink red. Even when I know I’m doing everything right so far, seeing all of the status lights get mad is a very uncomfortable feeling.

I plugged the Ethernet cable into the Cloud Gateway, and I sat there as the cutesy screen flashed to life. I got out the UniFi mobile app, created a new account, and connected to the gateway. The network settings weren’t detected automatically (presumably because I have a static IP), so I had to manually enter the IP/gateway.

And… it just worked! Seriously. I sat there in shock as it just configured everything perfectly without my input. Compared to the first time I set up pfSense, UniFi took a lot less button pressing and head scratching to get it to work. It turns out paying for convenience does work sometimes! Maybe Apple was right.

After the WiFi was up, I had to get all the Ethernet connections back online. I plugged in my home server, Philips Hue bridge, and a few other things that were downstairs. I left my old home server (which had been unused for months) and the pfSense box sitting there, disconnected from the rest of the network, but still powered on for no reason. Is it possible I just violated the computer version of the Geneva Conventions?

I fiddled around in the UI on my phone to get the HTTP ports forwarded to the server. My site was back online, after just a few minutes of downtime, so hopefully nobody noticed anything. Nice! While the other devices around the house still had to be connected, my biggest priority was complete, and I was relieved.

I spent some time playing around in the UI, and I really enjoyed it. Everything seemed easy to get to and simple to configure. It’s a lot nicer than dealing with crusty NETGEAR interfaces or having to read FreeBSD logs.

When I first opened the page, I saw a list of devices that were on my network, as well as the services they were connecting to. Seeing the Let’s Encrypt logo inside of my router page struck deep fear into me, so I instantly turned that feature off. I presume it just inspects SNI or something.

The upstairs access point

Of course, the entire reason I replaced this network was for the upstairs access point that failed. I brought the switch and access point upstairs and plugged it in on my desk. The access point started as soon as I plugged it into the switch, which was the exact moment I became a Power over Ethernet enthusiast.

I looked at my phone, and I got a new notification to “adopt” the upstairs AP. I admire the term “adopt”, it could have been “claim” or “register” or “setup” or literally anything else, but instead it’s the one term that makes me feel like a proud mother. Perhaps my mom was right about naming it Marcus.

Anyways, after I hit the button, it took a few minutes to configure and then I got one more prompt for the switch it was connected to. After it was finished, I went to go configure the access point, and… it was already setup???

The UniFi app already set up the upstairs AP to mesh automatically, registered the switch properly, and both WiFi and Ethernet worked fine upstairs. I was very happy about this, so I ran a quick speed test on my computer, and went to collapse in bed for a bit.

When I got out of bed, I walked over to my desk and saw Discord wasn’t loading. I began to wonder if my Discord client mod broke, but I realized that the entire network was offline. Again.

At this point, I was angry but amused at the situation. I bought into a new ecosystem expecting it to just work, only for the same exact scenario to happen once more. However, I doubted that the hardware was faulty, and I assumed that it might have just been a quirk in setup. I just rebooted the AP and walked away.

When I woke up the next morning, it happened again, so I began to troubleshoot. I knew that AP meshing wasn’t a very popular feature as most people just wire devices directly, so I suspected it was some issue involving that.

Going to the app, I could see that the AP was marked as “Isolated”, and the upstairs switch was offline:

From a quick search, “isolated” apparently means that the AP was detected but not connected. This felt oddly familiar to the issue I had with the Orbi. The upstairs switch being offline makes sense, as the AP is the only way it can wirelessly communicate to the gateway.

I spent a lot of time investigating this and getting nowhere. I restarted the Cloud Gateway as well, but nothing really changed. I tried resetting and re-adopting the AP, but it still had the same issue. Eventually, after poking through the web UI, I found something concerning:

The Cloud Gateway had meshing on, which I expected (it should be meshing to the upstairs AP). However, the tooltip showed that it wasn’t meshing to the AP, it was meshing to itself. This makes absolutely no sense, and it means that something must have gone wrong in my setup - perhaps an infinite loop of some kind. Bad Marcus!

I restarted the AP and Cloud Gateway, and took a look at the settings of the AP. Unfortunately, I didn’t take a screenshot of what I saw in that moment, but here’s what I see now:

You’ll notice that, in this image, the uplinking checkbox is greyed out. This is because the device is currently uplinking to the Cloud Gateway for an Internet connection. However, when I looked at this for the first time, both checkboxes were enabled and greyed out.

The connection is supposed to be provided by the Cloud Gateway, which would downlink to the AP. Because of this, the AP would need to have uplinking enabled. However, it had uplinking and downlinking enabled, so at one point the AP automatically reconfigured itself to reverse the flow of traffic. The Cloud Gateway would try and connect to the AP for the network!

This would cause the AP to get disconnected on my network, stranded as it couldn’t figure out how to connect back to the Cloud Gateway. This manifested in the AP slowly blinking blue, which indicates that there wasn’t a connection. It would eventually become “isolated” and bring down the network upstairs, but still broadcasting the SSID, causing the upstairs to turn into a network black hole.

From my understanding, the UniFi app would automatically enable meshing in both directions when pairing the AP. I just had to reset and re-adopt the AP, wait for it to “get ready”, and rush to disable downlinking before it automatically kicked in. Problem solved!

Not the first time I’ve dealt with computers isolating, I guess.

Now that this AP was in place, I was able to connect all of the devices around the house. There were a lot of painful input methods across the various devices, but my (least) favorite was the Nest thermostat. I had to rotate the dial and press down to select a letter, one letter at a time. Who designed that???

Creating VLANs

I wanted to get VLANs going for the house as soon as I could. I wanted to split the home devices, homelab stuff, and IoT devices into three separate networks. I didn’t want to setup any isolation rules just yet, though - my current goal was just setting up everything and configuring the devices.

I created a VLAN in the UI and watched as the Cloud Gateway seemingly reconfigured itself. My status monitoring alerts detected a brief connection loss to my server. However, even though the alert said the server went back online, I didn’t have any networking upstairs. Not again…

Once more, the upstairs network was broken, but the downstairs network was fine. This time, the scenario was even more peculiar. I could ping addresses from my LAN and the Internet just fine, but I couldn’t SSH into my server. Confused, I went to the Cockpit web UI for my server, but it wasn’t loading. Great!

I was very confused on how this happened, especially because I hadn’t even assigned any devices yet. Having a VLAN enabled broke the entire network, even if it wasn’t being used at all. Disabling it fixed everything, but I obviously still wanted to use VLANs.

Looking at the network tab in Firefox, it seemed like the connection was being dropped halfway through a request. This also lined up with what I saw with SSH, it would get halfway through the handshake and just stop responding. I got out my phone and got into my server with Tailscale, then tried a basic TCP connection through Termux. As expected, it just gave up after I sent a certain amount of data.

I was very confused at how this happened, especially when the UniFi web page reported everything was okay. In fact, I even connected to the Cloud Gateway perfectly, but anything else on the local network or Internet was broken. I was able to see that my device was assigned an IP, but the Cloud Gateway couldn’t even tell I was making any traffic:

I ended up stumbling around and found a button to download a Wireshark capture. This was really cool (and a little scary that it was so easy), so I let it run for a bit and opened it up in Wireshark:

Sure enough, I got a .pcap in a .tar in another .tar, and I ran sudo dnf install wireshark from my mobile hotspot to take a look. One painfully slow repository update later, I opened Wireshark and saw that traffic would… indeed just eventually reset. The capture wasn’t very helpful, unfortunately.

At this point, I was super confused. No amount of searching my problems would help, as I could barely find information about meshing, much less this specific problem. At this point, I had spent my entire evening working on this, and I was starting to get stressed out. I didn’t want to post on a support forum just yet, so instead I asked some friends for help.

A few days ago, my friend Kaz tested UniFi meshing for me before I bought everything, just to make sure my setup would work fine. I let him know I was having issues with VLANs, and he let me know he would take a look after he was done with his FFXIV raid night. However, he suggested I just take the AP downstairs and plug it in physically for a bit.

God damnit.

I spent maybe six hours troubleshooting this, tearing my hair out trying to get this to work, changing all sorts of overrides and VLAN configs. And all I had to do was plug it in for a few minutes. Lesson learned, I guess.

My assumption here is that the updated VLAN configurations weren’t being properly synced to the AP when it was meshing. Having it plugged in via Ethernet allows it to sync whatever information is required without any potential interruption. I guess I’ll see how accurate this theory is if I make a fourth VLAN in the future.

Now that the VLANs actually worked, I was able to force assign each device a IP/VLAN based on its MAC address. This made it a lot easier to set up the IoT devices, since I just had to set some overrides and reconnect them.

Some devices (like my WLED controllers) have their own static IP settings, so I didn’t specify a fixed address for those devices. I figured that if I set a fixed address in UniFi, but the device itself had a separate fixed address, they would fight over which address to force it to and cause issues.

Again, I’m not actually separating any traffic here. I’m using VLANs mainly for organization, not for the firewall, and the SSID isn’t forced to a specific VLAN. I eventually plan to make a second SSID for the IoT devices, but I don’t want to go around and change the connection info on everything for a second time right now.

In the future, I might set up strict firewall rules between everything (e.g. IoT VLAN can only reach the Home Assistant IP), but that’s not very important for me at the moment. I’m happy with how it is right now, and my threat model does not include my toaster being hacked and traversing my home network (but I encourage you to try).

Getting scammed by Verizon

While I was happy with my setup so far, there were two glaring issues I had with it, and both were the fault of my ISP (Verizon):

• My internet plan is currently capped by our plan’s speed. I could clearly push way more than this with the new hardware, but it’s not gonna be any faster outside of the local network.
• We don’t have IPv6 in the house. I live in an area that should be able to get IPv6, but I never got it working with pfSense, and UniFi didn’t automatically detect IPv6 when setting everything up.

I asked my dad to recheck the pricing for the plans we were offered. I vaguely remember discussing plans with him years ago, and we decided that the faster options were too unreasonably expensive for us at the time. Here’s what we saw in our account settings:

That is extremely unreasonable. $269? Seriously? To add insult to injury, we currently pay for 300/300 Mbps for $159/mo, and the 500/500 plan is $10 cheaper. We were so grandfathered into a plan for so long that a cheaper option became available, and they just never told us. Sigh.

I started to research, because that felt extremely wrong. Looking at their marketing page, gigabit is… $99??? So the price they offer on their marketing page is $60 cheaper than what we’re paying now for three times the speed, and the price they show us in the account settings is over double that marketing price.

I began to think that maybe the price was just different due to some quirk of my region, since I know that network plans can depend on the hardware and lines installed in your area. I put in my neighbor’s address (on the same street) onto their marketing page, and it said gigabit was available for $99. Great.

I assumed this had to be something related to our current plan. Maybe this is the result of a shitty contract, or maybe our grandfathered plan is ruining our upgrade path. Perhaps there’s some hidden fee that we weren’t aware of, even though the marketing page says the price doesn’t factor in any taxes or fees.

I thought that we should probably call up my ISP and ask what the hell was going on, and also maybe see if we could get IPv6 in the process. At this point, it was the end of the week, so my dad called them up when their phone line opened on Monday morning.

He let me know that he tried his best to talk with the support representative, but they wouldn’t budge on gigabit’s pricing. At this point, I began foaming at the mouth with rage, ready to go start up my own NotNite ISP and fuck up BGP in the process.

I don’t think this is the fault of the support rep, I’m sure they’re just following some company guideline (or maybe I’m missing something in the fine print), but I was still annoyed. Eventually, I think my dad just gave up and moved to the extremely overpriced gigabit plan. He’s paying for it, so I presume he’s fine with the price, but I still feel bad that he’s getting scammed. :(

Getting scammed by Verizon (IPv6 edition)

Beyond the pricing issue, I still wanted to figure out IPv6. The support rep informed us that IPv6 should already work for our plan, but I didn’t believe that for a few reasons:

• There is no information about IPv6 anywhere in the account settings. The plan specifies our static IPv4 configuration, but there’s no mention of IPv6 anywhere, not even in the plan’s description.
• UniFi didn’t setup IPv6 automatically when I setup the Cloud Gateway for the first time, so I presume it couldn’t find anything. It’s possible I need to configure it manually (like I did for my IPv4 address), but again, there’s no information to use.
• Back when I used pfSense, I tried (and failed) to get a IPv6 prefix. I doubt it would magically work now…

I decided to turn on IPv6 in UniFi and see if it did anything automatically, but nothing happened. I tried both SLAAC and DHCPv6, changing the prefix size, but still nothing.

At this point, I started to think that either the support rep was wrong or my hardware is too old. I have no idea what model ONT I have, and to my knowledge that’s a completely different layer where it wouldn’t impact IPv6, but I considered it could be the issue based on the support rep’s wording.

Confused, I asked self-proclaimed IPv6 expert Maddy about what buttons to hit, but we couldn’t figure anything out. I also asked actually-certified IPv6 expert Kaz about this scenario, and he let me know that the most likely configuration for IPv6 would be through SLAAC. However, he noted that our account is a business plan with static addressing for IPv4, and apparently that just doesn’t work with Verizon’s IPv6?

This would make a lot of sense. It’s possible we needed to ask for a static IPv6 prefix on our account, since we already had a static IPv4 address, and you might not be able to mix static+dynamic plans. The support rep might not have noticed that we have a static address, which would require our plan to be reconfigured.

However, we called back, and it turns out that this just isn’t possible. The support reps were very confused and I guess they really just can’t do it. I think IPv6 just isn’t rolled out to business customers with static addresses - looking at the forums, it looks like this is the case, and this post is from March of this year. Ouch.

For now, I’ve given up on IPv6, even though I really want it. I hope Verizon actually makes some progress on this, since it looks like they provide IPv6 to other residential customers (and even business customers with dynamic addresses, I think?).

Final thoughts

UniFi itself is great. I’m happy to have cleaned up my home network quite a bit, and the user interface seems a lot more friendly than what I was dealing with before. I also enjoy the fact that integrating new UniFi products into the household should be extremely painless, but I’m not in the market to spend any more money on networking equipment right now (lol).

Speed wise, even though I’m upset about the insane pricing, I’m enjoying the new plan, even though I’ve yet to make good use of it. I was getting about 930/930 downstairs and 650/890 upstairs. As expected from mesh networking, the upstairs AP is slightly slower. Maybe I can tweak the AP’s settings in the future, but I don’t care enough right now.

The upstairs AP still needs to be properly mounted somewhere. Right now it’s just laying against the side of my desk, and the switch is taking up space where I’d usually put my phone. I’m not entirely sure where I’ll mount it, especially because I need an Ethernet cable to the switch, so bringing it through my room’s wall into the hallway might be required.

I still need to make some final touches on the network configuration (e.g. moving some things into VLANs), and I need to go downstairs and clean up the Cthulhu-ass cable management, but otherwise I’m satisfied with everything right now. In sysadmin tradition, let’s see how long it takes for me to hate something about it.