The Story of pfSense, VLANs, and the one-armed router

The Situation

I have an older SMB router (Draytek 3900) at home that is end of life.  I was looking at alternatives and wanted to test out pfSense.  I borrowed a Small Form Factor (SFF) PC from a friend to do the test and connected it to a couple of isolated VLANs at my house.  My plan was to create two new VLANs for testing and have them broadcast as wireless networks from my APs.  I would also let this device connect to the two internet VLANs (wired and 5G) for Internet access.   I setup the switches and APs as needed and connected the PC.  In this case, I was planning for a "one-armed router" config where the pfSense router just has a single network adapter connected and all the different networks are passed over that as VLANs.

According to the docs, this should work...

The Problem

After installing pfSense on the PC I ran the console configuration process.  The configuration asks you to define a WAN interface and a LAN interface.  Since I only had one, I had to create VLANs which it will let you do as part of the setup process. Cool, Cool...

At the end, it gives you an HTTP URL and tells you to go there to continue setup.  No matter what I tried, I was unable to connect to the page.  I initially tried to define all the VLANs I would need for the project (2 WAN, 2 LAN).  I decided to simplify:  1 WAN, 1 LAN.  Same result.  I tried to define just the LAN interface (since that's the one that was used in the URL at the end).  However, it won't let you do that.  You must have a WAN port defined to proceed.

I went through all my Switch and AP configs multiple times trying to find the misconfiguration. Why can I not connect to this page?  I eventually gave up on the wireless network plan and just connected a cable directly to the switch.  No change...

The Solution

So, to the Internet I went.  I found several walkthroughs that covered my scenario and it appeared they were all doing the same thing I was...and reporting success.  Eventually, this lead me to an article that mentioned making sure the network adapter supported VLANs.  I hadn't even considered this as an option.  This PC is a 2012 vintage model.  So, it was worth checking.  The adapter is a RealTek 8168 integrated adapter.  I checked the docs and it's supported...with a message that specifically says it also supports VLANs.

Well, crud.

I almost moved on until I ran across another individual in the comments who was talking about MTU sizes.  At first, this seemed like it would be unrelated.  This is ethernet directly not some sort of tunnel config.  I read on out of desperation.  It turns out that some ethernet adapters support VLAN tagging, but don't allow a big enough ethernet frame to include the VLAN tag overhead.  This adapter is one of those.  Default ethernet frame is 1514 bytes.  The VLAN tag is 4 bytes.  Maximum frame size for this card is...1514 bytes.  It's too small.

At this point, this is all theory and I look like this guy:

Now, that I have a working theory about the problem, how do I fix it?  Well, it turns out you have to have a WAN adapter configured in pfSense, BUT you don't have to have a LAN adapter.  If you configure it this way (and only this way), it will present the webConfigurator interface on the WAN network.  Since I had one network that I could configure without VLANs, I was able to access the webConfigurator.  From there, I was able to lower the MTU on the NIC and create all my VLANs (also with lower MTUs).  Now, I can happily access the page from the LAN side and the WAN side (automatically it appears) stopped accepting connections for the webConfigurator.

Let this be a lesson to you and may it save you similar problems in the future.