I recently ran into a frustrating and baffling issue: my FreeBSD jails were experiencing painfully slow downloads, even though the host system’s network was perfectly fine. I spent over a week trying to figure out what was wrong, as this was a problem on my updated 14.0-RELEASE server.

The setup for my jails is a bit complex. They’re connected to my LAGG bundle (using my em0 physical interface) through a series of epair and if_bridge interfaces. It was a head-scratcher because all the traffic on the host machine—direct to the LAGG bundle—was fast. The problem was isolated to traffic coming out of the jails, which meant it was crossing the virtual epair and if_bridge interfaces.

I finally stumbled upon a comment on r/FreeBSD that offered a potential solution that, at first, seemed illogical. The fix was to disable several hardware offloading options on my physical em0 interface by adding the following to my /etc/rc.conf file:

ifconfig_em0="up -tso -lro -vlanhwtso -rxcsum -txcsum"

I gave it a try, and to my amazement, it worked perfectly. My jail downloads immediately returned to full speed. But even though it fixed my issue, I still had a burning question: why did this work?

---

The “Why”: Offloading, Jails, and a Key Conflict

The solution lies in the networking concepts of TCP Segmentation Offload (TSO) and Large Receive Offload (LRO). These are features built into network cards to reduce the workload on the CPU.

• TSO lets the network card take a single, large chunk of data and split it into multiple smaller packets before sending them out.
• LRO is the reverse. It allows the network card to take a bunch of incoming packets and combine them into one large packet before passing it to the operating system’s kernel.

These features are usually great for performance, but they can cause major problems in virtualized environments like FreeBSD jails, especially when using virtual bridges. Offloading these tasks to the network card can interfere with the way the virtual interfaces handle and process packets. In a sense, the hardware and software are getting their signals crossed.

In my specific case, it seems the aggressive packet handling of LRO was conflicting with my jail’s network stack. As I learned from this excellent LWN.net article, LRO is particularly notorious because it can reassemble packets in a “lossy” way, which can break things that rely on perfect packet integrity. This is likely why it was replaced by a more robust solution (GSO) in the Linux kernel.

Is This a Common Problem? It Depends on Your NIC

You might not ever run into this specific issue, and that’s probably because it depends entirely on the Network Interface Card (NIC) you’re using. These hardware offloading features, like TSO and LRO, are specific to certain NIC models and their chipsets. It’s a key feature of more modern or enterprise-grade network cards.

Commonly, you’ll find these offloading capabilities in cards from major manufacturers like:

• Intel: Many of their cards, especially those using chipsets like the 82574L, I210, and I350, have robust offloading features. You’ll often see these in server motherboards or dedicated NICs.
• Broadcom: Their enterprise-level NetXtreme series is well-known for its offloading capabilities.
• Chelsio: Their high-performance adapters are specifically designed with extensive offloading to reduce CPU overhead.

If you’re using an older or a cheaper consumer-grade network card, it’s less likely to have these advanced features, which means you might never experience this specific problem. So, while this fix worked for me, it’s not a universal solution for all slow jail downloads—it’s highly dependent on the hardware underneath.

Ultimately, by disabling the offloading options, I shifted the packet processing back to the CPU. While this might slightly increase CPU usage, it resolved the conflict between the physical and virtual interfaces, bringing my download speeds back to where they should be.

So, if you find yourself pulling your hair out over slow downloads in your FreeBSD jails, don’t overlook your hardware offloading. It’s a simple change that just might solve the problem you’ve been chasing.

– S