WireGuard Site-to-Site Kernel Mesh: High-Throughput Crypto Routing & MTU Optimization

WireGuard's in-kernel implementation offers a performant, secure, and streamlined solution for site-to-site VPNs, significantly outperforming traditional IPsec deployments in many scenarios. This guide details the construction of a high-throughput WireGuard kernel mesh, focusing on cryptokey routing, NAT traversal, MTU optimization, and advanced routing with iproute2 and BGP.

WireGuard Fundamentals: Cryptokey Routing

WireGuard operates on a principle of cryptokey routing. Unlike IPsec, which often relies on IP addresses for peer identification and policy enforcement, WireGuard uses public keys. Each peer is identified by its public key, and traffic encryption/decryption is intrinsically linked to this key. This simplifies configuration and enhances security by making the identity cryptographic rather than network-address-based.

A WireGuard interface, typically named wg0 or similar, acts as a virtual network interface. When a packet is destined for an IP address configured within a peer's AllowedIPs range, WireGuard automatically encrypts it using that peer's public key and sends it to the peer's endpoint. Conversely, incoming encrypted packets are decrypted, and if their source IP matches an AllowedIPs range for the sending peer, they are accepted.

Peer Configuration Example

Consider two sites, SiteA and SiteB, each with a WireGuard gateway.

SiteA Gateway (wg0)

• Public IP: 203.0.113.10
• Internal Network: 10.0.1.0/24
• WireGuard IP: 10.255.255.1/32
• Public Key: SITEA_PUBLIC_KEY

SiteB Gateway (wg0)

• Public IP: 198.51.100.20
• Internal Network: 10.0.2.0/24
• WireGuard IP: 10.255.255.2/32
• Public Key: SITEB_PUBLIC_KEY

The WireGuard configuration on SiteA's gateway would include a peer entry for SiteB:

# /etc/wireguard/wg0.conf on SiteA Gateway
[Interface]
PrivateKey = SITEA_PRIVATE_KEY
Address = 10.255.255.1/32
ListenPort = 51820 # Standard WireGuard port
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
# SiteB Gateway
PublicKey = SITEB_PUBLIC_KEY
Endpoint = 198.51.100.20:51820
AllowedIPs = 10.255.255.2/32, 10.0.2.0/24 # WireGuard IP of SiteB, and SiteB's internal network
PersistentKeepalive = 25 # Important for NAT traversal

And on SiteB's gateway, a peer entry for SiteA:

# /etc/wireguard/wg0.conf on SiteB Gateway
[Interface]
PrivateKey = SITEB_PRIVATE_KEY
Address = 10.255.255.2/32
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
# SiteA Gateway
PublicKey = SITEA_PUBLIC_KEY
Endpoint = 203.0.113.10:51820
AllowedIPs = 10.255.255.1/32, 10.0.1.0/24 # WireGuard IP of SiteA, and SiteA's internal network
PersistentKeepalive = 25

The PostUp/PostDown rules enable IP forwarding and NAT for the WireGuard interface, allowing internal networks to communicate through the tunnel. Note that eth0 should be replaced with the actual public-facing interface name.

NAT Traversal with Persistent Keepalive

When a WireGuard peer is behind a NAT device (e.g., a home router or cloud NAT gateway), the NAT mapping can expire if no traffic flows for an extended period. This leads to connection drops until traffic is initiated from the NAT'd side. PersistentKeepalive addresses this by sending a small, encrypted keepalive packet to the peer every N seconds. This keeps the NAT mapping alive, ensuring bidirectional connectivity. A value of 25 seconds is typically sufficient for most NAT devices.

MTU Optimization and MSS Clamping

WireGuard encapsulates IP packets, adding its own header (typically 20 bytes for IPv4, 40 bytes for IPv6) plus UDP headers (8 bytes) and the outer IP header (20 bytes for IPv4, 40 bytes for IPv6). This overhead reduces the effective Maximum Transmission Unit (MTU) available for the encapsulated traffic.

Standard Ethernet MTU is 1500 bytes.

• Outer IP header: 20 bytes
• UDP header: 8 bytes
• WireGuard header: 20 bytes (approx)
• Total overhead: ~48 bytes

Therefore, the effective MTU for the inner packet becomes 1500 - 48 = 1452 bytes. Setting the WireGuard interface MTU to 1452 (or slightly lower, e.g., 1420 for safety or if IPv6 is involved) prevents fragmentation of the outer WireGuard packets.

# /etc/wireguard/wg0.conf (excerpt)
[Interface]
# ...
MTU = 1420 # Or 1452 if only IPv4 and no other overheads
# ...

However, simply setting the WireGuard interface MTU is not enough. Applications on the internal networks might still send packets larger than 1420 bytes, expecting the underlying network to handle fragmentation. This leads to "Path MTU Discovery (PMTUD) black holes" if ICMP Fragmentation Needed messages are blocked.

To mitigate this, we employ MSS Clamping. Maximum Segment Size (MSS) is the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. MSS clamping modifies the TCP SYN packets to advertise an MSS value that is less than or equal to the WireGuard tunnel's effective MTU minus the TCP/IP header size.

For an MTU of 1420:

• IP header: 20 bytes
• TCP header: 20 bytes
• Effective MSS: 1420 - 20 - 20 = 1380 bytes

This ensures that TCP sessions initiated across the tunnel will negotiate an MSS that prevents fragmentation within the tunnel.

# Add to PostUp section of wg0.conf on both gateways
# For IPv4
iptables -A FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380
# For IPv6 (if applicable)
ip6tables -A FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380

The complete PostUp/PostDown for wg0.conf would look like this:

# /etc/wireguard/wg0.conf (excerpt)
[Interface]
# ...
PostUp = \
  iptables -A FORWARD -i %i -j ACCEPT; \
  iptables -A FORWARD -o %i -j ACCEPT; \
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; \
  iptables -A FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380; \
  ip6tables -A FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380
PostDown = \
  iptables -D FORWARD -i %i -j ACCEPT; \
  iptables -D FORWARD -o %i -j ACCEPT; \
  iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; \
  iptables -D FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380; \
  ip6tables -D FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1380
# ...

Replace eth0 with your actual public-facing interface.

Advanced Routing: iproute2 and BGP with FRRouting

For simple site-to-site setups, static routes defined in AllowedIPs are sufficient. However, in a mesh network with multiple sites and dynamic network topologies, or when integrating with existing routing infrastructure, dynamic routing protocols like BGP are essential.

iproute2 for Static Routes

For scenarios where BGP is overkill, iproute2 can be used to manage routes. The AllowedIPs in WireGuard automatically create routes for the specified subnets via the WireGuard interface. However, if you need to route traffic from the WireGuard interface to specific internal subnets that are not directly connected to the gateway, you'd use ip route.

Example: SiteA gateway needs to reach 10.0.3.0/24 via an internal router 10.0.1.254.

# On SiteA Gateway
ip route add 10.0.3.0/24 via 10.0.1.254 dev eth1 # Assuming eth1 is internal interface

This is outside the WireGuard configuration itself but crucial for end-to-end connectivity.

Dynamic Routing with BGP and FRRouting

FRRouting (FRR) is a suite of IP routing protocols for Linux and Unix platforms, including BGP, OSPF, RIP, IS-IS, and PIM. We'll use BGP to dynamically exchange routes between WireGuard gateways.

Architecture:

1. Each WireGuard gateway runs FRR.
2. BGP sessions are established over the WireGuard tunnel IPs (e.g., 10.255.255.1 and 10.255.255.2).
3. Each gateway advertises its local internal networks into BGP.

Installation (Debian/Ubuntu):

sudo apt update
sudo apt install frr frr-pythontools

FRR Configuration (/etc/frr/frr.conf):

SiteA Gateway (10.255.255.1):

!
hostname SiteA-WG-Gateway
password zebra
enable password zebra
!
router bgp 64512 # SiteA's ASN
  bgp router-id 10.255.255.1
  neighbor 10.255.255.2 remote-as 64513 # SiteB's ASN
  neighbor 10.255.255.2 update-source wg0 # Use WireGuard interface for BGP session
  !
  address-family ipv4 unicast
    network 10.0.1.0/24 # Advertise SiteA's internal network
    neighbor 10.255.255.2 activate
  exit-address-family
!

SiteB Gateway (10.255.255.2):

!
hostname SiteB-WG-Gateway
password zebra
enable password zebra
!
router bgp 64513 # SiteB's ASN
  bgp router-id 10.255.255.2
  neighbor 10.255.255.1 remote-as 64512 # SiteA's ASN
  neighbor 10.255.255.1 update-source wg0 # Use WireGuard interface for BGP session
  !
  address-family ipv4 unicast
    network 10.0.2.0/24 # Advertise SiteB's internal network
    neighbor 10.255.255.1 activate
  exit-address-family
!

After configuring FRR, restart the service: sudo systemctl restart frr. Verify BGP status using vtysh:

sudo vtysh
show ip bgp summary
show ip route

The AllowedIPs in WireGuard must be configured to permit the BGP session traffic (i.e., the WireGuard tunnel IPs themselves) and the networks advertised by BGP. If AllowedIPs only contains the tunnel IP, BGP will establish, but routes for the internal networks won't be installed by WireGuard.

A common pattern for BGP over WireGuard is to set AllowedIPs to just the peer's WireGuard IP (e.g., 10.255.255.2/32). This ensures that only BGP traffic (and other control plane traffic) uses the WireGuard tunnel initially. Then, BGP advertises the actual internal networks (e.g., 10.0.2.0/24). When FRR receives these routes, it installs them into the kernel's routing table, pointing to the WireGuard interface.

Crucial Note on AllowedIPs and BGP: If you use BGP, you generally want AllowedIPs to be minimal, typically just the peer's WireGuard tunnel IP. This allows BGP to establish. BGP then advertises the actual internal networks. WireGuard's AllowedIPs acts as a routing policy: it dictates which IP ranges WireGuard will decrypt and send traffic for. If AllowedIPs includes 10.0.2.0/24, WireGuard will automatically create a route for it. If BGP also tries to install a route for 10.0.2.0/24 via wg0, it might conflict or be redundant.

The best practice for BGP over WireGuard is to:

1. Set AllowedIPs to only the peer's WireGuard IP (e.g., 10.255.255.2/32).
2. Ensure net.ipv4.ip_forward=1 is enabled.
3. Let BGP handle the advertisement and installation of routes for the internal networks. WireGuard will then decrypt traffic for these networks because they are routed via the wg0 interface, even if not explicitly in AllowedIPs.

This setup allows for dynamic route updates without reconfiguring WireGuard.

Benchmarking: WireGuard vs. IPsec

WireGuard's design, particularly its kernel-space implementation and simpler cryptographic handshake, often results in significantly higher throughput and lower latency compared to IPsec.

Throughput Comparison (Conceptual 10Gbps Link):

On a 10Gbps link with modern CPUs, WireGuard can often achieve 8-9 Gbps of encrypted throughput. IPsec, depending on the cipher suite and CPU, might struggle to exceed 3-5 Gbps without specialized hardware acceleration. The kernel-space implementation of WireGuard minimizes context switching overhead, which is a major performance bottleneck for user-space VPN solutions.

Production Gotchas & Troubleshooting

1. AllowedIPs Misconfiguration:

   • Symptom: BGP sessions don't establish, or internal networks are unreachable.
   • Issue: AllowedIPs on a peer must include the peer's WireGuard IP for the BGP session to form. If you're using BGP for internal networks, AllowedIPs should not include those internal networks on the peer, only the tunnel IP. Otherwise, WireGuard will install static routes that might conflict with BGP-learned routes or prevent BGP from installing its own.
   • Fix: For BGP, set AllowedIPs = <PEER_WG_IP>/32. Ensure net.ipv4.ip_forward=1 is enabled.
   • Verification: wg show wg0 dump to see current peer configurations and ip route show table main to check routing table entries.

2. MTU/MSS Clamping Issues (PMTUD Black Holes):

   • Symptom: Large file transfers stall, SSH connections hang after authentication, certain websites don't load. ping -M do -s 1472 <remote_host> fails.
   • Issue: Packets are being fragmented, and ICMP Fragmentation Needed messages are blocked, or the MSS is too large.
   • Fix: Ensure MTU is correctly set on the WireGuard interface (e.g., 1420). Implement TCPMSS clamping on the PostUp script for both IPv4 and IPv6. Verify iptables rules are active.
   • Verification: iptables -t mangle -nvL FORWARD. Use tcpdump -i wg0 -n -s0 port 51820 to inspect packet sizes.

3. NAT Traversal Failures:

   • Symptom: Connectivity drops after inactivity, only restores when traffic is initiated from the NAT'd side.
   • Issue: NAT device's mapping expires.
   • Fix: Set PersistentKeepalive = 25 (or similar) in the [Peer] section for peers behind NAT.
   • Verification: wg show wg0 latest-handshakes. If latest-handshakes is updating regularly, keepalive is working.

4. Firewall Rules:

   • Symptom: No connectivity, even if WireGuard peers show handshake.
   • Issue: Firewall (e.g., ufw, firewalld, iptables) blocking UDP port 51820 or blocking forwarded traffic.
   • Fix: Allow UDP traffic on the WireGuard listen port. Ensure FORWARD chain rules permit traffic between the WireGuard interface and internal/external interfaces.
   • Verification: sudo iptables -nvL or sudo ufw status verbose.

5. IP Forwarding Disabled:

   • Symptom: WireGuard tunnel establishes, but hosts on internal networks cannot reach each other.
   • Issue: Kernel IP forwarding is disabled.
   • Fix: Enable net.ipv4.ip_forward = 1 and net.ipv6.conf.all.forwarding = 1 (if using IPv6) in /etc/sysctl.conf and apply with sudo sysctl -p.
   • Verification: sysctl net.ipv4.ip_forward.

Frequently Asked Questions

1. Why is WireGuard faster than IPsec? WireGuard's performance advantage stems from several factors: its minimalist design, modern cryptographic primitives (ChaCha20/Poly1305) optimized for speed, and its native kernel-space implementation. This minimizes context switching between user and kernel space, reduces cryptographic overhead, and simplifies the state machine, leading to higher throughput and lower latency.

2. Can I run WireGuard on a non-Linux system? Yes. WireGuard has official clients for macOS, Windows, Android, and iOS. While the kernel module is Linux-specific, these clients use user-space implementations or OS-specific network extensions that provide similar functionality, though potentially with different performance characteristics than the native Linux kernel module.

3. How do I scale a WireGuard mesh to many sites? For a small number of sites (e.g., under 10), a full mesh where each gateway peers with every other gateway is manageable. For larger deployments, consider a hub-and-spoke model with central WireGuard gateways, or leverage dynamic routing protocols like BGP over the WireGuard tunnels. Each gateway would peer with a central BGP router (or other gateways) over its WireGuard interface, advertising its local subnets. This avoids N^2 peer configurations.

4. What if my public IP changes (dynamic DNS)? If a peer's Endpoint is a dynamic IP, WireGuard can resolve a DNS hostname. If the IP changes, WireGuard will eventually re-resolve the DNS name. However, the PersistentKeepalive from the other side is crucial. If the peer with the dynamic IP is behind NAT, its PersistentKeepalive will keep its NAT mapping alive and inform the other side of its current public IP. If the peer with the dynamic IP is not behind NAT, but its IP changes, the other peer will eventually update its endpoint after a new handshake is initiated. For robust dynamic endpoint updates, consider using a custom script that updates the Endpoint via wg set when the IP changes, or rely on PersistentKeepalive from the other side.

5. Is it secure to expose WireGuard's UDP port to the internet? Yes, WireGuard is designed to be secure even when its UDP port is exposed. Its protocol is resistant to common attacks like port scanning and denial-of-service. It only responds to correctly authenticated packets, and unauthenticated packets are silently dropped. This "silent discard" behavior makes it difficult for attackers to even determine if a WireGuard endpoint is active.