How to Build a High‑Performance P2P NAT‑Traversal Network with Headscale (Step‑by‑Step Guide)

1. Network topology overview

Domestic broadband often lacks a public IP, so external access to home devices relies on VPNs or remote‑control software that suffer from low speed. In a traditional star topology, two devices behind multiple NATs must route traffic through a central server, limiting the maximum throughput to the slowest link (e.g., 512 KB/s) and requiring paid bandwidth to improve performance.

Using NAT traversal creates a mesh topology: when device A initiates traffic to device B, the firewall temporarily opens a mapping that allows B to send traffic back directly, effectively “punching a hole” through the NAT.

2. Tailscale and Headscale

Tailscale is an open‑source VPN that leverages NAT traversal (P2P). Its client is open‑source, but the official control server is not. Headscale is an open‑source implementation of the control server, enabling users to run their own “self‑controlled” backend.

3. Deploying Headscale

3.1 Host installation (Ubuntu 22.04 example)

# Download Headscale binary
wget https://github.com/juanfont/headscale/releases/download/v0.16.4/headscale_0.16.4_linux_amd64 -O /usr/local/bin/headscale
chmod +x /usr/local/bin/headscale

# Create configuration directory and user
mkdir -p /etc/headscale
useradd \
  --create-home \
  --home-dir /var/lib/headscale/ \
  --system \
  --user-group \
  --shell /usr/sbin/nologin \
  headscale

# Systemd service file (/lib/systemd/system/headscale.service)
[Unit]
Description=headscale controller
After=syslog.target
After=network.target

[Service]
Type=simple
User=headscale
Group=headscale
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/headscale /var/run/headscale
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=headscale

[Install]
WantedBy=multi-user.target

3.2 Configuration (important fields only)

---
server_url: https://your.domain.com          # address clients must reach
listen_addr: 0.0.0.0:8080                  # actual listening port
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false
ip_prefixes:
  - fd7a:115c:a1e0::/48
  - 100.64.0.0/10

derp:
  server:
    enabled: false
  urls: []
  paths:
    - /etc/headscale/derper.yaml
  auto_update_enabled: true
  update_frequency: 24h

db_type: sqlite3
db_path: /var/lib/headscale/db.sqlite

tls_letsencrypt_hostname: ""

tls_cert_path: ""

tls_key_path: ""

randomize_client_port: false

3.3 TLS / reverse‑proxy

To use ACME certificates, set tls_letsencrypt_hostname to your domain and expose port 443.

For custom certificates, fill tls_cert_path and tls_key_path.

If you prefer a plain HTTP backend behind Nginx or Caddy, clear the TLS fields and proxy listen_addr (e.g., reverse_proxy headscale:8080).

3.4 Docker Compose deployment

version: "3.9"
services:
  headscale:
    container_name: headscale
    image: headscale/headscale:0.16.4
    ports:
      - "8080:8080"
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv6.conf.all.forwarding=1
    restart: always
    volumes:
      - ./conf:/etc/headscale
      - data:/var/lib/headscale
    command: ["headscale", "serve"]
volumes:
  config:
  data:

4. Client installation

4.1 Linux

curl -fsSL https://tailscale.com/install.sh | sh
tailscale up \
  --login-server https://your.domain.com \
  --advertise-routes=192.168.11.0/24 \
  --accept-routes=true \
  --accept-dns=false

--login-server

: URL of the Headscale control server (required). --advertise-routes: Tell Headscale which internal subnet this node can route. --accept-routes: Accept routes advertised by other nodes. --accept-dns: Whether to use DNS settings pushed by Headscale (usually disabled).

4.2 macOS

# Install Go
brew install go
# Build CLI
go install tailscale.com/cmd/tailscale{,d}@main
# Install system daemon
sudo tailscaled install-system-daemon
# Start with the same <code>tailscale up</code> flags as Linux.

4.3 Windows

Visit https://your.domain.com/windows to obtain a registration command, then install the official Tailscale GUI and run the command.

5. DERP relay (fallback server)

When NAT traversal fails, a DERP (Designated Encrypted Relay Point) server can act as a relay.

5.1 Build DERP

# Compile DERP server
go install tailscale.com/cmd/derper@main
mv ${GOPATH}/bin/derper /usr/local/bin/derper
# Create user
useradd \
  --create-home \
  --home-dir /var/lib/derper/ \
  --system \
  --user-group \
  --shell /usr/sbin/nologin \
  derper
# Systemd service (/lib/systemd/system/derper.service)
[Unit]
Description=tailscale derper server
After=syslog.target
After=network.target

[Service]
Type=simple
User=derper
Group=derper
ExecStart=/usr/local/bin/derper -c=/var/lib/derper/private.key -a=:8989 -stun-port=3456 -verify-clients
Restart=always
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/derper /var/run/derper
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=derper

[Install]
WantedBy=multi-user.target

By default DERP listens on port 443 and will request an ACME certificate unless -a specifies a non‑443 port or --certmode=manual is used.

5.2 Integrate DERP with Headscale

# derper.yaml (example)
regions:
  901:
    regionid: 901
    regioncode: private-derper
    regionname: "My Private Derper Server"
    nodes:
      - name: private-derper
        regionid: 901
        hostname: derper.example.com
        ipv4: 123.123.123.123
        stunport: 3456

# Add to Headscale config
derp:
  server:
    enabled: false
  urls: []
  paths:
    - /etc/headscale/derper.yaml
  auto_update_enabled: true
  update_frequency: 24h

6. Debugging commands

tailscale ping <IP>

– tests connectivity; initially uses DERP then switches to direct P2P. tailscale status – shows each peer, connection method, and traffic stats. tailscale netcheck – prints a detailed network environment report (UDP support, NAT type, DERP latency, captive portal detection, etc.).

7. Additional tips and known issues

When using proxy tools on macOS, prefer the CLI client and add process rules to force tailscale and tailscaled to use DIRECT routing.

CPU spikes can occur if the default route is overridden; the issue is fixed in the mian branch (see issue #5879).

Aliyun instances use the CGNAT 100.64.0.0/10 range, which conflicts with Tailscale’s internal address allocation; the workaround is to rebuild Tailscale after removing the conflicting DROP rules.

To enable routing through a node, start the client with --advertise-routes=192.168.1.0/24 and other nodes with --accept-routes=true, then enable the route on Headscale via headscale node route enable -a -i <node‑id>.