How to Build a Self‑Hosted P2P VPN with Headscale and Tailscale

1. NAT Traversal Overview

Because many domestic broadband connections lack a public IP, accessing a home network from outside usually requires VPNs or remote tools, which are often slow and unstable.

1.1 Traditional Star Topology

When two devices are behind multiple NAT layers, they must communicate through a central relay server (VPN or remote software). The connection speed is limited by the slowest link in the chain, typically the central server's bandwidth.

Maximum transfer speed between a work laptop and a home NAS is Up/Down: 512K/s because all traffic passes through the central server.

1.2 NAT Traversal and Mesh Topology

This section is a brief overview; for detailed rules see "How NAT traversal works".

After device A initiates traffic to device B, the firewall temporarily creates a mapping that allows B to send traffic back to A. In simple terms, it "hits you back through the same wire".

By understanding this rule, we can deploy a low‑cost central server that only helps negotiate which device contacts which. Once both devices initiate traffic, the firewall opens the NAT traversal rule and the devices can communicate directly, forming a mesh topology where speed depends on each device's own bandwidth.

Note that NAT traversal may fail in complex networks due to strict firewall or protocol restrictions, so a fallback central server is advisable.

2. Introduction to Tailscale

The first part explains basic NAT‑traversal concepts; now we focus on Tailscale.

Tailscale is a VPN tool that uses NAT traversal (P2P) technology. The client is open‑source, but the control server is not. Tailscale offers a free tier that can run at full speed when traversal succeeds.

If traversal fails, the official Tailscale server is slow in China, but the open‑source community provides Headscale , a self‑hosted control server, enabling full autonomy.

3. Setting Up Headscale Server

The following commands assume Ubuntu 22.04; adjust for other systems.

3.1 Host Installation

Headscale is written in Go and distributed as a single binary. Download the latest release:

# Download
wget https://github.com/juanfont/headscale/releases/download/v0.16.4/headscale_0.16.4_linux_amd64 -O /usr/local/bin/headscale

# Make executable
chmod +x /usr/local/bin/headscale

Create a dedicated user and directory for Headscale:

# Create config directory
mkdir -p /etc/headscale

# Create system user
useradd \
  --create-home \
  --home-dir /var/lib/headscale/ \
  --system \
  --user-group \
  --shell /usr/sbin/nologin \
  headscale

Create a Systemd service file:

[Unit]
Description=headscale controller
After=syslog.target
After=network.target

[Service]
Type=simple
User=headscale
Group=headscale
ExecStart=/usr/local/bin/headscale serve
Restart=always
RestartSec=5

# Optional security enhancements
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/headscale /var/run/headscale
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=headscale

[Install]
WantedBy=multi-user.target

3.2 Headscale Configuration

Edit /etc/headscale/config.yaml with the essential settings:

---
# Headscale server address (what clients will contact)
server_url: https://your.domain.com

# Listening address
listen_addr: 0.0.0.0:8080

metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false

# Private IP ranges allocated to clients
ip_prefixes:
  - fd7a:115c:a1e0::/48
  - 100.64.0.0/10

# DERP (relay) configuration
derp:
  server:
    enabled: false
  urls: []
  paths:
    - /etc/headscale/derper.yaml
  auto_update_enabled: true
  update_frequency: 24h

# SQLite backend
db_type: sqlite3
db_path: /var/lib/headscale/db.sqlite

tls_letsencrypt_hostname: ""
tls_cert_path: ""
tls_key_path: ""

randomize_client_port: false

3.3 Certificates and Reverse Proxy

Headscale supports three modes:

If tls_letsencrypt_hostname is set, ACME will request a certificate for the given hostname (requires ports 80/443).

If tls_cert_path is set (and tls_letsencrypt_hostname empty), Headscale uses the provided certificate.

If neither is set, Headscale serves plain HTTP only.

When using Nginx or Caddy as a reverse proxy, clear tls_letsencrypt_hostname and tls_cert_path, then set server_url to the external HTTPS address and proxy the listen_addr HTTP endpoint.

Example Nginx snippet: reverse_proxy headscale:8080.

3.4 Docker‑Compose Installation

Compose file example:

# docker-compose.yaml
version: "3.9"
services:
  headscale:
    container_name: headscale
    image: headscale/headscale:0.16.4
    ports:
      - "8080:8080"
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv6.conf.all.forwarding=1
    restart: always
    volumes:
      - ./conf:/etc/headscale
      - data:/var/lib/headscale
    command: ["headscale", "serve"]
volumes:
  config:
  data:

Place a conf directory beside the compose file containing config.yaml and any DERP yaml.

4. Client Installation

4.1 Linux Client

curl -fsSL https://tailscale.com/install.sh | sh

After installation, start the client with:

tailscale up --login-server https://your.domain.com \
  --advertise-routes=192.168.11.0/24 \
  --accept-routes=true \
  --accept-dns=false

The --login-server flag points to your Headscale instance; --advertise-routes reports the local subnet; --accept-routes enables receiving routes from other nodes.

4.2 macOS Client

Two options:

Install the App Store version, set the server address, and follow the browser‑based registration flow.

Compile the client from source for a command‑line experience:

# Install Go
brew install go

# Build client binaries
go install tailscale.com/cmd/tailscale{,d}@main

# Install as a system daemon
sudo tailscaled install-system-daemon

Then run tailscale up with the same flags as on Linux.

4.3 Other Platforms

Windows requires creating a registry entry and installing the official app; after launching, copy the registration command from the browser to the Headscale server. Mobile clients are not covered here.

5. Relay (DERP) Server Setup

5.1 Building a DERP Server

Install a Tailscale client on the relay host, then compile the DERP binary:

# Compile DERP server
go install tailscale.com/cmd/derper@main

# Move binary to /usr/local/bin
mv ${GOPATH}/bin/derper /usr/local/bin

# Create system user
useradd \
  --create-home \
  --home-dir /var/lib/derper/ \
  --system \
  --user-group \
  --shell /usr/sbin/nologin \
  derper

Create a Systemd service:

# /lib/systemd/system/derper.service
[Unit]
Description=tailscale derper server
After=syslog.target
After=network.target

[Service]
Type=simple
User=derper
Group=derper
ExecStart=/usr/local/bin/derper -c=/var/lib/derper/private.key -a=:8989 -stun-port=3456 -verify-clients
Restart=always
RestartSec=5

# Optional security enhancements
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/derper /var/run/derper
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=derper

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl enable derper --now

5.2 Configuring Headscale to Use the DERP Server

Create /etc/headscale/derper.yaml with your relay information:

regions:
  901:
    regionid: 901
    regioncode: private-derper
    regionname: "My Private Derper Server"
    nodes:
      - name: private-derper
        regionid: 901
        hostname: derper.example.com
        ipv4: 123.123.123.123
        stunport: 3456

Reference this file in the main Headscale config:

derp:
  server:
    enabled: false
  urls: []
  paths:
    - /etc/headscale/derper.yaml
  auto_update_enabled: true
  update_frequency: 24h

6. Client Network Debugging

6.1 Ping

Use tailscale ping <IP> to test connectivity. The command first tries the DERP relay, then falls back to direct P2P when possible.

6.2 Status

Run tailscale status to see which peers are reachable and whether the connection is direct or via DERP.

6.3 NetCheck

tailscale netcheck

provides a detailed report of the local network environment, including UDP support, NAT mapping, port‑mapping methods, captive portal detection, and nearest DERP latency.

7. Additional Tips

7.1 Compatibility with Proxy Tools

When using enhanced proxy software on macOS, prefer the command‑line client and add tailscale and tailscaled to a DIRECT rule.

7.2 High CPU Usage on macOS

Some network tools can prevent tailscaled from detecting the default route, causing a CPU‑intensive loop. Use the mian branch to fix this issue.

7.3 Alibaba Cloud CGNAT Conflict

Tailscale allocates the CGNAT range 100.64.0.0/10 for internal addresses, which collides with Alibaba Cloud DNS and apt sources. The workaround is to modify the source code to remove the conflicting DROP rules and recompile.

7.4 Enabling Route Forwarding

Install Tailscale on a few nodes, advertise routes with --advertise-routes=192.168.1.0/24, and enable acceptance on other nodes with --accept-routes=true. Then enable the route on Headscale using headscale node route enable -a -i <ID>.

7.5 Other Issues

For any remaining problems, search existing GitHub issues or read the source code; many Tailscale options are undocumented and require source inspection.