How to Build a Self‑Hosted P2P Network with Headscale and Tailscale

Overview of Internal Network Penetration

Due to most home broadband lacking a public IP, traditional star‑topology VPNs or remote‑desktop tools suffer from low speed because traffic must pass through a central server.

NAT Traversal and Mesh Topology

By using NAT traversal, a low‑capacity central server only helps negotiate the connection; after the firewall opens a temporary mapping, the two peers communicate directly, achieving speeds limited only by their own bandwidth.

Headscale Introduction

Headscale is an open‑source control server compatible with Tailscale that enables self‑hosted NAT‑traversal VPNs. The official Tailscale server is not open source and may be slow in China, so Headscale provides a “self‑controlled” alternative.

Installing Headscale Server

Assuming Ubuntu 22.04, download the binary, make it executable, create a dedicated user and configuration directory, and add a Systemd service file.

# download
wget https://github.com/juanfont/headscale/releases/download/v0.16.4/headscale_0.16.4_linux_amd64 -O /usr/local/bin/headscale
chmod +x /usr/local/bin/headscale

# create user and dirs
mkdir -p /etc/headscale
useradd --create-home --home-dir /var/lib/headscale --system --user-group --shell /usr/sbin/nologin headscale

# systemd unit
[Unit]
Description=headscale controller
After=network.target

[Service]
Type=simple
User=headscale
Group=headscale
ExecStart=/usr/local/bin/headscale serve
Restart=always
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/headscale /var/run/headscale
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=headscale

[Install]
WantedBy=multi-user.target

Headscale Configuration

Edit /etc/headscale/config.yaml to set server_url, listen_addr, IP prefixes, DERP settings, database path, and TLS options. Example snippet:

---
server_url: https://your.domain.com
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false
ip_prefixes:
  - fd7a:115c:a1e0::/48
  - 100.64.0.0/10
derp:
  server:
    enabled: false
  urls: []
  paths: []
  auto_update_enabled: true
  update_frequency: 24h
db_type: sqlite3
db_path: /var/lib/headscale/db.sqlite
tls_letsencrypt_hostname: ""
tls_cert_path: ""
tls_key_path: ""
randomize_client_port: false

Certificate and Reverse‑Proxy Setup

To avoid ACME on the server, clear tls_letsencrypt_hostname and tls_cert_path, then configure Nginx or Caddy to proxy the HTTP address defined by listen_addr. Example Nginx snippet and Caddy line:

# Nginx (simplified)
server {
    listen 80;
    server_name your.domain.com;
    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}

reverse_proxy headscale:8080

Internal Address Allocation

Avoid the default CGNAT prefix 100.64.0.0/10 because many cloud services (e.g., Alibaba Cloud apt sources) also use it, which can cause conflicts.

Starting Headscale

Enable and start the service with:

systemctl enable headscale --now

For automatic ACME certificates, set tls_letsencrypt_hostname to your domain and listen on port 443.

Client Installation

Linux:

curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --login-server https://your.domain.com \
    --advertise-routes=192.168.11.0/24 \
    --accept-routes=true \
    --accept-dns=false

macOS: install from the App Store or compile from source using Go. After installing, run the same tailscale up command. Windows: follow the URL shown by the Headscale server (e.g., https://your.domain.com/windows) to download the official client and register with the same tailscale up options.

DERP (Relay) Server

Compile the DERP server:

# compile DERP
go install tailscale.com/cmd/derper@main
mv $(go env GOPATH)/bin/derper /usr/local/bin/derper
# create user
useradd --create-home --home-dir /var/lib/derper --system --user-group --shell /usr/sbin/nologin derper

Add a Systemd unit (simplified):

[Unit]
Description=tailscale derper server
After=network.target

[Service]
Type=simple
User=derper
Group=derper
ExecStart=/usr/local/bin/derper -c=/var/lib/derper/private.key -a=:8989 -stun-port=3456 -verify-clients
Restart=always
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/derper /var/run/derper
AmbientCapabilities=CAP_NET_BIND_SERVICE
RuntimeDirectory=derper

[Install]
WantedBy=multi-user.target

Configure Headscale to use the custom DERP by disabling the built‑in server and adding the path to derper.yaml in the derp section.

Docker Compose Deployments

Headscale example:

version: "3.9"
services:
  headscale:
    image: headscale/headscale:0.16.4
    container_name: headscale
    ports:
      - "8080:8080"
    cap_add:
      - NET_ADMIN
      - NET_RAW
      - SYS_MODULE
    sysctls:
      - net.ipv4.ip_forward=1
      - net.ipv6.conf.all.forwarding=1
    restart: always
    volumes:
      - ./conf:/etc/headscale
      - data:/var/lib/headscale
    command: ["headscale", "serve"]
volumes:
  data:

DERP example:

version: "3.9"
services:
  derper:
    image: mritd/derper
    container_name: derper
    restart: always
    ports:
      - "8080:8080/tcp"
      - "3456:3456/udp"
    environment:
      TZ: Asia/Shanghai
    volumes:
      - /etc/timezone:/etc/timezone
      - /var/run/tailscale:/var/run/tailscale
      - data:/var/lib/derper
volumes:
  data:

Client Network Debugging

Use the built‑in Tailscale commands: tailscale ping <IP> – tests connectivity, initially via DERP then direct P2P. tailscale status – shows peers and how each connection is established. tailscale netcheck – reports UDP support, NAT mapping, STUN results, and nearest DERP latency.

Additional Tips

When using proxy tools on macOS, prefer the CLI client and whitelist tailscale and tailscaled processes to keep them on DIRECT rules.

Recent Tailscale builds fix high CPU usage caused by missing default routes.

On Alibaba Cloud, the default CGNAT prefix conflicts with internal services; modify the source to remove the DROP rules if needed.

To enable routing, advertise routes on a node with --advertise-routes=192.168.1.0/24, accept routes on other nodes with --accept-routes=true, then enable the route on Headscale using headscale node route enable -a -i <ID>.