Master SSH: From Basic Connections to Secure, High‑Performance Remote Workflows

1. Overview of SSH

SSH (Secure Shell) is a network protocol that provides encrypted remote login and other services, replacing insecure Telnet and protecting against eavesdropping, tampering, and impersonation.

2. Basic Usage

Connecting

# Standard connection
ssh username@server_ip

# Connect to a non‑standard port (e.g., 2222)
ssh -p 2222 username@server_ip

Host‑key verification – on first connection confirm the fingerprint by typing yes to prevent man‑in‑the‑middle attacks.

Running remote commands

# Single command
ssh username@server_ip "nvidia-smi"

# Stream a log file
ssh username@server_ip "tail -f /project/train.log"

# Multiple commands in one line
ssh username@server_ip "cd /project && ls -l && pwd"

3. Password‑less (key‑based) Authentication

Generate a key pair, copy the public key to the server, and verify that login no longer prompts for a password.

# 1. Generate a 4096‑bit RSA key
ssh-keygen -t rsa -b 4096 -C "[email protected]"

# 2. Deploy the public key
ssh-copy-id username@server_ip

# 3. Test password‑less login
ssh username@server_ip   # should open a shell without a password prompt

Security tip : Ensure ~/.ssh directory is 700 and authorized_keys is 600.

4. SSH Config for Aliases and Common Options

# ~/.ssh/config example
Host ai-server
    HostName 10.10.100.208
    User xlsys
    IdentityFile ~/.ssh/id_rsa
    ServerAliveInterval 60
    LocalForward 8888 localhost:8888
    LocalForward 6006 localhost:6006

After adding the entry, connect with ssh ai-server and the defined tunnels are created automatically.

5. Port Forwarding for Deep‑Learning Workflows

# Forward TensorBoard
ssh -L 6006:localhost:6006 username@server_ip

# Forward multiple services (Jupyter, TensorBoard, etc.)
ssh -L 8888:localhost:8888 -L 6006:localhost:6006 -L 8080:localhost:8080 username@server_ip

6. Large‑File Transfer

Use scp with bandwidth limit or rsync for resumable transfers.

# Limit bandwidth to 5 Mbps
scp -l 5000 dataset.zip username@server_ip:/data/

# Resume with rsync
rsync -Pavz --partial data/ username@server_ip:/backup/

7. Server‑Side Hardening (requires sudo)

# Disable password authentication
sudo sed -i 's/#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config

# Disallow root login
sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config

# Change the default port (example: 22222)
echo "Port 22222" | sudo tee -a /etc/ssh/sshd_config

# Restart SSH service
sudo systemctl restart sshd

Verify the new port is listening:

sudo ss -tulpn | grep ':22222'

8. Running Deep‑Learning Jobs in the Background

# nohup example
ssh username@server_ip "nohup python train.py > /logs/train_$(date +%F).log 2>&1 &"

# screen example
ssh username@server_ip
screen -S training
python train.py   # detach with Ctrl‑A D

# tmux (recommended)
ssh username@server_ip "tmux new-session -d -s training 'python train.py'"

9. Common Troubleshooting

Host key changed : ssh-keygen -R <host> Immediate disconnect : Check permissions of /var/log/auth.log and SSH directories.

Public‑key authentication fails : Ensure ~/.ssh and authorized_keys have correct SELinux context, e.g., restorecon -Rv ~/.ssh.

Large file transfer stalls : Prefer rsync -P over scp for resumable transfers.