Skip to content
SSH Login Latency Troubleshooting: Diagnose Every Connection Stage
Operations 25 min read

SSH Login Latency Troubleshooting: Diagnose Every Connection Stage

This guide provides a systematic approach to diagnose SSH login slowness by breaking down the connection chain into DNS resolution, TCP handshake, SSH negotiation, authentication, PAM, and shell initialization, with concrete commands and evidence-based fixes for each stage.

Raymond Ops
Raymond Ops
Raymond Ops

SSH login slowness is not a single-point failure but a wait along a request chain: name resolution, TCP connection, SSH negotiation, authentication, PAM session, and shell initialization can all block. Troubleshooting must first identify which segment the wait occurs in, to avoid breaking domain authentication, auditing, or automated login.

Phenomenon: Quantify "Slow"

Record the problem client, source IP, target host, account, jump host usage, authentication method, and start/end times. Prepare at least one fast sample and one slow sample; a fast sample can be the same client accessing another server or another client accessing the same server.

Run a minimal test on the client:

ssh -vvv -o ConnectTimeout=10 [email protected] true

Replace ops with the test account and ssh.example.internal with the target hostname or IP. The trailing true exits immediately after authentication, separating shell initialization effects from authentication effects. The command only establishes a normal SSH session without modifying remote configuration.

Example debug output:

debug1: Connecting to ssh.example.internal [10.20.30.40] port 22.
debug1: Connection established.
debug1: Authenticating to ssh.example.internal:22 as 'ops'
debug1: Offering public key: /home/ops/.ssh/id_ed25519 ED25519 SHA256:...
debug1: Server accepts key: /home/ops/.ssh/id_ed25519 ED25519 SHA256:...
Authenticated to ssh.example.internal ([10.20.30.40]:22) using "publickey".
debug1: Sending command: true

If the output pauses before Connecting to, investigate DNS, routing, and TCP. If it pauses near Authenticating to, focus on GSSAPI, reverse DNS, PAM, and directory services. If Authenticated to appears but the command still waits, examine session scripts, home directory, mounts, and motd. This segmentation is more valuable than a single "login took 8 seconds" metric.

Information Gathering: Confirm Actual Client Behavior

Client configuration often overrides system defaults. First check resolution results, actual routing, and the expanded SSH configuration:

date -Is
getent ahosts ssh.example.internal
ssh -G ssh.example.internal | sed -n '1,140p'
ip route get 10.20.30.40

Pay special attention to proxyjump, proxycommand, addressfamily, gssapiauthentication, preferredauthentications, and connecttimeout. If a jump host is used, run ssh -vvv separately against the jump host and the final target; the actual connection involves two network segments and two authentications, so testing only the final alias is insufficient.

On the server, perform read-only checks:

sudo sshd -T | sort
sudo ss -ltnp '( sport = :22 )'
sudo journalctl -u sshd --since '20 minutes ago' --no-pager
sudo journalctl -u ssh --since '20 minutes ago' --no-pager

Service names differ by distribution ( sshd or ssh). RHEL-family authentication logs are typically in /var/log/secure, Debian/Ubuntu in /var/log/auth.log, but prefer journalctl to avoid assuming log paths. For configurations with Match conditions, check effective values for the problematic user and source address:

sudo sshd -T -C user=ops,addr=10.20.30.50,host=ssh.example.internal | sort

Replace 10.20.30.50 with the real client source IP. This does not reload the service but can reveal that the user matched a rule different from the main configuration.

Preliminary Judgment: Use Comparisons to Narrow Scope

First compare domain name versus IP:

time ssh -o ConnectTimeout=10 [email protected] true
time ssh -o ConnectTimeout=10 [email protected] true

If IP is fast and domain is slow, investigate DNS; if both are slow, do not assume DNS is the default root cause. Next test by address family:

time ssh -4 -o ConnectTimeout=10 [email protected] true
time ssh -6 -o ConnectTimeout=10 [email protected] true

If only IPv6 is slow, fix IPv6 routing, ACLs, AAAA records, or the corresponding egress policy. Globally forcing IPv4 affects pure IPv6 environments and masks underlying network issues.

Classify findings into four stages:

Pre-resolution : domain slow, IP fast → DNS, search domains, IPv6

Connection : SYN retransmits, no Connection established → routing, ACL, listener, packet loss

Authentication : pause at Authenticating → GSSAPI, reverse DNS, LDAP/AD

Session : authenticated but command does not return → PAM, NFS, profile, motd

Root cause conclusions must be supported by logs, packet captures, NSS queries, or command timings, not merely by "it worked after the change."

Command Checks: DNS and Reverse Lookup

The server may perform a reverse lookup of the client IP during authentication. OpenSSH's UseDNS controls whether sshd uses that result, but the default varies by version and distribution policy; verify the actual output:

sudo sshd -T | grep '^usedns '
getent hosts 10.20.30.50
grep -E '^(hosts|passwd|group|shadow):' /etc/nsswitch.conf

Example nsswitch.conf output:

hosts: files dns myhostname
passwd: files sss
group:  files sss

If getent hosts for the problem source IP repeatedly waits several seconds and that wait aligns with SSH authentication logs, reverse DNS is strong evidence. Also check DNS server reachability, reverse zone records, and nsswitch.conf. Do not remove sss, ldap, or winbind because of a single timeout; doing so may break domain user/group resolution and sudo authorization entirely.

After confirming that sshd's reverse lookup causes the wait, implement a minimal change via configuration management. In environments supporting Include, use a drop-in fragment:

# /etc/ssh/sshd_config.d/90-login-latency.conf
UseDNS no

Execution steps:

sudo install -D -m 0600 /etc/ssh/sshd_config /var/backups/sshd_config.$(date +%F-%H%M%S)
sudo sshd -t
sudo systemctl reload sshd
sudo sshd -T | grep '^usedns '

The change affects only new SSH sessions on that host; existing sessions are typically unaffected by reload. Keep a second sudo-enabled SSH session or an out-of-band console available before executing. Backup by saving the original config; canary on a non-critical node; verify by repeating the minimal test from the problematic segment; rollback by restoring the backup, running sshd -t, then reload. If configuration is managed by Ansible, Puppet, Salt, or an image baseline, the change must be made in the declarative source, otherwise it will be overwritten on the next deployment.

Command Checks: TCP and Intermediate Network

When the pause occurs during TCP connection establishment, check the listener and policy, but do not treat "port is listening" as proof that the path is clear:

sudo ss -ltnp '( sport = :22 )'
sudo nft list ruleset
sudo firewall-cmd --state

Availability of nft and firewall-cmd depends on the system; cloud security groups, hardware firewalls, and host-level rules may also intercept traffic. If necessary, capture packets on the server for a short period targeting a test client:

sudo timeout 30 tcpdump -ni any 'host 10.20.30.50 and tcp port 22' -vv

Example output showing a complete three-way handshake:

IP 10.20.30.50.52134 > 10.20.30.40.22: Flags [S], seq 123456789, win 64240
IP 10.20.30.40.22 > 10.20.30.50.52134: Flags [S.], seq 987654321, ack 123456790
IP 10.20.30.50.52134 > 10.20.30.40.22: Flags [.], ack 1, win 502

If the server repeatedly receives SYN but never sees the final ACK, the return path, client-side policy, or a middle device needs further verification. If the handshake completes but the SSH banner is delayed, continue investigating the application layer. Packet captures contain network metadata; keep capture duration short and follow organizational data policies. When needed, capture simultaneously on the client; do not assume the client received the SYN-ACK just because the server sent it.

For VPN, tunnel, or specific segment slowness, perform an MTU probe: ping -M do -s 1472 10.20.30.40 1472 bytes plus the 28-byte IPv4 header equals 1500; IPv6 or tunnel environments require recalculation. If ICMP is restricted, this test is inconclusive and cannot alone diagnose an MTU issue.

Command Checks: GSSAPI, PAM, and Directory Services

GSSAPI/Kerberos is a common source of pre-authentication waits. First run a comparison test with GSSAPI disabled:

ssh -vvv -o GSSAPIAuthentication=no [email protected] true
ssh -G ssh.example.internal | grep -i '^gssapiauthentication '

If disabling GSSAPI consistently speeds up the connection, GSSAPI is in the path, but that does not mean it should be globally disabled. Environments that rely on Kerberos single sign-on would lose functionality; changes should be scoped only to hosts where it is not needed, after confirmation from the identity platform.

If slowness occurs after password entry or only for domain accounts, check NSS and SSSD:

sudo systemctl status sssd --no-pager
getent passwd 'domain-user'
id 'domain-user'
sudo journalctl -u sssd --since '30 minutes ago' --no-pager
sudo grep -R --line-number --no-messages -E 'pam_(sss|ldap|winbind|mkhomedir|exec|motd)' /etc/pam.d

Replace domain-user with the actual authorized test account. If getent passwd and id latency matches the login delay and SSSD logs show upstream timeouts, the directory service call can be considered the root cause. Do not repeatedly test with wrong passwords on production accounts, as this may trigger lockouts. Do not delete PAM lines directly; PAM is shared by SSH, sudo, and console login; changes require console verification, configuration backup, and single-node canary.

Still Slow After Authentication: Check Session, Not Auth

Compare a non-interactive command with a TTY session:

time ssh [email protected] true
time ssh -tt [email protected] 'exit'

The second command triggers interactive shell logic. If only the interactive mode is slow, examine profile, motd, and home directory:

sudo find /etc/profile.d -maxdepth 1 -type f -printf '%f
' | sort
sudo grep -R --line-number --no-messages -E '(curl|wget|nc |dig |ldapsearch|mount |df )' \
  /etc/profile /etc/bash.bashrc /etc/profile.d /home/ops/.profile /home/ops/.bashrc
findmnt -T /home/ops
df -hT /home/ops

Paths must be adjusted for the real user; /etc/bash.bashrc does not exist on all distributions. Login scripts that invoke network commands, dynamically generate welcome messages, NFS home directory disconnects, or automount timeouts can all manifest as a missing prompt after successful authentication. Grep hits are only clues; you must read the scripts, compare changes, and verify on a test account — do not delete blindly.

For authorized complex issues, briefly trace the corresponding sshd child process to observe file and network system calls:

sudo strace -ff -tt -T -p <sshd_child_pid> -e trace=file,network

Replace <sshd_child_pid> with the child PID of the test connection; do not attach to the listener long-term. If connect(), openat(), or stat() show latency on the same external service, mount path, or script within the same test window, that constitutes actionable evidence. strace may expose paths and account information and has slight overhead; limit its duration and access scope.

Fix, Verify, and Rollback

Fixes should target the proven segment: DNS issues → fix DNS records, resolvers, or routing; directory service issues → fix LDAP/AD/KDC reachability, certificates, or cache policy; TCP issues → fix minimal ACL, routing, or MTU; profile issues → fix scripts or mount dependencies. Do not globally disable authentication features to mask upstream faults.

All changes to SSH configuration, PAM, directory services, or firewalls are high-risk. Impact may include new SSH logins, domain users, automation tasks, and management channels. Pre-execution checks must include: a second admin session or out-of-band console, target configuration backup, sshd -t syntax validation, current firewall rules export, and a defined change window. Backup via version control or local timestamp copy; canary limited to one non-critical host or a small host group; firewalls lack a universal dry-run, so use a minimal rule and source-segment canary.

Verification must not only confirm "admin can log in"; it must cover the originally slow segments, domain names and IPs, necessary jump paths, key authentication, and originally allowed password/domain authentication:

ssh -o BatchMode=yes -o ConnectTimeout=10 [email protected] true
ssh -o PreferredAuthentications=publickey -o PasswordAuthentication=no [email protected] true
sudo journalctl -u sshd --since '10 minutes ago' --no-pager
BatchMode=yes

disables interactive input, suitable for key-based automation verification; environments requiring passwords or MFA will fail by design. Prometheus can observe TCP retransmits, host network errors, DNS request failures, SSSD service status, etc.; specific metric names depend on the actual exporter.

If new accounts cannot log in, domain authentication anomalies appear, or automation fails, stop expanding the scope, restore the backup or revert the configuration management change, run sshd -t then reload. External DNS, directory service, and firewall changes must be rolled back via their own change tickets; they cannot be replaced by commands on the SSH host.

Retrospective

The retrospective should clearly document the wait phase, comparison samples, log or packet capture evidence, final root cause, change scope, verified accounts, and rollback time. Cement ssh -vvv, sshd -T, DNS/NSS checks, directory service checks, and the dual-session requirement into runbooks. This way, the next "SSH slow" incident is no longer a vague feeling but an observable, segmentable, minimally fixable, and safely rollbackable chain.

On-Call Troubleshooting Checklist: Make Tests Repeatable

When the problem is not yet stably reproducible, the most common error is each on-call engineer running a different set of commands, resulting in unaligned timelines. Establish a minimal collection template for SSH slowness: the client records date -Is, ssh -vvv ... true, domain/IP comparison, and source IP; the server records the same minute's sshd logs, sshd -T, listener state, and a conditional packet capture; for domain users, additionally record the latency of one getent passwd call. All timestamps must use a unified timezone, preferably ISO 8601 with explicit UTC indication.

Do not permanently write debug options into the global /etc/ssh/ssh_config. For example, temporarily setting LogLevel DEBUG3, forcing authentication order, or disabling an authentication method changes connection behavior for all users and may leak more connection metadata into logs. Test parameters should only be placed on a single command; after troubleshooting, retain an evidence summary, delete temporary capture files, or archive them per data retention policy.

For jump host chains, treat "to the jump host" and "from the jump host to the target" as separate monitoring and on-call items. Jump host CPU saturation, DNS resolution failure, directory service slowness, or connection reuse socket failure can all make the end user perceive the target host as slow. Only segmented measurement avoids misattributing jump host issues to the business server.

When a fix is confirmed effective, write the "effective" criteria as a re-verifiable condition: for example, the same source segment's repeated minimal command stage logs no longer show the original pause, while domain accounts, key accounts, and jump paths remain usable. This enables detection of regressions during future configuration upgrades or identity system changes.

Finally, account authentication failures, directory service timeouts, and network blocks should use different alerting and escalation paths. Lumping them together as "SSH unavailable" prolongs diagnosis and deprives the truly responsible systems of actionable evidence.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

TCPlatencytroubleshootingDNSSSHPAMGSSAPISSSD
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.