Why Every SSH Keystroke Sends 100+ Packets and How to Eliminate the Overhead

How the Issue Was Discovered

The author built a high‑performance TUI game with Go's bubbletea and wish, targeting 2000+ concurrent players over SSH. During load testing a bug caused the server to stop sending the game screen, yet CPU usage only dropped by about 50%, prompting investigation.

Packet Capture with tcpdump

$ ./summarize_pcap.sh single-key.pcap
Total packets: 270

 36-byte msgs:   179 packets ( 66.3%)   6444 bytes
 Other data:        1 packet  (  0.4%)    564 bytes
 TCP ACKs:          90 packets ( 33.3%)

 Data packet rate: ~90 packets/second (avg 11.1 ms between data packets)

A single key press produced 270 packets, of which 179 were mysterious 36‑byte messages sent every ~11 ms, while only one packet (564 bytes) contained real game data.

Investigation Process

SSH flow‑control mechanisms?

PTY polling?

Issues in the bubbletea/wish framework?

The decisive clue was that all 36‑byte packets originated from the client, not the server.

Running a plain ssh session reproduced the problem, confirming it was unrelated to the game framework.

Root Cause: Keystroke Timing Obfuscation

In 2023 OpenSSH introduced a security feature called Keystroke Timing Obfuscation . It mitigates timing side‑channels by sending a flood of dummy packets ( SSH2_MSG_PING) roughly every 20 ms while the user types, masking the real keystroke intervals.

The client enables this feature when the server advertises the [email protected] extension during the handshake. For latency‑sensitive applications like online games, these chaff packets add substantial CPU and bandwidth overhead.

Solution

Clients could disable the feature with ObscureKeystrokeTiming=no, but requiring every player to change their SSH config is impractical.

Instead, the author examined Go's crypto/ssh source and discovered that the client only activates the obfuscation if the server declares the [email protected] extension. By preventing the server from advertising this extension, the client stops sending chaff packets.

The fix involved forking the Go crypto library, removing support for the [email protected] extension, and using a replace directive in go.mod to point to the custom fork.

Total CPU:  29.90% → 11.64%
Syscalls:   3.10s  → 0.66s
Crypto:     1.6s   → 0.11s
Bandwidth:  ~6.5 Mbit/sec → ~3 Mbit/sec

The changes halved CPU usage, reduced syscall time by 80 %, and cut bandwidth roughly in half. Maintaining a fork of the standard library, however, adds ongoing maintenance overhead.

LLM’s Role in Debugging

The author also reflected on using large language models for assistance. While ChatGPT and Claude Code offered useful hints—such as suggesting the issue might be normal SSH behavior—they occasionally mis‑identified the problem. Human intuition and deep protocol knowledge remained essential for the critical insight linking the test‑tool bug to SSH overhead.

Key Takeaways

OpenSSH’s 2023 keystroke‑timing obfuscation sends many dummy packets to hide typing patterns, which is a valuable security feature.

In high‑performance scenarios like online gaming, this feature can cause significant CPU and bandwidth penalties.

Disabling the [email protected] extension on the server side prevents the client from generating chaff packets.

Performance optimizations often require looking beyond the obvious code paths; protocol‑level features can be hidden bottlenecks.