Why Ansible Misinterprets ‘#’ Passwords: Uncovering the SSH ControlPersist Bug

Background

When performing large‑scale operations, DBAs often rely on Ansible. A colleague encountered a puzzling situation where Ansible could not connect using certain passwords, which seemed like a bug and prompted a detailed investigation.

Phenomenon

Using Ansible 2.7.8 (colleague) and the latest Ansible 2.14 with Python 3.11 (author), both reproduced the issue: passwords that match the pattern digit+"#"+any‑string (e.g., 1#fander, 12#fander) cause Ansible to report Invalid/incorrect password: Permission denied, please try again. Direct SSH connections with the same credentials work fine.

Environment

Three hosts were used:

Ansib​le server: 192.168.199.121 (no password needed)

Problematic target: 192.168.199.99 with password 1#fander Normal target: 192.168.199.131 with password

Root-123

Initial Tests

Running ansible 192.168.199.99 -m ping produced an UNREACHABLE error with the above passwords, while the normal target succeeded.

Debugging Process

1. Increase verbosity

Using -vvvvv revealed that Ansible ultimately invokes sshpass and ssh. A missing file error in the debug log pointed to an abnormal path.

2. Examine SSH command

The generated SSH command includes ControlPersist=60s, which creates a persistent control socket under /root/.ansible/cp/…. This socket allows subsequent connections to reuse the existing SSH session without re‑authenticating.

sshpass -d10 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s \
    -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 \
    -o 'ControlPath="/root/.ansible/cp/e2f9f7759b"' 192.168.199.99 \
    '/bin/sh -c "'"'echo ~root && sleep 0'"'"'

3. Socket reuse explains intermittent success

When the correct password is used first, the control socket is created and remains alive for the duration set by ControlPersist. Later, even if the password in /etc/ansible/hosts is wrong, Ansible reuses the existing socket, so the connection appears to succeed.

Extending ControlPersist to 600 seconds confirmed the behavior: the socket disappears only after the last activity plus the timeout, after which the faulty password again triggers the error.

4. Verify with paramiko

Switching the transport to paramiko (which does not use the SSH control socket) still reproduced the error, showing that the root cause lies in how Ansible parses the /etc/ansible/hosts file.

[defaults]
host_key_checking = False
callback_whitelist = timer
transport = paramiko

5. Identify the parsing bug

Inspection of Ansible’s inventory parser ( ini.py) revealed that passwords containing a hash are truncated at the hash character during parsing. Consequently, only the leading digits are passed to sshpass, causing the authentication failure.

Resolution

The issue is not with SSH or sshpass but with Ansible’s handling of the hosts file. To avoid the bug, avoid using # in passwords defined directly in the inventory file, or escape the character appropriately. Updating to a version where the parser correctly handles such strings, or using variables instead of inline passwords, also mitigates the problem.

Conclusion

Understanding the interaction between Ansible, sshpass, and SSH’s ControlPersist feature clarifies why certain passwords appear to work intermittently. By adjusting the inventory syntax or the control socket timeout, operators can prevent the misleading “successful” connections and ensure reliable automation.