Speed Up Ansible: Deep Dive into Execution Flow, Async Tasks, and Performance Tweaks

Understanding Ansible Execution and Measuring Speed

Ansible is convenient but often criticized for slow task execution, especially with large numbers of tasks or loops. Before optimizing, you must understand the execution flow and learn how to measure task duration.

11.1 Measuring Task Execution: profile_tasks Plugin

Ansible provides callback plugins for timing: profile_tasks: times each task profile_roles: times each role timer: times each play

Enable them by adding the plugin names to callback_whitelist in ansible.cfg:

[defaults]
callback_whitelist = profile_tasks
# callback_whitelist = profile_tasks, profile_roles, timer

Only profile_tasks is activated in the example.

Running a playbook with three hosts and 201 loop iterations (603 total tasks) produces timing output such as:

scp ------------------------------------ 57.96s
shell ---------------------------------- 42.78s
only one debug ------------------------- 0.07s

11.2 Ansible Execution Flow Analysis

Running ansible or ansible-playbook with -vvv shows detailed debug information, including the many SSH connections Ansible creates. In Ansible 2.9 each task establishes seven SSH connections.

For a single task the steps are:

1. Get remote user home directory.

2. Create a temporary directory under remote_tmp.

3. Discover the target node’s platform and Python interpreter.

4. Transfer the module code to the remote temporary file via SFTP.

5. Set execute permission on the remote file.

6. Execute the module.

7. Remove the temporary directory.

At the play level, Ansible selects a batch of hosts (controlled by forks), runs the first task on all hosts (each creating seven connections), then proceeds to the next task, and so on.

11.3 Review of Ansible Execution Strategies

The forks, serial, and strategy directives control concurrency and ordering. By default forks=5. serial runs a subset of hosts as a batch. strategy can be linear (default, all hosts wait for each other per task) or free (hosts proceed independently).

11.4 Increasing forks Value

Raising forks allows more hosts to run tasks concurrently, improving overall throughput.

11.5 Modifying Execution Strategy

Using strategy=free with a large forks value lets many hosts finish tasks quickly, though it may cause unfairness for dependent nodes.

11.6 Asynchronous Task Execution

Tasks can be run asynchronously with the async and poll parameters. Example:

- name: it is an async task
  copy:
    src:
    dest:
  async: 200
  poll: 2
- name: a sync task
  copy:
    src:
    dest:

If poll is greater than 0, Ansible still waits for the task to finish before moving on; if poll=0, the task truly runs in the background.

Command‑line shortcuts -B N (async timeout) and -P N (poll interval) are also available.

To wait for an async job, use the async_status module with the job’s ansible_job_id:

- name: Asynchronous yum task
  yum:
    name: nginx
    state: present
  async: 1000
  poll: 0
  register: yum_sleeper

- name: Wait for asynchronous job to end
  async_status:
    jid: "{{ yum_sleeper.ansible_job_id }}"
  register: job_result
  until: job_result.finished
  retries: 30

Multiple async jobs can be waited on with a loop over their job IDs.

11.7 Enabling SSH Persistent Connections

Long‑lived SSH connections reduce connection overhead. Set in ansible.cfg:

ssh_args = -C -o ControlMaster=auto -o ControlPersist=5d

Or configure the client’s /etc/ssh/ssh_config. Persistent connections create socket files under ~/.ansible/cp/, whose location is controlled by control_path_dir.

11.8 Enabling Pipelining

Pipelining sends module code over the SSH channel’s stdin instead of using SFTP, reducing the number of SSH connections per task. Enable with: pipelining = True When using sudo, the remote /etc/sudoers must have requiretty disabled.

With pipelining, a task’s execution flow drops from seven SSH connections to just the necessary discovery connections and a single execution connection.

Performance comparison (three hosts, 603 tasks):

# Without pipelining
scp ------------------------------------ 57.96s
shell ---------------------------------- 42.78s
# With pipelining
scp ------------------------------------ 39.99s
shell ---------------------------------- 20.29s

Pipelining roughly halves total runtime.

11.9 Tweaking Fact Collection

Fact gathering is slow. Disable it with gather_facts: no or limit to subsets, e.g., gather_subset=!all,!any,network. Facts can also be cached.

11.10 Shell‑Level Optimizations: Splitting Roles

Instead of a single playbook that runs all roles sequentially, run each role on its own batch of hosts in parallel, e.g.:

$ ansible-playbook nginx.yml >/tmp/nginx.log &
$ ansible-playbook mysql.yml >/tmp/mysql.log   # dependency, run foreground
$ ansible-playbook php.yml   >/tmp/php.log

This maximizes concurrency while respecting dependencies.

11.11 Third‑Party Strategy Plugin: Mitogen for Ansible

Mitogen provides faster strategy plugins ( mitogen_linear, mitogen_free, mitogen_host_pinned). Install and configure:

$ wget 'https://networkgenomics.com/try/mitogen-0.2.9.tar.gz'
$ mkdir -p ~/.ansible/plugins
$ tar xf mitogen-0.2.9.tar.gz -C ~/.ansible/plugins/

[defaults]
strategy_plugins = ~/.ansible/plugins/mitogen-0.2.9/ansible_mitogen/plugins/strategy
strategy = mitogen_linear

Performance test (with pipelining enabled):

# Native linear strategy
scp ------------------------------------ 39.99s
shell ---------------------------------- 20.29s

# Mitogen linear strategy
scp ------------------------------------ 3.37s
shell ---------------------------------- 8.02s

Mitogen may limit concurrent connections to 32 by default; adjust with the MITOGEN_POOL_SIZE environment variable. It also handles sudo differently, requiring appropriate sudoer entries.

Conclusion

By understanding Ansible’s execution steps and applying optimizations—profiling, adjusting forks/serial/strategy, using async, enabling persistent SSH and pipelining, limiting fact collection, parallelizing roles, and optionally adopting Mitogen—you can dramatically improve automation speed and efficiency.