Ansible: Accelerated

So, you’ve been using Ansible, and have realized it can be really slow.

You ask, why is Ansible slow? The answer is simple. Ansible uses ssh as a ‘transport layer’, which starts a new connection for every task in a playbook.

This is not ideal, but luckily, we can change this behavior with a few simple steps.

How to Speed Up Ansible

To speed up Ansible, first locate your ansible config. For example, in an Ubuntu system it is located in /etc/ansible directory by default. You can save your ansible.cfg file in a few locations:

• Default
• User home directory
• If you use version control system (highly recommended), in project root directory.

Personally, I prefer the third solution. By keeping ansible.cfg in the project directory, combined with python virtual environments (where I can keep different versions of Ansible needed for specific project), I can be sure that every person who starts using this project will get the environment I expect, and ansible will behave as planned on each user machine.

When you look inside the default ansible.cfg, you will see the whole section [ssh] and [accelerate] is empty. This is what causes slow downs during runs of large playbooks with dozens of tasks and on multiple nodes.

To show you how to speed up Ansible, here is a simple playbook with a set of simple tasks:

1. 16 updates for sysctl.conf file, where you can modify operating system, network settings, etc.
2. Copy one ulimit file
3. Install Nginx repo
4. Nginx installation
5. Remove Nginx default config file
6. Add custom Nginx config file
7. Install logrotate file for Nginx
8. Create 10 directories

This presumes a certain level of are familiarity with those simple tasks in Ansible, so I will not be detailing the specifics of a basic playbook here.

Each task was created for one operation (so, I’ve got 10 tasks for 10 directories, and this was done on purpose – you can read about it below).

Every run was done twice, the first time on clean, fresh system and for the second one on system modified earlier (that is why no changes were made during the second run). I ran this playbook from my station and boxes were created in the closest AWS region.

So, let’s look at the timetable:

First the clean run, where all tasks were executed (installations, creations, etc) took almost 90 seconds. Wow… A lot of time for simple tasks, isn’t it?

The second run, which is really an empty run because everything was already set, took 50 seconds. Again – definitely too long.

Let’s modify the Ansible, then!

This is where the magic happens. Open your ansible.cfg file (wherever it is) and add this section:

[ssh_connection]
ssh_args=-C -o ForwardAgent=yes -o StrictHostKeyChecking=no -o ControlMaster=auto -o ControlPersist=60s
pipelining=True
control_path = /tmp/ansible-ssh-%%h-%%p-%%r

What did we do?

We just added ssh section with couple of parameters. The most important thing is:

pipelining=true

This option significantly reduces the number of SSH operations needed for every task and will definitely make Ansible work faster. But the crucial piece is that you have to remove the requiretty option from your sudoers file.

Pipelining will speed things up. You will see significant improvement here. Sometimes pipelining alone is not enough.

That is why we should add the rest of parameters to this section.

ForwardAgent=yes

I use it for the same reason that I have it in my ssh config. And although it isn’t directly related to accelerate Ansible, it is very useful.

StrictHostKeyChecking=no

In a dynamic environment I also want to have control of my knownhosts file. Again, this is not directly related to speeding up Ansible, but makes life easier.

ControlMaster=auto

This setting activates multiplexing. Auto means if a master connection exists, use it. If not – create a new one. Important tip – check you ssh config to see what the value of MaxSession parameter is. Set this option to meet your requirements here.

ControlPersist=60s

Tune this parameter to get the best results. 60 seconds is a default and can be enough, though.

These settings are passed to Ansible as the ssh_args parameters, as seen in the example above.

control_path = /tmp/ansible-ssh-%%h-%%p-%%r

There is a limit of characters in the filename for ssh sessions control files. I suggest keeping the filename (with path) as short as possible from one side and readable from the other. You can find examples of this naming convention in many places.

Now ansible.cfg is updated, lets see how it looks.

My first run, on a clean fresh system, was done in 29 seconds. The second, identical like before finished in 16 seconds. This is a significant improvement!

This is all that is needed to speed Ansible up. But there are some other things you should be aware of:

• If for any reason you cannot use pipelining, use accelerate.
• Play with transfer_method, especially if you copy a lot of files to remote machines.
• When installing files with apt or yum or any other package manager, using with_items can make it faster. Using with_items in your playbooks is always necessarily, but it will only make your run faster  in specific use cases such as package installations. One more tip for installing packages with apt; use two parameters: update_cache: yes and cache_valid_time: 3600.
• Use serial: X where X in bigger than 1. You can serialize your run, when you are executing a playbook on more than one server. Output will be a little bit less clear, but the whole process will be faster.
• Use async in your playbooks. This option gives Ansible the possibility to not to wait for finishing particular tasks in serial manner, but statuses of some tasks can be checked later. So this is some kind of a parallel run. First, check where you can use it. And then… use it 🙂
• Use handlers where possible. This isn’t a tip for speeding up Ansible, but it can help you if you mess up your playbooks and need to have commands for restarting the service.

I hope this helps. Learning how to speed up Ansible has been crucial for me, and I’m happy to share my config with you. Please let me know how it works for you in the comments below.

Read More →