Ansible Projects

On this page we are going to discuss exactly how Edmonds Commrce creates and manages Ansible projects.

Coding Standards

Here are the coding standars for Edmonds Commerce Ansible projects

Folders and Filenames

If the ansible project is a sub folder of another project the folder name should be ansible-project

If the ansible project is a standalone project with it's own repository the root of the repo is the ansible project root

Inventory

Inventory should be managed with a folder per environment that then contains multiple files.

The directory name should be inventory.{envirnoment-name}

Inventory files should be in ini format and have the .ini extension

Example Directory Structure

1
2
3
4
5
6
7
.
├── group_vars
├── host_vars
├── inventory.local
├── inventory.production
├── roles
└── plays

Playbooks

Playbook files should be in the plays folder though there may be a playbook-main.yaml in the root of the project

The file name should be in the format playbook-{playbook-name}.yml

Playbooks should not contain any actual tasks they should only include roles. Your tasks should be in roles

Whilst it is possible to have sub playbooks that are imported into the main playbook it is preferable to keep as much as possible in roles and to only have the main playbook.

Main Playbook

Each project should have a single "main" playbook that builds everything

Our main playbook must be called playbook-main.yml

The best practices refer to this file as site.yml however I do not think this is explicit enough.

Testing Single Hosts

When you are building, you probably dont want to run your playbook against all your hosts

You can pass in a --limit argument to ansible-playbook which then only runs against the host(s) you specify

Pass in a comma separated list of host, for example:

1
ansible-playbook playbook-main --limit=localhost,cnt-web-1

Managing Project Dependencies

Coming from the PHP world that now enjoys the wonderful features of Composer you might hope that Ansible has something similar. It doesn't

However we can get pretty close by following these rules:

Warning

Whilst following these rules get's something like Composer. It is no where near as fully featured or good. Main differences include: * ansible-galaxy will not create git repos in the roles folder

they will just be files

You can use this playbook to install your roles if you would like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
- hosts: localhost

  tasks:
    - name: Remove any none git folders from the roles directory. We assume that if it is a git repo
 its being developed on locally and should not be removed.
      shell: find roles -mindepth 1 -maxdepth 1 -type d '!' -exec test -e '{}/.git' ';' -print | xargs rm -rf

    - name: Install Galaxy Roles in the requirements.yml file
      local_action:
        command ansible-galaxy install -r {{ playbook_dir }}/../requirements.yml --roles-path {{ playbook_dir }}/../roles

    - name: Make sure the roles directory is being git ignored
      shell: printf "*\n!.gitignore" > {{ playbook_dir }}/../roles/.gitignore

Ignore the Roles Folder in Git

Add the roles folder to your .gitignore file. This then becomes the equivalent of vendor in the composer world.

If you want to work on custom roles you can you simply need to add an exclusion to your .gitignore file such as:

1
2
roles/*
!roles/my-custom-role

Track Dependencies in a Text File

ansible-galaxy allows us install multiple roles as listed in a file.

We can use this to provide us with rudimentary project level depenedency management

In your project root create a file called requirements.yml and in this file list all of your roles.

For example:

1
2
- src: edmondscommerce.lxc
- src: edmondscommerce.akeneo

Always Use the File Based Install to Update Roles

To install roles get in the habit of always using the mulitple roles file based approach

1
2
3
4
5
6
7
8
# go to the ansible project root
cd ansible-project

# ensure the roles directory exists
mkdir -p roles

# install all dependencies
ansible-galaxy install -r requirements.yml --roles-path=roles

Create Ansible Playbook to Install Roles

To make this easier you might want to create this ansible playbook in your Ansible project root:

Eg the file playbook-install-roles.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
- hosts: localhost
  tasks:
    - name: Check for uncommited work
      shell: for d in {{ playbook_dir }}/../roles/edmondscommerce.*; do cd $d;printf "\n\n--------------------\n\n"; pwd;  git status; cd ../; done
      args:
        executable: /bin/bash
      register: uncommited
    - debug: var=uncommited.stdout_lines

    - name: Confirm you want to proceed with nuking roles
      pause: prompt="Press return to contiue..."

    - name: Install Galaxy Roles in the requirements.yml file
      local_action:
        command ansible-galaxy install \
          --force \
          --keep-scm-meta \
          --role-file={{ playbook_dir }}/../requirements.yml \
          --roles-path={{ playbook_dir }}/../roles

    - name: Make sure the roles directory is being git ignored
      shell: printf "*\n!.gitignore" > {{ playbook_dir }}/../roles/.gitignore

Warning - thi

Warning - this playbook will totally delete your roles folder It's good in a purist sense because it forces you to

Ansible Configuration File

It is possible to manage the default inventory file and where the roles should be installed using an ansible.cfg file.

This should live in the root folder and have the following content

1
2
3
4
5
[defaults]

inventory = ./inventory.local/hosts.ini
roles_path = ./roles
retry_files_enabled = False

Using this means that it is no longer required to specify the inventory file or that the roles should be installed in the roles directory

This also stops Ansible making retry files - not sure what they are for but they cause clutter

Working with Roles as Git Based Repos

I suggest that the roles folder should not be tracked in your main project

Instead this should be treat more like the vendor directory when using composer

That means that the directory contents should be ignored by the parent project and it should then contain things retreived from ansible galaxy or cloned as repos from github etc

For roles that are ours they should be hosted on Github and Galaxy but we can clone from Github so that we can work on them.

For example if I want to work on the ansible lxc role

  • Add it as a dependency to the requirements.yml file in the project root
  • use ansible-galaxy to install all requirements ansible-galaxy install -r requirements.yml --roles-path roles
  • cd into the role and set it up as a repo tracking github version:
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    cd roles/edmondscommerce.lxc
    git init .
    git add -A
    git commit -am 'starting work on this role'
    git remote add master git@github.com:edmondscommerce/ansible-role-lxc.git#
    git branch --set-upstream-to=origin/master master
    git pull --allow-unrelated-histories
    # This may cause merge conflicts if you have already started to edit it
     if so just checkout your HEAD
    git checkout HEAD . && git commit -m 'reapplying latest changes'
    git push
    

Using Edmonds Commerce Roles

To use Edmonds Commerce roles you want to clone the actual git repo

To do this your requirements.yml file needs to have our roles defined like this:

1
2
3
4
- name: edmondscommerce.projectsetup
  src: git@github.com:edmondscommerce/ansible-role-projectSetup.git
  scm: git
  version: master

Secrets - Ansible Vault

For secrets we should encrypt things using Ansible Vault

The Ansible Vault password should be stored in the password manager

You should also store the file in your project but it must be git ignored

The way vault works we can store as many secrets as we want in the code they are all encrypted with one master password which ansible-playbook needs to be supplied with when running playbooks. You can provide this as a file path or via a prompt. You can configure the default behaviour in your ansible.cfg file in your project root:

1
2
3
4
5
6
7
[defaults]
inventory = ./hosts.ini
roles_path = ./roles
retry_files_enabled = False
pipelining = True
ask_vault_pass = False
vault_password_file=./vault-pass.secret

Read the docs:

Suggested easiest approach is like this:

1
2
3
4
fileContainingSecret="/tmp/ansible_rsa_key"
ansibleVaribleName="ansible_rsa_key"

cat "$fileContainingSecret" | ansible-vault encrypt_string --ask-vault-pass --stdin-name "$ansibleVaribleName"

And this will output something that you can then copy/paste into a vars file etc

For example generating an SSH key and then getting the encrypted version:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
✘-INT /opt/Projectsansible-project [master|●1✚ 8…2] 
17:25 $ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/joseph/.ssh/id_rsa): /tmp/ansible_rsa_key    
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /tmp/ansible_rsa_key.
Your public key has been saved in /tmp/ansible_rsa_key.pub.
The key fingerprint is:
SHA256:a0i83Y8pTFs0z8EusQIct/UOeDgv5Gd9Z584QbAsgUo joseph@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
|        .        |
|     E......     |
|    ...o =ooo    |
|     oo *.*o+.   |
|      o+S=.@..   |
|     . +=oB *.. o|
|      oo+B.. .ooo|
|       .+  + o ..|
|         .o . .  |
+----[SHA256]-----+
✔ /opt/Projects/ansible-project [master|●1✚ 8…2] 
17:29 $ cat /tmp/ansible_rsa_key | ansible-vault encrypt_string --ask-vault-pass --stdin-name 'ansible_rsa_key'
New vault password (default): 
Confirm vew vault password (default): 
Reading plaintext input from stdin. (ctrl-d to end input)
ansible_rsa_key: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          62333162363064616461306430303533623562303933633732393736343834373863613964643135
          [...snip...]
          633633313834363164653738633339363564
Encryption successful
✔ /opt/Projects/ansible-project [master|●1✚ 8…2] 
17:45 $ ^C
✘-INT /opt/Projects/ansible-project [master|●1✚ 8…2] 
17:45 $ cat /tmp/ansible_rsa_key.pub | ansible-vault encrypt_string --ask-vault-pass --stdin-name 'ansible_rsa_key_pub'
New vault password (default): 
Confirm vew vault password (default): 
Reading plaintext input from stdin. (ctrl-d to end input)
ansible_rsa_key_pub: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          33303961656133653932623965633739323861306236636438623864626133313936363131383431
          [...snip...]
          6666
Encryption successful

Shell Script to Generate and Encrypt Passwords

Here is a nice shell script which you can use to quickly generate encrypteed passwords for use in your playbooks:

useage:

./createPass.bash (ansible_variable_name defaults to 'varname')

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/usr/bin/env bash
readonly DIR=$(dirname $(readlink -f "$0"))
cd $DIR;
set -e
set -u
set -o pipefail
standardIFS="$IFS"
IFS=$'\n\t'
echo "
===========================================
$(hostname) $0 $@
===========================================
"
# Error Handling
backTraceExit () {
    local err=$?
    set +o xtrace
    local code="${1:-1}"
    printf "\n\nError in ${BASH_SOURCE[1]}:${BASH_LINENO[0]}. '${BASH_COMMAND}'\n\n exited with status: \n\n$err\n\n"
    # Print out the stack trace described by $function_stack
    if [ ${#FUNCNAME[@]} -gt 2 ]
    then
        echo "Call tree:"
        for ((i=1;i<${#FUNCNAME[@]}-1;i++))
        do
            echo " $i: ${BASH_SOURCE[$i+1]}:${BASH_LINENO[$i]} ${FUNCNAME[$i]}(...)"
        done
    fi
    echo "Exiting with status ${code}"
    exit "${code}"
}
trap 'backTraceExit' ERR
set -o errtrace
# Error Handling Ends

readonly varname=${1:-'variable_name'}

readonly password="$(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c32;echo;)"

readonly passFilePath=./../vault-pass.secret

if [[ ! -f $passFilePath ]]
then
    echo "Vault Pass File not found at $passFilePath
 you need to create this first"
    exit 1
fi

echo "$password"| ansible-vault encrypt_string --vault-password-file="$passFilePath" --stdin-name "$varname"

Plugins

Ansible has a plugins system. There are a lot of plugins but its not immediately obvious how you are supposed to use them

The answer is you need to whitelist them in your ansible.cfg file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
[defaults]
inventory = ./hosts.ini
roles_path = ./roles
retry_files_enabled = False
pipelining = True
ask_vault_pass = False
vault_password_file=./vault-pass.secret
strategy = debug

; plugins
callback_whitelist = debug
 profile_roles
profile_tasks

Plugins are separated by type:

  • cache
  • callback
  • connection
  • inventory
  • lookup
  • shell
  • module
  • strategy
  • vars

You need to whitelist them by type

You can see the list of plugins and the docs using the ansible-docs command, eg ansible-doc -lt callback

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
ansible-doc -lt callback
actionable          shows only items that need attention                                                         
cgroup_memory_recap Profiles maximum memory usage of tasks and full execution using cgroups                      
context_demo        demo callback that adds play/task context                                                    
counter_enabled     adds counters to the output items (tasks and hosts/task)                                     
debug               formatted stdout/stderr display                                                              
default             default Ansible screen output                                                                
dense               minimal stdout output                                                                        
foreman             Sends events to Foreman                                                                      
full_skip           suppresses tasks if all hosts skipped                                                        
grafana_annotations send ansible events as annotations on charts to grafana over http api.                       
hipchat             post task events to hipchat                                                                  
jabber              post task events to a jabber server                                                          
json                Ansible screen output as JSON                                                                
junit               write playbook output to a JUnit file.                                                       
log_plays           write playbook output to log file                                                            
logdna              Sends playbook logs to LogDNA                                                                
logentries          Sends events to Logentries                                                                   
logstash            Sends events to Logstash                                                                     
mail                Sends failure events via email                                                               
minimal             minimal Ansible screen output                                                                
null                Don't display stuff to screen                                                                
oneline             oneline Ansible screen output                                                                
osx_say             oneline Ansible screen output                                                                
profile_roles       adds timing information to roles                                                             
profile_tasks       adds time information to tasks                                                               
selective           only print certain tasks                                                                     
skippy              Ansible screen output that ignores skipped status                                            
slack               Sends play events to a Slack channel                                                         
splunk              Sends task result events to Splunk HTTP Event Collector                                      
stderr              Splits output, sending failed tasks to stderr                                                
sumologic           Sends task result events to Sumologic                                                        
syslog_json         sends JSON events to syslog                                                                  
timer               Adds time to play stats                                                                      
tree                Save host events to files                                                                    
unixy               condensed Ansible output                                                                     
yaml                yaml-ized Ansible screen output 

You can get the playbook/task yaml snippet for a plugin like this:

1
2
3
4
5
6
7
ansible-doc -s debug
- name: Print statements during execution
  debug:
      msg:                   # The customized message that is printed. If omitted, prints a generic message.
      var:                   # A variable name to debug.  Mutually exclusive with the 'msg' option.
      verbosity:             # A number that controls when the debug is run, if you set to 3 it will only run debug
                               when -vvv or above

And you can get the full docs by just putting the plugin name, eg ansible-doc debug

Ansible Tips

Here are some general tips:

Fix Unquoted Variables

Find and replace regexp using these patterns:

1
2
# Find
: \{\{(.+?)\}\}
1
2
# Replace
: "{{ $1 }}"