Ansible Projects
On this page we are going to discuss exactly how Edmonds Commrce creates and manages Ansible projects.
Coding Standards¶
Here are the coding standars for Edmonds Commerce Ansible projects
Folders and Filenames¶
If the ansible project is a sub folder of another project
the folder name should be ansible-project
If the ansible project is a standalone project with it's own repository the root of the repo is the ansible project root
Inventory¶
Inventory should be managed with a folder per environment that then contains multiple files.
The directory name should be inventory.{envirnoment-name}
Inventory files should be in ini format and have the .ini extension
Example Directory Structure¶
.
├── group_vars
├── host_vars
├── inventory.local
├── inventory.production
├── roles
└── plays
Playbooks¶
Playbook files should be in the plays folder
though there may be a playbook-main.yaml in the root of the project
The file name should be in the format playbook-{playbook-name}.yml
Playbooks should not contain any actual tasks they should only include roles. Your tasks should be in roles
Whilst it is possible to have sub playbooks that are imported into the main playbook it is preferable to keep as much as possible in roles and to only have the main playbook.
Main Playbook¶
Each project should have a single "main" playbook that builds everything
Our main playbook must be called playbook-main.yml
The best practices refer to this file as site.yml
however I do not think this is explicit enough.
Testing Single Hosts¶
When you are building, you probably dont want to run your playbook against all your hosts
You can pass in a --limit argument to ansible-playbook which then only runs against the host(s) you specify
Pass in a comma separated list of host, for example:
ansible-playbook playbook-main --limit=localhost,cnt-web-1
Managing Project Dependencies¶
Coming from the PHP world that now enjoys the wonderful features of Composer you might hope that Ansible has something similar. It doesn't
However we can get pretty close by following these rules:
Warning
Whilst following these rules get's something like Composer. It is no where near as fully featured or good.
Main differences include:
* ansible-galaxy will not create git repos in the roles folder
they will just be files
You can use this playbook to install your roles if you would like:
- hosts: localhost
tasks:
- name: Remove any none git folders from the roles directory. We assume that if it is a git repo
its being developed on locally and should not be removed.
shell: find roles -mindepth 1 -maxdepth 1 -type d '!' -exec test -e '{}/.git' ';' -print | xargs rm -rf
- name: Install Galaxy Roles in the requirements.yml file
local_action:
command ansible-galaxy install -r {{ playbook_dir }}/../requirements.yml --roles-path {{ playbook_dir }}/../roles
- name: Make sure the roles directory is being git ignored
shell: printf "*\n!.gitignore" > {{ playbook_dir }}/../roles/.gitignore
Ignore the Roles Folder in Git¶
Add the roles folder to your .gitignore file. This then becomes the equivalent of vendor in the composer world.
If you want to work on custom roles
you can
you simply need to add an exclusion to your .gitignore file
such as:
roles/*
!roles/my-custom-role
Track Dependencies in a Text File¶
ansible-galaxy allows us install multiple roles as listed in a file.
We can use this to provide us with rudimentary project level depenedency management
In your project root
create a file called requirements.yml
and in this file list all of your roles.
For example:
- src: edmondscommerce.lxc
- src: edmondscommerce.akeneo
Always Use the File Based Install to Update Roles¶
To install roles get in the habit of always using the mulitple roles file based approach
# go to the ansible project root
cd ansible-project
# ensure the roles directory exists
mkdir -p roles
# install all dependencies
ansible-galaxy install -r requirements.yml --roles-path=roles
Create Ansible Playbook to Install Roles¶
To make this easier you might want to create this ansible playbook in your Ansible project root:
Eg the file playbook-install-roles.yml
- hosts: localhost
tasks:
- name: Check for uncommited work
shell: for d in {{ playbook_dir }}/../roles/edmondscommerce.*; do cd $d;printf "\n\n--------------------\n\n"; pwd; git status; cd ../; done
args:
executable: /bin/bash
register: uncommited
- debug: var=uncommited.stdout_lines
- name: Confirm you want to proceed with nuking roles
pause: prompt="Press return to contiue..."
- name: Install Galaxy Roles in the requirements.yml file
local_action:
command ansible-galaxy install \
--force \
--keep-scm-meta \
--role-file={{ playbook_dir }}/../requirements.yml \
--roles-path={{ playbook_dir }}/../roles
- name: Make sure the roles directory is being git ignored
shell: printf "*\n!.gitignore" > {{ playbook_dir }}/../roles/.gitignore
Warning - thi
Warning - this playbook will totally delete your roles folder It's good in a purist sense because it forces you to
Ansible Configuration File¶
It is possible to manage the default inventory file
and where the roles should be installed using an ansible.cfg file.
This should live in the root folder and have the following content
[defaults]
inventory = ./inventory.local/hosts.ini
roles_path = ./roles
retry_files_enabled = False
Using this means that it is no longer required to specify the inventory file or that the roles should be installed in the roles directory
This also stops Ansible making retry files - not sure what they are for but they cause clutter
Working with Roles as Git Based Repos¶
I suggest that the roles folder should not be tracked in your main project
Instead this should be treat more like the vendor directory when using composer
That means that the directory contents should be ignored by the parent project and it should then contain things retreived from ansible galaxy or cloned as repos from github etc
For roles that are ours they should be hosted on Github and Galaxy but we can clone from Github so that we can work on them.
For example if I want to work on the ansible lxc role
- Add it as a dependency to the
requirements.ymlfile in the project root - use ansible-galaxy to install all requirements
ansible-galaxy install -r requirements.yml --roles-path roles - cd into the role and set it up as a repo tracking github version:
cd roles/edmondscommerce.lxc git init . git add -A git commit -am 'starting work on this role' git remote add master git@github.com:edmondscommerce/ansible-role-lxc.git# git branch --set-upstream-to=origin/master master git pull --allow-unrelated-histories # This may cause merge conflicts if you have already started to edit it if so just checkout your HEAD git checkout HEAD . && git commit -m 'reapplying latest changes' git push
Using Edmonds Commerce Roles¶
To use Edmonds Commerce roles you want to clone the actual git repo
To do this your requirements.yml file needs to have our roles defined like this:
- name: edmondscommerce.projectsetup
src: git@github.com:edmondscommerce/ansible-role-projectSetup.git
scm: git
version: master
Secrets - Ansible Vault¶
For secrets we should encrypt things using Ansible Vault
The Ansible Vault password should be stored in the password manager
You should also store the file in your project but it must be git ignored
The way vault works
we can store as many secrets as we want in the code
they are all encrypted with one master password which ansible-playbook needs to be supplied with when running playbooks. You can provide this as a file path
or via a prompt. You can configure the default behaviour in your ansible.cfg file in your project root:
[defaults]
inventory = ./hosts.ini
roles_path = ./roles
retry_files_enabled = False
pipelining = True
ask_vault_pass = False
vault_password_file=./vault-pass.secret
Read the docs:
- https://docs.ansible.com/ansible/latest/user_guide/vault.html
- https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html#best-practices-for-variables-and-vaults
Suggested easiest approach is like this:
fileContainingSecret="/tmp/ansible_rsa_key"
ansibleVaribleName="ansible_rsa_key"
cat "$fileContainingSecret" | ansible-vault encrypt_string --ask-vault-pass --stdin-name "$ansibleVaribleName"
And this will output something that you can then copy/paste into a vars file etc
For example generating an SSH key and then getting the encrypted version:
✘-INT /opt/Projectsansible-project [master|●1✚ 8…2]
17:25 $ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/joseph/.ssh/id_rsa): /tmp/ansible_rsa_key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /tmp/ansible_rsa_key.
Your public key has been saved in /tmp/ansible_rsa_key.pub.
The key fingerprint is:
SHA256:a0i83Y8pTFs0z8EusQIct/UOeDgv5Gd9Z584QbAsgUo joseph@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
| . |
| E...... |
| ...o =ooo |
| oo *.*o+. |
| o+S=.@.. |
| . +=oB *.. o|
| oo+B.. .ooo|
| .+ + o ..|
| .o . . |
+----[SHA256]-----+
✔ /opt/Projects/ansible-project [master|●1✚ 8…2]
17:29 $ cat /tmp/ansible_rsa_key | ansible-vault encrypt_string --ask-vault-pass --stdin-name 'ansible_rsa_key'
New vault password (default):
Confirm vew vault password (default):
Reading plaintext input from stdin. (ctrl-d to end input)
ansible_rsa_key: !vault |
$ANSIBLE_VAULT;1.1;AES256
62333162363064616461306430303533623562303933633732393736343834373863613964643135
[...snip...]
633633313834363164653738633339363564
Encryption successful
✔ /opt/Projects/ansible-project [master|●1✚ 8…2]
17:45 $ ^C
✘-INT /opt/Projects/ansible-project [master|●1✚ 8…2]
17:45 $ cat /tmp/ansible_rsa_key.pub | ansible-vault encrypt_string --ask-vault-pass --stdin-name 'ansible_rsa_key_pub'
New vault password (default):
Confirm vew vault password (default):
Reading plaintext input from stdin. (ctrl-d to end input)
ansible_rsa_key_pub: !vault |
$ANSIBLE_VAULT;1.1;AES256
33303961656133653932623965633739323861306236636438623864626133313936363131383431
[...snip...]
6666
Encryption successful
Shell Script to Generate and Encrypt Passwords¶
Here is a nice shell script which you can use to quickly generate encrypteed passwords for use in your playbooks:
useage:
./createPass.bash (ansible_variable_name defaults to 'varname')
#!/usr/bin/env bash
readonly DIR=$(dirname $(readlink -f "$0"))
cd $DIR;
set -e
set -u
set -o pipefail
standardIFS="$IFS"
IFS=$'\n\t'
echo "
===========================================
$(hostname) $0 $@
===========================================
"
# Error Handling
backTraceExit () {
local err=$?
set +o xtrace
local code="${1:-1}"
printf "\n\nError in ${BASH_SOURCE[1]}:${BASH_LINENO[0]}. '${BASH_COMMAND}'\n\n exited with status: \n\n$err\n\n"
# Print out the stack trace described by $function_stack
if [ ${#FUNCNAME[@]} -gt 2 ]
then
echo "Call tree:"
for ((i=1;i<${#FUNCNAME[@]}-1;i++))
do
echo " $i: ${BASH_SOURCE[$i+1]}:${BASH_LINENO[$i]} ${FUNCNAME[$i]}(...)"
done
fi
echo "Exiting with status ${code}"
exit "${code}"
}
trap 'backTraceExit' ERR
set -o errtrace
# Error Handling Ends
readonly varname=${1:-'variable_name'}
readonly password="$(< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c32;echo;)"
readonly passFilePath=./../vault-pass.secret
if [[ ! -f $passFilePath ]]
then
echo "Vault Pass File not found at $passFilePath
you need to create this first"
exit 1
fi
echo "$password"| ansible-vault encrypt_string --vault-password-file="$passFilePath" --stdin-name "$varname"
Plugins¶
Ansible has a plugins system. There are a lot of plugins but its not immediately obvious how you are supposed to use them
The answer is
you need to whitelist them in your ansible.cfg file:
[defaults]
inventory = ./hosts.ini
roles_path = ./roles
retry_files_enabled = False
pipelining = True
ask_vault_pass = False
vault_password_file=./vault-pass.secret
strategy = debug
; plugins
callback_whitelist = debug
profile_roles
profile_tasks
Plugins are separated by type:
- cache
- callback
- connection
- inventory
- lookup
- shell
- module
- strategy
- vars
You need to whitelist them by type
You can see the list of plugins and the docs using the ansible-docs command, eg ansible-doc -lt callback
ansible-doc -lt callback
actionable shows only items that need attention
cgroup_memory_recap Profiles maximum memory usage of tasks and full execution using cgroups
context_demo demo callback that adds play/task context
counter_enabled adds counters to the output items (tasks and hosts/task)
debug formatted stdout/stderr display
default default Ansible screen output
dense minimal stdout output
foreman Sends events to Foreman
full_skip suppresses tasks if all hosts skipped
grafana_annotations send ansible events as annotations on charts to grafana over http api.
hipchat post task events to hipchat
jabber post task events to a jabber server
json Ansible screen output as JSON
junit write playbook output to a JUnit file.
log_plays write playbook output to log file
logdna Sends playbook logs to LogDNA
logentries Sends events to Logentries
logstash Sends events to Logstash
mail Sends failure events via email
minimal minimal Ansible screen output
null Don't display stuff to screen
oneline oneline Ansible screen output
osx_say oneline Ansible screen output
profile_roles adds timing information to roles
profile_tasks adds time information to tasks
selective only print certain tasks
skippy Ansible screen output that ignores skipped status
slack Sends play events to a Slack channel
splunk Sends task result events to Splunk HTTP Event Collector
stderr Splits output, sending failed tasks to stderr
sumologic Sends task result events to Sumologic
syslog_json sends JSON events to syslog
timer Adds time to play stats
tree Save host events to files
unixy condensed Ansible output
yaml yaml-ized Ansible screen output
You can get the playbook/task yaml snippet for a plugin like this:
ansible-doc -s debug
- name: Print statements during execution
debug:
msg: # The customized message that is printed. If omitted, prints a generic message.
var: # A variable name to debug. Mutually exclusive with the 'msg' option.
verbosity: # A number that controls when the debug is run, if you set to 3 it will only run debug
when -vvv or above
And you can get the full docs by just putting the plugin name, eg ansible-doc debug
Ansible Tips¶
Here are some general tips:
Fix Unquoted Variables¶
Find and replace regexp using these patterns:
# Find
: \{\{(.+?)\}\}
# Replace
: "{{ $1 }}"