Several years ago I built my own dotfiles repository as a way of being able to easily transport the config for my desktop environment between computers. At the time, I put together a Travis CI process to run the dotfiles setup to ensure that the setup script would run successfully whenever I pushed any changes.
The testing process was rather crude. Travis CI would run the setup script and provided the script exits with an exit code of 0, Travis CI would pass the build.
I've become familiar with Test Kitchen and InSpec for testing infrastructure and configuration management (Ansible, SaltStack, Puppet, Chef, etc). Whilst making a recent update to my dotfiles, I decided that I wanted a more robust testing process that tests the specific behaviours of the dotfiles setup process rather than just treating the script as a black box.
I've ended up with a testing process that will run the dotfiles setup and test that the script has performed specific behaviours such as creating files and symbolic links or checking that packages were installed properly. All of this runs as a CI process on GitHub actions using Test Kitchen and InSpec.
What is Test Kitchen?
Test Kitchen is a testing harness for testing infrastructure code on one or more platforms in isolation.
Test Kitchen will take a YAML configuration file (typically
.kitchen.yml) and will use this to run the defined test suites.
Within the configuration file, four mandatory elements that need to be defined for Test Kitchen to run:
The driver determines what is used for the compute instance on which the code will be tested.
Supported drivers include: Vagrant, Amazon EC2, Azure, Google Cloud Platform, Docker, DigitalOcean, Openstack
The provisioner is responsible for configuring the compute instance.
Supported provisioners include: Ansible, SalStack, Chef, Puppet, PowerShell DSC, Shell
The platform are the operating system(s) on which you want to test your infrastructure code
The set of test suites to run to verify the correctness of your infrastructure code.
Supported testing frameworks include: InSpec, Serverspec, Bats.
What is InSpec?
InSpec is a testing framework developed by Chef for testing infrastructure code. InSpec is built on top of RSpec and primarily provides a DSL and assertions for defining tests for infrastructure code.
Setting up Test Kitchen
The first thing we need to do will be to setup a Gemfile for the dependencies and to get Test Kitchen and InSpec installed:
bundle init bundle add test-kitchen bundle add kitchen-inspec
For simplicity, I will start by using Vagrant with VirtualBox as the driver for my Test Kitchen setup.
bundle add kitchen-vagrant
Next, we need a
.kitchen.yml file which we can customise. Run
kitchin init to generate
.kitchen.yml configuration file.
I have altered the original
.kitchen.yml to include the specifics I need to test my dotfiles.
I'm using Vagrant for the driver, and I have specified that the memory requirements for the VM will be 1024MB. I am also using
synced_folders parameter, so that
/home/vagrant/dotfiles contains a copy of the current working directory. With Ubuntu images for vagrant, these images have a
vagrant user by default, hence the path of
For the provisioner, I am using the built-in Shell provision that is part of Test Kitchen to run a script that I have created. This script is responsible for running the dotfiles installation within the Vargant VM:
For the purpose of testing this initial setup we can put together some basic tests:
I can now run
kitchen test which will create the VM, provision the configuration, run the test suite before deleting the VM.
... a short while later ...
Now that I have an end-to-end testing process working I can add in additional tests to the test suite.
Using Test Kitchen with Amazon EC2 and GitHub Actions
I want to be able to run Test Kitchen as part of a CI process. When I first created my dotfiles repository, GitHub Actions didn't exist, so I went with Travis CI. However, I will be replacing Travis CI with GitHub Actions.
Another change I will need to make will be to use Amazon EC2 as the driver because nested virtualisation isn't currently supported in GitHub Actions.
Firstly, let's remove
kitchen-vagrant from the Gemfile and add
bundle remove kitchen-vagrant bundle add kitchen-ec2
The driver and provision configuration in
.kitchen.yml will also need to be adjusted:
I have updated the driver block to use Amazon EC2. I have also specified my preference for the region and instance type. I have also chosen to use spot instances for Test Kitchen to help keep costs to a minimum. The maximum spot price has been set to the same as the
When I was using the Vagrant driver I used
synced_folders which is specific to that driver. However, I can use
data_path in the provision configuration to achieve the same effect. Finally, it is worth noting that the
root_path has changed because the default user and home directory is different on the AWS ubuntu image.
Finally, I can now put in place a GI GitHub Actions workflow:
The credentials used by the GitHub Action have the following associated IAM policy. This policy aims to provide the least privileges required for the GitHub Action to function. I achieved this by running the GitHub Action when the IAM user had no permissions and reviewing logs from AWS CloudTrail to incrementally add permissions as necessary.
I now have a more robust testing process for my dotfiles repository that tests the specifics of the dotfiles behaviour rather than relying on a black-box approach of checking the exit code of the setup script.
Over time I'll be able to add in more tests and possibly even more platforms to ensure that my dotfiles configuration will work exactly as I'd want it to on any platform.