Several years ago I built my own dotfiles repository as a way of being able to easily transport the config for my desktop environment between computers. At the time, I put together a Travis CI process to run the dotfiles setup to ensure that the setup script would run successfully whenever I pushed any changes.

The testing process was rather crude. Travis CI would run the setup script and provided the script exits with an exit code of 0, Travis CI would pass the build.

I've become familiar with Test Kitchen and InSpec for testing infrastructure and configuration management (Ansible, SaltStack, Puppet, Chef, etc). Whilst making a recent update to my dotfiles, I decided that I wanted a more robust testing process that tests the specific behaviours of the dotfiles setup process rather than just treating the script as a black box.

I've ended up with a testing process that will run the dotfiles setup and test that the script has performed specific behaviours such as creating files and symbolic links or checking that packages were installed properly. All of this runs as a CI process on GitHub actions using Test Kitchen and InSpec.

What is Test Kitchen?

Test Kitchen is a testing harness for testing infrastructure code on one or more platforms in isolation.

Test Kitchen will take a YAML configuration file (typically .kitchen.yml) and will use this to run the defined test suites.

Within the configuration file, four mandatory elements that need to be defined for Test Kitchen to run:

• Driver
  The driver determines what is used for the compute instance on which the code will be tested.
  
  Supported drivers include: Vagrant, Amazon EC2, Azure, Google Cloud Platform, Docker, DigitalOcean, Openstack
• Provisioner
  The provisioner is responsible for configuring the compute instance.
  
  Supported provisioners include: Ansible, SalStack, Chef, Puppet, PowerShell DSC, Shell
• Platform
  The platform are the operating system(s) on which you want to test your infrastructure code
• Suites
  The set of test suites to run to verify the correctness of your infrastructure code.
  
  Supported testing frameworks include: InSpec, Serverspec, Bats.

What is InSpec?

InSpec is a testing framework developed by Chef for testing infrastructure code. InSpec is built on top of RSpec and primarily provides a DSL and assertions for defining tests for infrastructure code.

describe port(443) do
  it { should be_listening }
  its('protocols') {should include 'tcp'}
end

Setting up Test Kitchen

The first thing we need to do will be to setup a Gemfile for the dependencies and to get Test Kitchen and InSpec installed:

bundle init
bundle add test-kitchen
bundle add kitchen-inspec

For simplicity, I will start by using Vagrant with VirtualBox as the driver for my Test Kitchen setup.

bundle add kitchen-vagrant

Next, we need a .kitchen.yml file which we can customise. Run kitchin init to generate .kitchen.yml configuration file.

I have altered the original .kitchen.yml to include the specifics I need to test my dotfiles.

#.kitchen.yml
driver:
  name: vagrant
  synced_folders:
    - ['.', '/home/vagrant/dotfiles']
  customize:
    memory: 1024

platforms:
  - name: ubuntu-20.04

provisioner:
  name: shell
  script: 'test/scripts/setup.sh'
  root_path: '/home/vagrant/'

suites:
  - name: default
    verifier:
      name: inspec
      inspec_tests:
        - test/integration/default

I'm using Vagrant for the driver, and I have specified that the memory requirements for the VM will be 1024MB. I am also using synced_folders parameter, so that /home/vagrant/dotfiles contains a copy of the current working directory. With Ubuntu images for vagrant, these images have a vagrant user by default, hence the path of /home/vagrant/dotfiles.

For the provisioner, I am using the built-in Shell provision that is part of Test Kitchen to run a script that I have created. This script is responsible for running the dotfiles installation within the Vargant VM:

#!/bin/bash
cd /home/vagrant/dotfiles

# Use sed to replace the SSH URL with the public URL, then initialize submodules
sed -i 's/git@github.com:/https:\/\/github.com\//' .gitmodules
git submodule update --init
sed -i 's/git@github.com:/https:\/\/github.com\//' .gitmodules
git submodule update --init

# Run the dotfiles setup process
./setup

For the purpose of testing this initial setup we can put together some basic tests:

# Check that autojump, curl, etc.. commands exist
%w(autojump curl htop python3 pip3 tmux vim zsh).each do |cmd|
  describe command(cmd) do
    it { should exist }
  end
end

I can now run kitchen test which will create the VM, provision the configuration, run the test suite before deleting the VM.

... a short while later ...

Now that I have an end-to-end testing process working I can add in additional tests to the test suite.

Using Test Kitchen with Amazon EC2 and GitHub Actions

I want to be able to run Test Kitchen as part of a CI process. When I first created my dotfiles repository, GitHub Actions didn't exist, so I went with Travis CI. However, I will be replacing Travis CI with GitHub Actions.

Another change I will need to make will be to use Amazon EC2 as the driver because nested virtualisation isn't currently supported in GitHub Actions.

Firstly, let's remove kitchen-vagrant from the Gemfile and add kitchen-ec2:

bundle remove kitchen-vagrant
bundle add kitchen-ec2

The driver and provision configuration in .kitchen.yml will also need to be adjusted:

#.kitchen.yml
driver:
  name: ec2
  region: eu-west-2
  instance_type: t3.small
  spot_price: on-demand

platforms:
  - name: ubuntu-20.04

provisioner:
  name: shell
  script: 'test/scripts/setup.sh'
  root_path: '/home/ubuntu/'
  data_path: '.'

suites:
  - name: default
    verifier:
      name: inspec
      inspec_tests:
        - test/integration/default

I have updated the driver block to use Amazon EC2. I have also specified my preference for the region and instance type. I have also chosen to use spot instances for Test Kitchen to help keep costs to a minimum. The maximum spot price has been set to the same as the on-demand price.

When I was using the Vagrant driver I used synced_folders which is specific to that driver. However, I can use data_path in the provision configuration to achieve the same effect. Finally, it is worth noting that the root_path has changed because the default user and home directory is different on the AWS ubuntu image.

Finally, I can now put in place a GI GitHub Actions workflow:

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check out code
      uses: actions/checkout@v2
    - name: Setup ruby
      uses: ruby/setup-ruby@v1
      with:
        ruby-version: 3.0
    - name: Install dependencies
      run: bundle install
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: eu-west-2
    - name: Test Kitchen
      run: kitchen test

The credentials used by the GitHub Action have the following associated IAM policy. This policy aims to provide the least privileges required for the GitHub Action to function. I achieved this by running the GitHub Action when the IAM user had no permissions and reviewing logs from AWS CloudTrail to incrementally add permissions as necessary.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/created-by": "test-kitchen"
                }
            },
            "Action": [
                "ec2:TerminateInstances"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:*"
            ],
            "Effect": "Allow",
            "Sid": "TerminateTestKitchenInstances"
        },
        {
            "Action": [
                "ec2:RunInstances"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:*"
            ],
            "Effect": "Allow",
            "Sid": "RunTestKitchenInstances"
        },
        {
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVpcs",
                "ec2:DescribeImages",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:DeleteKeyPair",
                "ec2:CreateKeyPair",
                "ec2:CreateKeyPair",
                "ec2:CreateSecurityGroup",
                "ec2:DeleteSecurityGroup",
                "ec2:CreateTags"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "RunAndTagTestKitchenInstances"
        },
        {
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow",
            "Sid": "GetCallerIdentity"
        }
    ]
}

Summary

I now have a more robust testing process for my dotfiles repository that tests the specifics of the dotfiles behaviour rather than relying on a black-box approach of checking the exit code of the setup script.

Over time I'll be able to add in more tests and possibly even more platforms to ensure that my dotfiles configuration will work exactly as I'd want it to on any platform.