My plan to save up to 90% on EC2 on-demand pricing by migrating my projects to spot instances which can be terminated with two-minutes notice.

A few months ago I decided to challenge myself to migrate all of my projects to run on AWS EC2 Spot Instances. The catalyst for this was hearing how Yelp runs most of its platform on Spot Instances. A Spot Instance can save 90% of the on-demand price; the trade-off is that your instance can be taken away with only two-minutes warning. Running production workloads on spots is only possible if you engineer fault-tolerant systems and design for ephemeral environment.

Over the next few weeks, I will show you how I have migrated my projects from a VPS to AWS Spot Instances, through to putting together an entire CI/CD pipeline (also running on Spots) that is responsible for deploying my projects.

Before Spot Instances

Before I transitioned to Spot Instances, I had a couple of small VMs running on Linode. These VMs would run Ubuntu, and I used SaltStack to ensure that all of their configurations are in version control. This might be overkill for a personal project, but if needed, I could re-build the entire machine from the ground up with minimal effort on my part.

On top of my salted Ubuntu VMs I typically deploy Rails web applications via Capistrano.

Immutable Servers

The pattern of an Immutable Server presents the idea that the server is entirely immutable — once the server is built, it cannot be changed.

If the server needs to be changed, it has to be re-built from scratch with the new configuration change before switching the old server out for the new one.

The underlying philosophy is to remove the problem of configuration drift, where a change will be made at some point in time that isn't properly recorded or version controlled. Thus, making it difficult if not impossible to reliably reproduce the server setup in the future.

Adopting an Immutable Servers pattern in this environment would require building the server image as part of the build process, so that the final build artifact is the VM image, with your application ready to go. In my context, the end state should be a VM image with the rails application fully baked into the image.

The Plan

To achieve the Immutable Server pattern I will need to automate even more of my existing setup:

  • Instead of launching a vanilla Ubuntu image and bootstrapping it with SaltStack, I'll need a fully-salted AMI image ready to go with my basic setup.
  • Instead of shipping the application using Capistrano I will need to find a way to package and bundle my applications into AMIs that are built from a base image of my choosing.
  • I'm moving my projects from Linode to AWS, so I'll need to automate all the infrastructure that I'll need to support running Spot Instances.

So there are a few challenges to tackle, in the following blog posts I'll explain how I'll address them:

  1. Building Immutable Server images with SaltStack and Packer
  2. Building Infrastructure for Immutable Web Applications with Terraform
  3. Baking a Rails application into an Immutable Server image