I've set myself the challenge of migrating all of my projects to run on EC2 Spot Instances, which can be terminated with 2 minutes' notice.

Over the next few weeks I have set myself the challenge of migrating all of my personal projects from a typical Linode VPS setup, into AWS and running on AWS EC2 Spot Instances.

What's a Spot Instance?

A Spot Instance is a spare EC2 compute resource that is made available to you at a discount of up to 90% off the on-demand price.

That sounds good, right? However, EC2 can interrupt a Spot Instance with a two-minute warning when it needs the spare capacity back.

My interest in spot instances started when I found out that Yelp runs most of its platform on AWS Spot Instances. The cost savings are attractive, as is the engineering challenge of setting up my projects to survive in an ephemeral environment.

EC2 Costs

If you've not used EC2 before, you'll find that pricing varies depending on what you're using. The instance type and pricing structure matter.

The type of instance determines how many CPUs, how much RAM, network performance, etc. The pricing structure also varies depending on what you need:

  • On-Demand
    On-demand pricing is fixed at a given hourly rate. This is the most common pricing model that you'd expect from any typical compute or VPS provider.

    Provided EC2 has the capacity, you will get the instance at the hourly rate and the instance will run as long as you need it to, without being interrupted.
  • Reserved Instances (RI)
    RI is designed to reserve compute resources for a given period of time. Because you're reserving compute resources upfront (and paying upfront) AWS offer a discount of up to 75% off the on-demand price.

    RI can be useful for long-lived services such as database clusters. As with on-demand, provided the capacity exists you can start an RI instance and use it for the reserved duration without it being interrupted.
    savings are attractive, as is the engineering challenge
  • Spot Instances
    Save up to 90% off on-demand prices by using spare and unused compute capacity. However, be prepared that the spare capacity can be recalled with two minutes' notice. The exact pricing of a spot instance changes over time based on supply and demand.

For AWS a Spot Instance is the most cost-effective way of using EC2, but it does have the limitation that it could be taken away at any moment with little notice!

The Challenge

Migrating my projects to run not only in AWS but also on Spot Instances will require some re-engineering of how my projects work. I anticipate that this will only be possible if I adopt more automation than I currently use for my personal projects.

This will be an opportunity to experiment with the concept of immutable servers. I also plan to use AWS's other services, such as CloudFront and RDS.

I'll use my blog to record the progress and lessons learned as I transition my projects over to Spot Instances over the next few weeks.