Legacy systems can be scary, they’re often vast and sprawling with incomplete documentation. And as with all legacy systems, they’re business critical.

It can be a real challenge re-writing a legacy applications, especially if the tech stack is so old that doing a phased migration is out of the question.

A few years ago I worked on a project which was a significant re-write so that a system could move away from it’s dependency on other legacy applications.. It took several teams 9 months to implement the changes, and 14 people to deploy the application over an 8 hour deploy window one Friday night. There were many missing dependencies and configurations which wasn’t helped by incomplete release notes.

More recently I worked on migrating a project from a legacy Microsoft tech stack to a slightly less legacy tech stack. The technology was so old that we knew it would have to be a “big bang” migration when the new systems was ready.

Keen to avoid all of the headache of past experiences we looked at building an immutable development server that could be used to test the application throughout the development. I’ve previously written on my blog about some of the benefits of immutable servers.

The plan is to use Packer (a provisioning tool) to provision our server with everything we need. When it comes to deploy day, our Packer setup and associated scripts will provide all the documentation we need.

The Tech Stacks

Legacy application:

  • Microsoft Access
  • Windows Server 2012
  • Microsoft SQL Server 2012

New Application:

  • .Net Core 3.1
  • Windows Server 2012
  • Microsoft SQL Server 2012

This is a bit of hybrid migration, we're sticking with 2012, but ditching Microsoft Access for something a bit more modern — .Net Core 3.1.

Introducing: Packer

Packer is a tool produced by HashiCorp that automates the creation of machine images:

It embraces modern configuration management by encouraging you to use automated scripts to install and configure the software within your Packer-made images. Packer brings machine images into the modern age, unlocking untapped potential and opening new opportunities.

Out of the box, Packer can produce machine images for:

  • Microsoft Azure
  • AWS
  • vmware
  • Google Cloud Platform
  • Docker
  • Digital Ocean
  • ...and many more

When Packer builds a machine image, the process will take an existing machine image and run through a series of provisioning steps to produce your new image. A provisioning step can be anything from a Bash script or a PowerShell script all the way through to applying configuration from SaltStack/Chef/Puppet/Ansible/etc.

As we're looking to produce a custom Windows Server 2012 image, we'll start with a plain Windows Server 2012 image and apply a series of provisioning steps to produce the setup that we want.

Note: In this blog post I will be building the image as an AMI for use in AWS. Packer supports many Builders including Azure, GCP, etc. I picked AWS just because I have my AWS account details to hand.

The Provisioning Process

We will be looking to take a Windows Server 2012 image and:

  1. Install OpenSSH and configure it to allow us to connect (by providing a public key)
  2. Install IIS
  3. Install .Net Core 3.1 Hosting Bundle
  4. Install SQL Server Express 2012
  5. Set the Administrator password for the server

At its simplest packer takes a JSON configuration file which provides it with all of the instructions needed to perform a build.

For example:

packer builder win-server.json

Two of our steps involve setting up credentials. OpenSSH will need to know our public key. We will also need to set a password for the Windows Administrator account.

For ease, we will setup a wrapper script that will allow us to generate a password for our administrator account and a key pair to use for SSH:

#!/bin/bash
set -e

# Generate an admin password
ADMIN_PASSWORD=$(</dev/urandom tr -dc '1234567890abcdefhijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ' | head -c18)
export ADMIN_PASSWORD

# Generate a public/private key pair for ssh
rm -f ssh-key ssh-key.pub
ssh-keygen -f ssh-key -q -N ""

packer build win-server.json

# Output administrator password
echo "Administrator password was set to: ${ADMIN_PASSWORD}"
build.sh

In the above build script a password is generated and assigned to the environment variable ADMIN_PASSWORD. ssh-keygen is used to generate an SSH key pair. Finally, we invoke packer and print out the Administrator password (assuming Packer runs successfully).

To run the above script we will also need an initial win-server.json:

{
    "variables": {
        "admin_password": "{{env `ADMIN_PASSWORD`}}"
    },
    "builders": [
        {
            "type": "amazon-ebs",
            "region": "eu-west-2",
            "instance_type": "t3a.xlarge",
            "source_ami_filter": {
                "filters": {
                    "virtualization-type": "hvm",
                    "name": "*Windows_Server-2012-R2*English-64Bit-Base*",
                    "root-device-type": "ebs"
                },
                "most_recent": true,
                "owners": "amazon"
            },
            "launch_block_device_mappings": [
                {
                    "device_name": "/dev/sda1",
                    "delete_on_termination": true,
                    "volume_size": 60,
                    "volume_type": "gp2"
                }
            ],
            "ami_name": "legacy-win2012-sql2012-netcore31 {{timestamp}}",
            "user_data_file": "./bootstrap.txt",
            "communicator": "winrm",
            "winrm_username": "Administrator",
            "winrm_port": 5986,
            "winrm_insecure": true,
            "winrm_use_ssl": true,
            "tags": {
                "Name": "Legacy: Windows Server 2012, SQL Server Express 2012, .Net Core 3.1"
            }
        }
    ],
    "provisioners": [
    ]
}
win-server.json

Our win-server.json file tells packer that we want to built a server image on top of a Windows Server 2012 image provided by Amazon (see source_ami_filter). We also tell Packer that we want the server used for provision to have a 60GB hard disk - see launch_block_device_mappings.

Packer needs to be able to talk to the server during the build process so that it can provide instructions for what needs provisioning. If you use Packer to provision Linux machines, you'll typically tell it how to communicate with the VM over SSH. With Windows, on the other hand you will need to tell it how to communicate with Windows Remote Management (winrm). Hence the values for winrm_username, winrm_port, winrm_insecure and winrm_use_ssl.

Most Windows images do not have Windows Remote Management setup out of the box. As a result we need to configure this. In AWS a server can run certain commands based on "user data" when it boots. In our example here we're telling Packer to provide AWS with some user data from a text file ("user_data_file": "./bootstrap.txt"):

<powershell>

write-output "Running User Data Script"
write-host "(host) Running User Data Script"

Set-ExecutionPolicy Unrestricted -Scope LocalMachine -Force -ErrorAction Ignore

# Don't set this before Set-ExecutionPolicy as it throws an error
$ErrorActionPreference = "stop"

# Remove HTTP listener
Remove-Item -Path WSMan:\Localhost\listener\listener* -Recurse

$Cert = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName "packer"
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $Cert.Thumbprint -Force

# WinRM
write-output "Setting up WinRM"
write-host "(host) setting up WinRM"

cmd.exe /c winrm quickconfig -q
cmd.exe /c winrm set "winrm/config" '@{MaxTimeoutms="1800000"}'
cmd.exe /c winrm set "winrm/config/winrs" '@{MaxMemoryPerShellMB="1024"}'
cmd.exe /c winrm set "winrm/config/service" '@{AllowUnencrypted="true"}'
cmd.exe /c winrm set "winrm/config/client" '@{AllowUnencrypted="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/client/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{CredSSP="true"}'
cmd.exe /c winrm set "winrm/config/listener?Address=*+Transport=HTTPS" "@{Port=`"5986`";Hostname=`"packer`";CertificateThumbprint=`"$($Cert.Thumbprint)`"}"
cmd.exe /c netsh advfirewall firewall set rule group="remote administration" new enable=yes
cmd.exe /c netsh firewall add portopening TCP 5986 "Port 5986"
cmd.exe /c net stop winrm
cmd.exe /c sc config winrm start= auto
cmd.exe /c net start winrm

</powershell>

If we run our ./build.sh now this will take the Amazon Windows Server 2012 image, spin up a server and configure Windows Remote Management. Packer would then go on to use Windows Remote Management to run any provisioners but we're yet to define these.

The Provisioning Steps

Most of us are use to downloading  Microsoft SQL Server (picking ENU\x64\SQLEXPRADV_x64_ENU.exe) and running through a graphical installed to configure and install what we need.

The challenge with provisioning a machine image with Packer is that we need to be able to describe everything via the command line. There are no provisions for interacting with a GUI or doing point-and-click operations.

Fortunately, the Microsoft SQL Server installer was designed to be executed on the CLI. Microsoft provides comprehensive documentation on the command prompt parameters that can be used with the SQL Server installer.

We can provide the installed with a configuration file:

Documentation of command prompt parameter for SQL Server

The syntax and configuration options for the configuration file are quite complicated, and I originally started trying to craft one from scratch by this proved (as you'd expect) to be error prone and tedious.

After a bit more research I discovered that the install could be run in a way that will generate a configuration file for you.

Run:

.\SQLEXPRADV_x64_ENU /ACTION=INSTALL /UIMODE=Normal

This will launch the installation wizard:

Click through the screens as you normally would configuring the installation with your desired settings, you will eventually get to a screen that looks like this:

"Ready to Install" step showing configuration file path

Note the "Configuration file path" displayed at the bottom of this wizard screen. This is the path to the configuration file you've just generated. Take a copy of the configuration file and save it as SqlConfigurationFile.ini alongside your win-server.json file.

We're now at a point where we can built out our provisioning steps as follows:

    ...
    "provisioners": [
        {
            "type": "file",
            "source": "ssh-key.pub",
            "destination": "C:\\Users\\Administrator\\.ssh\\authorized_keys"
        },
        {
            "type": "file",
            "source": "SqlConfigurationFile.ini",
            "destination": "C:\\SqlConfigurationFile.ini"
        },
        {
            "type": "file",
            "source": "sshd_config",
            "destination": "C:\\Users\\Administrator\\sshd_config"
        },
        {
            "type": "powershell",
            "environment_vars": [
                "ADMIN_PASSWORD={{ user `admin_password` }}"
            ],
            "elevated_user": "Administrator",
            "elevated_password": "{{.WinRMPassword}}",
            "script": "init.ps1"
        }
    ]
    ...

Other than a PowerShell provisioner, we're using a series of file provisioners.

  • ssh-key.pub is a generated by our build.sh
  • SqlConfigurationFile.ini is the configuration file that we generated from running the installation process manually
  • sshd_config is a basic sshd configuration file that can be found here.

Our PowerShell provisioner is looking to run an init.ps1 script with administrator privileges.

Let's start to flesh out our init.ps1 script.

Ensure that the script stops on an error, hides progress status reporting and supports HTTPS connections:

$ErrorActionPreference = "stop"
$ProgressPreference = 'SilentlyContinue'
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
init.ps1

Next we will define a function for unzipping zip files:

Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
    param([string]$zipfile, [string]$outpath)

    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
init.ps1 (continued)

Create a WebClient for downloading files.

Side note: Invoke-WebRequest is a built-in cmdlet for performing web requests, but as the request is buffered into memory before being flushed to disk it can be (a) very slow for large files, (b) cause memory issues. Hence the choice of WebClient.

$WebClient = New-Object System.Net.WebClient
init.ps1 (continued)

Download and install OpenSSH:

# Install OpenSSH
Write-Host "SSH: Installing..."
Invoke-WebRequest -Uri "https://github.com/PowerShell/Win32-OpenSSH/releases/download/v8.1.0.0p1-Beta/OpenSSH-Win64.zip" -OutFile "C:\OpenSSH-Win64.zip"
Unzip "C:\OpenSSH-Win64.zip" "C:\OpenSSH-Win64\OpenSSH"
Move-Item -Path "C:\OpenSSH-Win64\OpenSSH\*" -Destination "C:\Program Files\OpenSSH"
Set-Location -Path "C:\Program Files\OpenSSH"
powershell -File .\install-sshd.ps1
.\ssh-keygen.exe -A
New-NetFirewallRule -Protocol TCP -LocalPort 22 -Direction Inbound -Action Allow -DisplayName SSH
Start-Service sshd
Set-Service -Name sshd -StartupType 'Automatic'
Move-Item -Path "C:\\Users\\Administrator\\sshd_config" -Destination "C:\\ProgramData\\ssh\\sshd_config" -Force
Write-Host "SSH: Installed"
init.ps1 (continued)

Install IIS and the .Net Core 3.1 hosting bundle. This will also setup a new website in IIS.

# Install IIS
Write-Host "IIS: Installing..."
Install-WindowsFeature -Name Web-Server, Web-Mgmt-Tools
Write-Host "IIS: Installed."

# Install .Net 3.1
Write-Host "Hosting Bundle: Installing..."
$WebClient.DownloadFile("https://download.visualstudio.microsoft.com/download/pr/fa3f472e-f47f-4ef5-8242-d3438dd59b42/9b2d9d4eecb33fe98060fd2a2cb01dcd/dotnet-hosting-3.1.0-win.exe","C:\dotnet-hosting-3.1.0-win.exe")
$args = New-Object -TypeName System.Collections.Generic.List[System.String]
$args.Add("/quiet")
$args.Add("/norestart")
Start-Process -FilePath "C:\dotnet-hosting-3.1.0-win.exe" -ArgumentList $args -NoNewWindow -Wait -PassThru
New-WebSite -Name "Default Web Site" -Port 80 -HostHeader mywebsite.com -PhysicalPath "C:\inetpub\wwwroot\" -Force
Write-Host "Hosting Bundle: Installed"
init.ps1 (continued)

Now for SQL Server 2012. Download and install SQL Server 2012 using the configuration file that we generated:

# Install SQL Server Express 2012
Write-Host "SQL Server Express 2012: Installing"
$WebClient.DownloadFile("https://download.microsoft.com/download/8/D/D/8DD7BDBA-CEF7-4D8E-8C16-D9F69527F909/ENU/x64/SQLEXPRADV_x64_ENU.exe", "C:\SQLEXPRADV_x64_ENU.exe")
(Get-Content "C:\SqlConfigurationFile.ini").replace('WIN-HOSTNAME', [System.Net.Dns]::GetHostName()) | Set-Content "C:\SqlConfigurationFile.ini"
$args = New-Object -TypeName System.Collections.Generic.List[System.String]
$args.Add("/ConfigurationFile=""C:\SqlConfigurationFile.ini""")
Start-Process -FilePath "C:\SQLEXPRADV_x64_ENU.exe" -ArgumentList $args -NoNewWindow -Wait -PassThru
Write-Host "SQL Server Express 2012: Installed"
init.ps1 (continued)

Finally, we can set the Windows Administrator password:

# Set admin password
Write-Host "Admin password: setting"
net user Administrator $Env:ADMIN_PASSWORD
Write-Host "init.ps1 completed!"
init.ps1 (continued)

There we go, it's time to run ./build.sh

Automation is the best documentation

Figuring out how to reproduce each installation step in Packer can be a time consuming process to get it to the point where everything works. However, the end result is a build process that can be run from scratch that will result in a fully configured server image.

This is a highly repeatable process that can be performed at the start of the project and re-built throughout the project to ensure that there is no configuration drift or additional software dependency that has been installed on the server manually.

And when time comes to throw the project over to production your Packer provisioning steps provide the perfect documentation for everything that is needed for the project to run.

All of the code in this blog post is available as a ready-to-run project on GitHub: legacy-win-server-image