Something Like Production

Test your AMIs with Docker

Iterative Development of Config Management

I’ve recently begun using Ansible and Packer to bake “Golden Image” AMIs for AWS. This makes deploying applications with Spinnaker a breeze, but testing those “BaseAMIs” is difficult. Packer is unforgiving of failed builds, unless it’s invoked with the fully-interactive --debug flag it instantly terminates failed instances (removing the option to debug a partially-converged filesystem).

Testing Ansible and Packer configs with Docker makes iteration much faster. I avoid the standard Packer overhead of launching an EC2 instance, “waiting for ssh”, shutting down, creating the AMI, and all the cleanup tasks. This can save several minutes right off the bat, and combined with Docker’s filesystem cache iteration becomes much less painful.

The biggest caveat to drop-in testing Configuration Management for AMIs against Docker containers is that they’re not exactly the same base installations. If I find an “ubuntu trusty” Packer source_ami from https://cloud-images.ubuntu.com/locator/, it’s not going to contain the same base installation as the image I get from docker pull ubuntu:trusty.

Turning an AMI into a Docker Image

Docker images are supposed to be lightweight, single-purpose runtimes, with as little bloat as possible. They don’t need their own installation of grub, or the linux kernel, or sshd or vim. All of these and more are included in the AMI distribution, and are managed by our Ansible Playbooks (e.g. cis-ubuntu-ansible)

To get a Docker Image that matches your source_ami, let’s start by finding that AMI’s snapshot-id. Open your AWS Console and search for the AMI you’d like to image. The Block Device Mapping for /dev/sda1 shows the EBS Snapshot containing the root filesystem for the AMI.

Find Snapshot ID

Find that snapshot in the AWS Console, right-click and select “Create Volume”. The defaults are probably fine here, just ensure it’s in an Availability Zone where you have an existing EC2 instance.

Create Volume Dialog

When the volume goes into state “Available”, you can right-click and select “Attach Volume” to assign it to an existing EC2 instance.

After the volume is attached to the EC2 instance you specified, SSH to the machine and mount the filesystem (/dev/sd* is renamed /dev/xvd* on ubuntu):

1
2
3
4
5
heph@evenpanther:~$ sudo mount /dev/xvdf1 /mnt/
heph@evenpanther:~$ df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdf1      7.8G  794M  6.6G  11% /mnt
heph@evenpanther:~$

From here, /mnt contains the same content as the root filesystem for the chosen AMI. You can turn this into a Docker Image with tar and docker import:

1
2
3
4
5
heph@evenpanther:~$ sudo tar zcpP -C /mnt/ . | docker import - ubuntu:ami-d732f0b7
heph@evenpanther:~$ docker images
REPOSITORY  TAG           IMAGE ID      CREATED        SIZE
ubuntu      ami-d732f0b7  a0eb6c872499  2 minutes ago  680.8 MB
heph@evenpanther:~$

You now have a docker container that looks exactly like the AMI image you started from. Notice it’s significantly larger than ubuntu:trusty from dockerhub:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
heph@evenpanther:~$ docker pull ubuntu:trusty
trusty: Pulling from library/ubuntu

96c6a1f3c3b0: Pull complete
ed40d4bcb313: Pull complete
b171f9dbc13b: Pull complete
ccfc4df4fbba: Pull complete
Digest: sha256:9274d908eb6d9a3784e93290fcc49f3c5618db9e1b0174ee27f9fc75aa3c0fb0
Status: Downloaded newer image for ubuntu:trusty
heph@evenpanther:~$ docker images
REPOSITORY  TAG           IMAGE ID      CREATED         SIZE
ubuntu      ami-d732f0b7  a0eb6c872499  10 minutes ago  680.8 MB
ubuntu      trusty        0ccb13bf1954  12 days ago     188 MB
heph@evenpanther:~$

Testing Configuration Management

Now that you have a representative base image to test, you may need to modify your Configuration Management to support execution inside a Docker container.

For ansible, I modified a few tasks to avoid execution in a Docker context, for example:

1
2
3
4
5
6
7
8
9
10
11
12
- name: Docker doesn't use grub
  shell: /usr/sbin/update-grub
  when: ansible_virtualization_type != 'docker'

- name: These packages shouldn't be added to Docker Containers
  apt: name="{{ item }}" state=installed
  with_items:
    - linux-image-extra-{{ansible_kernel}}
    - linux-image-extra-virtual
    - nfs-common
    - apparmor
  when: ansible_virtualization_type != 'docker'

I install a custom /etc/apt/preferences to prevent those those packages from being upgraded by apt-get upgrade.

1
2
3
4
5
6
7
Package: libpam-systemd
Pin: origin ""
Pin-Priority: -1

Package: grub-pc
Pin: origin ""
Pin-Priority: -1

Then I use a simple Dockerfile to test my setup, ansible, and cleanup stages:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
FROM ubuntu:ami-d732f0b7

# Prevent apt from trying to upgrade some packages that
# don't work inside a docker environment (grub, kernel, etc)
ADD bootstrap_files/docker/apt_preferences /etc/apt/preferences

# Setup - Installs Ansible and its dependencies
ADD bootstrap_files/setup.sh /tmp/
RUN /tmp/setup.sh

# Ansible - Test our base-ami site.yml
ADD packer/base-ami/site.yml /tmp/site.yml
ADD ansible/roles /etc/ansible/roles
RUN sudo ansible-playbook -v -M /tmp/ansible/ /tmp/site.yml

# Cleanup - Delete logs and cache files
ADD bootstrap_files/cleanup.sh /tmp/
RUN /tmp/cleanup.sh

CMD ["/bin/bash"]

From here I can validate my base-ami/site.yml with docker build. Any errors will cause the build to fail, but subsequent runs will skip any successful stages, allowing quick iteration.

1
2
3
4
5
6
7
8
9
10
heph@evenpanther:~/packer-factory$ docker build .
Sending build context to Docker daemon 548.9 kB
Step 1 : FROM ubuntu:ami-d732f0b7
...
Step 10 : CMD /bin/bash
 ---> Running in 18086e0c8c39
 ---> 73ce2146f996
Removing intermediate container 18086e0c8c39
Successfully built 73ce2146f996
heph@evenpanther:~/packer-factory$

It’s always a good idea to clean up when you’re done:

1
2
3
heph@evenpanther:~$ docker rmi 73ce2146f996
Deleted: sha256:73ce214...
heph@evenpanther:~$