Environment Variable Templates

Inspiration

I’ve started to deploy CoreOs clair to analyze the security of my docker containers. This service is distributed as a Docker Container, which will allow me to easily deploy it into my stack, but it doesn’t support my platform’s configuration model. Clair stores its config in YAML files and (at the time of this writing) doesn’t support loading its config from the environment. I’m going to show a way to make dockerized services like this behave well with Twelve-Factor Section III (storing application configuration in Environment Variables.)

I’ve found a few examples using these techniques to inject values from environment variables into templatized config files:

They all follow the same model: rewrite application configuration before starting the service. Docker makes this easy by letting us wrap third-party containers with our own config templates and custom entrypoints.

Environment Variable Templating

Here’s a sample clair config file:

clair:
  database:
    options:
      source: postgresql://user@password:host:port/dbname

I’ll describe a few ways to populate that static config from your runtime environment variables:

For the Unix Neckbeards: sed (stream editor)

Sed is pre-installed on absolutely everything, including Alpine (as part of busybox). It’s been used for this very purpose for decades. It’s great at replacing strings in files, which covers most of your templating tasks. Use it when you can’t be bothered to install anything better. Here’s a template where we delimit environment variables with percent signs:

clair:
  database:
    options:
      source: "%%%CLAIR_DATABASE%%%"

You could use sed to replace that string with the value of an environment variable matching $CLAIR_DATABASE in multiple ways. To replace a single value in a file (repeat as necessary):

sed -i "s/%%%CLAIR_DATABASE%%%/${CLAIR_DATABASE}/g" config.template.yaml

Or to replace all strings matching environment variable keys:

template = "config.template.yaml"

# Read env as key/val variables
while IFS='=' read -r key val; do
  # Replace %%%key%%% with val in template
  if grep -q "%%%${key}%%%" $template; then
    sed -i "s#%%%${key}%%%#${val}#g" $template
  fi
done < <(env)

You can fake bash-style variable substitution using sed. This would replace ${ENV_VAR}:

sed -i 's/^\${'"$key"'}/'"$val"'/g' config.template.yaml

Don’t forget to clean up any template placeholders that weren’t populated from the environment:

sed -i 's/%%%[^%]*%%%##g' config.template.yaml
sed -i 's/^\${\([^}]*\)}//g' config.template.yaml

For the Hipster Hackers: envsubst

I’ve been using *nix for 20 years and have never heard of envsubst until I started writing this blog post. It’s provided by gettext, which isn’t in Alpine by default, but only adds around 17MB to install. It evaluates and replaces strings that look like bash variables ($VAR and ${VAR}) from the environment. Use envsubst when you want basic bash-style variable expansion:

(~)$ echo $OMG
HAX
(~)$ echo '$OMG'
$OMG
(~)$ echo 'OMG: ${OMG}' | envsubst
OMG: HAX

With envsubst your config.template.yaml can look a lot more like you’d expect:

clair:
  database:
    options:
      source: postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}/$DB_NAME

When invoked with a properly set up environment:

(~)$ cat config.template.yaml
clair:
  database:
    options:
      source: postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}/$DB_NAME
(~)$ envsubst < config.template.yaml | tee config.yaml
clair:
  database:
    options:
      source: postgresql://postgres:p4ssw0rd@pgdb01/clair

But what if your environment isn’t properly set up? What about those nice bashisms like default values when variables are missing (e.g. ${VAR:-default})?

For the Lulz: bash eval cat

Here’s where I think things start getting fun (thanks, plockc!) I can’t really recommend this approach, I mean it screams of security issues, but it’s certainly the most extensible. This requires proper bash (not ash), which adds about 18mb to alpine:3.4, but is included pretty much everywhere else:

eval "cat <<EOF
$(<config.template.yaml)
EOF
" | tee config.yaml 2> /dev/null

Here’s our new template:

# Template generated: $(date)
clair:
  database:
    options:
      source: ${DB_SCHEME:-postgresql}://${DB_USER}:${DB_PASS}@${DB_HOST}/$DB_NAME

This allows you to get really creative, not just using environment variables, but all forms of shell expansion including default values and command substitution:

# Template generated: Thu Aug 18 02:48:23 UTC 2016
clair:
  database:
    options:
      source: postgresql://postgres:p4ssw0rd@pgdb01/clair

For Everyone Else: dockerize

I wrote this whole post to show off some nifty sed magic. Along the way I found envsubst and the bash eval cat method described above. That doesn’t mean any of those are the right tools for the job. That would be dockerize. It’s a static binary that gives you Golang text/template for 7.6MB. Our example template rewritten for dockerize:

clair:
  database:
    options:
      source: {{ default .Env.DB_SCHEME "postgresql" }}://{{ .Env.DB_USER }}:{{ .Env.DB_PASS }}@{{ .Env.DB_HOST }}/{{ .Env.DB_NAME }}

Running dockerize:

(~)$ ./dockerize -template config.template.yaml:config.yaml cat config.yaml
clair:
  database:
    options:
      source: postgresql://postgres:p4ssw0rd@pgdb01/clair

Docker Wrappers

Now that you’ve decided on a template format and tool you need to create your docker wrapper. The clair Dockerfile ends with a simple ENTRYPOINT ["clair"] statement. We’ll evaluate our template then invoke that command ourselves with a custom entrypoint.sh:

#!/bin/bash

eval "cat <<EOF
$(</config/config.template.yaml)
EOF
" | tee /config/config.yaml 2> /dev/null

exec "$@"

Add it all together with a custom Dockerfile:

FROM quay.io/coreos/clair:v1.2.2

ADD entrypoint.sh /
ADD config.template.yaml /config/

ENTRYPOINT ["/entrypoint.sh", "clair"]

But really, I’m not even kidding, use dockerize:

FROM quay.io/coreos/clair:v1.2.2

RUN curl -sL https://github.com/jwilder/dockerize/releases/download/v0.2.0/dockerize-linux-amd64-v0.2.0.tar.gz \
    | tar zxf - -C /bin/
ADD config.template.yaml /config/

ENTRYPOINT ["dockerize", "-template", "/config/config.template.yaml:/config/config.yaml", "clair"]