Docker Parameterized Builds Using Git Tags, Part 2 of 2

In Part 1 we constructed an automated image build on Docker Hub in which a package (Dropbox) can be updated using the git tag. In this post we’ll go further and use the tag to feed three package versions and dynamically create the Dockerfile at build time.

The complete repository can be found at https://bitbucket.org/double16/gradle-dockercompose.

Dynamic Dockerfile

The build script gives us full control of how the image is built. We can leverage this by building the Dockerfile in the build script, rather than using a Dockerfile checked into the repo. Why would we want to do this? If our tag requires a different base image, now we can do that. If the version of a package we want requires different install instructions, we can do that too and keep the complexity out of the Dockerfile.

We’ll look at a composition of Gradle + Docker + Docker Compose. Gradle is a multi-language build system with support (via a plugin) for building, publishing and running containers. We’d like a variety of images expressing the combinations of versions of these tools.

Defining the Git Tag

First we need to decide on a format for the git tag. We’ll be combining three package versions into the tag. Here’s what we want:
1. Gradle is our main package, we want it to be our base image, so we need to allow for tags with version numbers and variants, such as ‘4.0.1-alpine’, etc.
2. Docker CE with versions such as 17.06.2-ce
3. Docker Compose with versions such as 1.15.2

The git tag will be defined as {GRADLE_VERSION}_{DOCKER_VERSION}_{DOCKER_COMPOSE_VERSION}. For example:
* 4.0_17.05.0-ce_1.15.0 is Gradle image ‘4.0’, Docker ‘17.05.0-ce’ and Docker Compose ‘1.15.0’
* 4.0.1-jre8-alpine_17.04.0-ce_1.15.0 is Gradle image ‘4.0.1-jre8-alpine’, Docker ‘17.04.0’ and Docker Compose ‘1.14.0’

Parsing the Tag

Parsing the three versions out of the tag is easy using the bash command read. Read will take input and using the
IFS variable as a separator, set environment variables by splitting the input. We continue the pattern from part 1 and set our variables in hooks/env that is shared between the build and test scripts.

hooks/env

# These values are passed by the Hub, but if we are running locally we can get them from git.
[ -n "$SOURCE_BRANCH" ]  || SOURCE_BRANCH=$(git symbolic-ref -q --short HEAD)
[ -n "$GIT_SHA1" ]       || GIT_SHA1=$(git rev-parse -q HEAD)
 
# Parse arguments from source branch
IFS=_ read -r GRADLE_VER DOCKER_VER DOCKER_COMPOSE_VER <<EOF  # (1)
$SOURCE_BRANCH  # (2)
EOF
 
# If the tag isn't in the expected format, clean all values to get defaults
if [ -z "$GRADLE_VER" -o -z "$DOCKER_VER" -o -z "$DOCKER_COMPOSE_VER" ]; then
	GRADLE_VER=''
	DOCKER_VER=''
	DOCKER_COMPOSE_VER=''
fi
 
# Set defaults for build arguments
[ -n "$GRADLE_VER" ]         || GRADLE_VER=latest
[ -n "$DOCKER_VER" ]         || DOCKER_VER=17.05.0-ce
[ -n "$DOCKER_COMPOSE_VER" ] || DOCKER_COMPOSE_VER=1.15.0
[ -n "$SOURCE_TYPE" ]        || SOURCE_TYPE=git
[ -n "$DOCKERFILE_PATH" ]    || DOCKERFILE_PATH=.
 
# Special tag case for 'latest'
if [ "$GRADLE_VER" == 'latest' ]; then
  IMAGE_NAME=pdouble16/gradle-dockercompose:latest
elif [ -n "$IMAGE_NAME"]; then
  IMAGE_NAME=pdouble16/gradle-dockercompose:${GRADLE_VER}_${DOCKER_VER}_${DOCKER_COMPOSE_VER}  # (3)
fi

We temporarily set IFS to _ and use read to parse the input
We use a bash ‘here’ document to send the branch or tag to stdin
IMAGE_NAME is what we’ll use to tag the image and it must make the name of the Docker Hub automated build

Building

We now have our target versions in GRADLE_VER, DOCKER_VER, and DOCKER_COMPOSE_VER and are ready to construct the build. The script for this image is much more complex than our previous example. We need to piece together parts of a Dockerfile based on the versions for each component.

Base Image

We start by choosing a base image using GRADLE_VER. We are always using the gradle image, but we could choose ANY base image. This gives us a great deal of flexibility. (If we only need to specify an image tag, we could use a an ARG build tag before the FROM tag.)

hooks/build

#!/bin/bash -xe
. ./hooks/env
# Temp file for assembled Dockerfile
DOCKERFILE=./Dockerfile.jit.$GRADLE_VER.$DOCKER_VER.$DOCKER_COMPOSE_VER
 
echo "FROM gradle:$GRADLE_VER" > $DOCKERFILE  # (1)
cat Dockerfile.part.args >> $DOCKERFILE  # (2)
echo "USER root" >> $DOCKERFILE

Base image is the first line in our dynamic Dockerfile
These are common arguments used in the build

We could include the arguments in the hooks/build script, but let’s put them into a separate file to keep things cleaner. We also don’t need arguments for the package versions because we’re constructing the Dockerfile dynamically. However, the resulting Dockerfile will look like a regular docker when we use build arguments. This will help debugging and make the hooks/build script cleaner.

Dockerfile.part.args

ARG BUILD_DATE
ARG SOURCE_COMMIT
ARG DOCKERFILE_PATH
ARG SOURCE_TYPE
ARG GRADLE_VER
ARG DOCKER_VER
ARG DOCKER_COMPOSE_VER

Base Image Specifics

For the Gradle image, there are two base images it uses: Debian and Alpine. We need to do some different things based on that. We’ll use the bash case statement to choose between parts.

hooks/build

# Base image specifics for packages we require
case "$GRADLE_VER" in
  *alpine*) cat Dockerfile.part.alpine >> $DOCKERFILE ;;
  *)        cat Dockerfile.part.debian >> $DOCKERFILE ;;
esac

Alpine

Docker needs glibc, and Alpine is based on musl libc. We need to install glibc which makes the image noticably larger, but still smaller than using Debian. We could do a conditional check in a Dockerfile, but it will make it less readable.

Dockerfile.part.alpine

ENV GLIBC_VERSION=2.23-r3
RUN apk add --no-cache libstdc++ curl gzip && \
    for pkg in glibc-${GLIBC_VERSION} glibc-bin-${GLIBC_VERSION} glibc-i18n-${GLIBC_VERSION}; do curl -sSL https://github.com/andyshinn/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/${pkg}.apk -o /tmp/${pkg}.apk; done && \
    apk add --allow-untrusted /tmp/*.apk && \
    rm -v /tmp/*.apk && \
    ( /usr/glibc-compat/bin/localedef --force --inputfile POSIX --charmap UTF-8 C.UTF-8 || true ) && \
    echo "export LANG=C.UTF-8" > /etc/profile.d/locale.sh && \
    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc-compat/lib && \
    echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
    apk del glibc-i18n && \
    rm -rf /tmp/* /var/cache/apk/*

Debian

The debian based images need curl and gzip, so the part in this case is less complicated than the Alpine requirements.

Dockerfile.part.debian

RUN rm -rf /var/lib/apt/lists/* && apt-get -q update && apt-get install -qy --force-yes curl gzip && apt-get clean && rm -rf /var/lib/apt/lists/*

Common Parts

Here are some more common parts of the Dockerfile that are kept in separate files for clarity, like Dockerfile.part.args.

hooks/build

# Install Docker and Docker Compose, uses Dockerfile build arguments to choose versions
cat Dockerfile.part.docker >> $DOCKERFILE
 
# Move back to original user so we aren't running as root
echo "USER gradle" >> $DOCKERFILE
 
# Include labels last because the BUILD_DATE changes for each build
cat Dockerfile.part.labels >> $DOCKERFILE

Build It!

Now we’re ready to build it! We use the -f argument to docker build to specify our newly created Dockerfile. Docker will build and tag it as any other automated repository build.

hooks/build

# Build it
docker build \
	--build-arg "GRADLE_VER=$GRADLE_VER" \
	--build-arg "DOCKER_VER=$DOCKER_VER" \
	--build-arg "DOCKER_COMPOSE_VER=$DOCKER_COMPOSE_VER" \
	--build-arg BUILD_DATE=`date -u +"%Y-%m-%dT%H:%M:%SZ"` \
	--build-arg "SOURCE_COMMIT=$GIT_SHA1" \
	--build-arg "DOCKERFILE_PATH=$DOCKERFILE_PATH" \
	--build-arg "SOURCE_TYPE=$SOURCE_TYPE" \
	-t $IMAGE_NAME -f $DOCKERFILE .  # (1)

We specify our dynamic Dockerfile here

Testing

We are not getting complicated with testing here. We use each tool to get the version and verify we installed the correct one. Much more could be done here. It’s a shell script and we can leverage docker images of more sophisticated test frameworks. Note that the Dockerfile isn’t involved in testing, it is the image. The hooks/env file is important so that we construct the $IMAGE_NAME variable the same way for build and test.

hooks/test

#!/bin/bash -xe
. ./hooks/env
docker run $IMAGE_NAME docker version | grep $DOCKER_VER
docker run $IMAGE_NAME docker-compose version | grep $DOCKER_COMPOSE_VER
if [[ "${GRADLE_VER}" =~ ^[0-9][0-9.]*$ ]]; then
	docker run $IMAGE_NAME gradle --version | grep $GRADLE_VER
else
	docker run $IMAGE_NAME gradle --version
fi

Updating Tags

We could manually create a git tag for each image we wanted. However, we have three versions and that’s tedious. Let’s automate that too. We’ll write a shell script that creates the permutations of versions we want and outputs tags. Then we can pipe those in git tag like this:

$ ./update-tags.sh
4.2.1-jre9_17.06.2-ce_1.15.0
4.2.1-jre9_17.09.0-ce_1.16.1
4.2.1-jre9_17.09.0-ce_1.15.0
 
$ update-tags.sh | xargs -L 1 git tag && git push --tags

Specifying Tags

We can specify desired tags using an environment variable in the update-tags.sh script. Simple, but requires manual updating.

DOCKER_VERSION="17.06.1-ce 17.09.0-ce"

Query for Tags

If a list of versions is available, we can also query for them. In our example, the Gradle base image is stored on Docker Hub and has an HTTP endpoint to query for the image tags.

$ curl -s https://registry.hub.docker.com/v1/repositories/gradle/tags \  # (1)
  | jq -r .[].name  # (2)

Returns a JSON array of objects, the interesting property being name
jq is a JSON query utility, here we are using it to extract the name property, -r supresses double-quotes normally around JSON strings

With this query we can run a script at any given time to update our list of tags. If we could locate HTTP endpoints for Docker and Docker Compose tags we could do the same thing.

Permutations

We can use a for loop to create the permutations of tags. The complete script to generate new tags that the repo is missing follows.

update-tags.sh

GRADLE_TAGS="$(curl -s https://registry.hub.docker.com/v1/repositories/gradle/tags | jq -r .[].name)"
 
DOCKER_VERSIONS="17.06.2-ce 17.09.0-ce"
DOCKER_COMPOSE_VERSIONS="1.16.1 1.15.0"
 
DESIRED=$(mktemp tmp.XXXXXXXXXX)  # (1)
for GRADLE_TAG in $GRADLE_TAGS; do
  for DOCKER_VERSION in $DOCKER_VERSIONS; do
    for DOCKER_COMPOSE_VERSION in $DOCKER_COMPOSE_VERSIONS; do
      echo ${GRADLE_TAG}_${DOCKER_VERSION}_${DOCKER_COMPOSE_VERSION} >> ${DESIRED}
    done
  done
done
 
EXISTING=$(mktemp tmp.XXXXXXXXXX)  # (2)
git tag > ${EXISTING}
 
grep -F -x -v -f ${EXISTING} ${DESIRED}  # (3)
rm ${EXISTING} ${DESIRED}

Create a file containing a list of all desired tags
Create a file containing the existing git tags
This grep command uses the existing file as a list of fixed (non-regex) patterns and outputs all lines from the desired list that are not found in the existing list.

Finally, the get images built and tagged for new versions, we run:

$ update-tags.sh | xargs -L 1 git tag && git push --tags

Docker Hub will be busy!

Conclusion

We wanted a container image is a composition of multiple software packages. One of these packages we wanted as a base image. Since the base image varies, and the operating system on which it is based varies, we need something more complex. We solved our problem by dynamically constructing a Dockerfile based on versions encoded in the git tag. We can update our images using a single shell command.

Maintenance is now much easier. We can spend more time coding (or automating something else), rather than typing in git tag or git branch hundreds of times. We could even run the update-tags.sh script in a weekly cron tab and forget about it until Docker Hub tells us our build or test fails.

Enjoy!