CI Performance: Multi-stage caching

Aug 31

For this guide we'll take a plain NPM & Docker-based project and make it fast. Its CI consists of npm run build, npm run test, and docker build. We'll implement the following capabilities into the CI:

☑️ Changes to dependencies rerun everything.
☑️ Changes to source code skips dependency resolution.
☑️ Non-code changes skip everything.
☑️ No changes at all also skips everything.

Let's take a look at how we can solve this with just Docker in a way that can be easily extended by developers to further skip or reuse existing results of tests, linters, scans and so on. Or if you'd like; Skip this guide, and check out the reference implementation on GitHub.

Abstract

Implementing caching in CI saves developers valuable time. A faster build is a better developer experience. Manually writing caching logic can become complex and difficult to maintain. CI has to be reliable, and on top of that, dynamic enough to where developers can choose their own build tools. Does a common solution exist that fits on your existing projects and infrastructure? We think so. This guide will teach you how to apply Docker multi-stage builds and its built-in caching to make most CI processes significantly more performant for a variety of scenarios.

Project setup

For reference, our project is a create-react-app template and can be built using the following commands:

#!/usr/bin/env bash
npm install
npm run build
npm run test -- --watchAll=false
docker build . --tag npm-demo

# Dockerfile
FROM nginx:1.25.2
COPY build /usr/share/nginx/html
ENTRYPOINT ["nginx", "-g", "daemon off;"]

The result is an nginx-based Docker image containing all static resources created by NPM.

Docker multi-stage migration

We can apply Docker's powerful caching capabilities to our project's CI by migrating our bash script to an intermediate stage using Docker multi-stage build. With multi-stage builds we can (quoting Docker) "selectively copy artifacts from one stage to another, leaving behind everything you don't want in the final image.". Meaning we can run our NPM commands in one stage and copy over static files in our Nginx image.

# Dockerfile
FROM node:20-alpine3.17 as ci_step_build
WORKDIR /app
COPY . /app/
RUN npm install
RUN npm run build
RUN npm run test -- --watchAll=false

FROM nginx:1.25.2
COPY --from=ci_step_build /app/build /usr/share/nginx/html
ENTRYPOINT ["nginx", "-g", "daemon off;"]

The result is still an nginx-based Docker image containing all static resources created by NPM. This rudimentary setup has some major benefits:

Our CI is now docker build . Any machine with Docker on it can build it.
Docker maintains a local cache; running it a second time is significantly faster.

However, it's a bit on slow as has to Docker copy over 400MB worth of NPM packages. Let's fix this and optimize the Dockerfile further. In the meantime, we can check off some of our goals;

✅ Changes to dependencies rerun everything.
☑️ Changes to source code skips dependency resolution.
☑️ Non-code changes skip everything.
✅ No changes skip everything.

Docker multi-stage optimization

To achieve our other goals in just Docker, we'll have to be a bit more specific in how we build. Docker needs to be able to see which files are the same so it can apply already-built (cached) layers.

We can change our ci_step_build target to this:

# Dockerfile
FROM node:20-alpine3.17 as ci_step_build
WORKDIR /app

COPY package.json /app/package.json
COPY package-lock.json /app/package-lock.json
RUN npm install

COPY src/ /app/src/
COPY public/ /app/public/
RUN npm run build
RUN npm run test -- --watchAll=false

FROM nginx:1.25.2
COPY --from=ci_step_build /app/build /usr/share/nginx/html
ENTRYPOINT ["nginx", "-g", "daemon off;"]

A bit more complex, but this should a recognizable pattern for developers familiar with Docker. We copy in only the dependencies of each command and then run only that command, that way when src/ has changes, everything up to COPY src/ /app/src can be reused. Try it out on our reference imeplementation here.

Goal check;

✅ Changes to dependencies rerun everything.
✅ Changes to source code skips dependency resolution.
✅ Non-code changes skip everything.
✅ No changes skip everything.

Results

To get a rough idea of how much we gained let's take a look at the performance with time docker build ..

Change	Time (Rounded down)
On package.json	20s
On src/ files	9s
On README.md	1s
(No change)	1s

Reducing our CI time by 95% for non-code changes is great, although it might not be that impressive for this project given that it builds very little. However, we've encountered a large amount of projects that take over an hour to build at clients. In those cases, performance gains such as being able to intelligently skip a build or tests will be much more impactful and can completely change the way your developers work.

For the scope of this guide, this is where we'll stop optimizing. Further steps could be separating stages to run in parallel, creating custom base images with tool-specific caches, or just using smaller size-optimized images in general.

Thanks for reading! — Thinking big about CI performance? Don't hesitate to contact us.

Notes

How do I make it work on GitHub Actions?

In most standard installations Docker performs caching by default. If you're running CI on one giant Jenkins node for your team or just a solo developer with your local machine, this will work without any additional configuration. Keep in mind that you'll have to prune these layers from time to time.

For our demos the CI platform of choice is GitHub Actions. GitHub Actions' servers are ephemeral meaning Docker caches will be gone when our CI jobs end. Instead we'll have to ask Docker to treat a directory as an external cache. We can use GitHub's actions/cache to keep it this directory persisted between runs. The equivalent of docker build . caching to and from a local directory for our GitHub Actions looks like this:

docker buildx create --use --driver=docker-container
docker buildx build \
  --cache-from=type=local,src="./docker-cache-directory" \
  --cache-to=type=local,dest="./docker-cache-directory" \
  .

You can use this mechanism to customize Docker's caching to your preferences. To keep this guide Docker-focused we'll skip the details on our GitHub Actions workflow. Find the reference implementation here. The result should match the performance of the build shown:

Alternatives:

The build-push-action by Docker. Do note that this may get convoluted when you have more than one build target.
The GitHub Actions cache backend for Docker. Do note that this creates a large amount of cache entries which in our experience is not as easy to manage let alone as fast as one entry.

Can I use this to build artifacts other than Docker images?

Absolutely, add a stage to your Dockerfile like this:

FROM scratch as ci_step_artifact
COPY --from=ci_step_build /path/to/artifact /

Next, instruct Docker to build this target and output the contents of the image to a local directory:

docker build --target ci_step_artifact --output ./path/to/output .

Philipp Gayret