Securely Access Private Git Repositories and Composer Packages in Docker Builds

When working on enterprise projects, it’s common to have to download private dependencies that require authentication to be installed (usually, internal or paid packages).

In modern setups, you’ll most likely use Docker to package your application (or service) and all its dependencies into a standalone image. Typically, building the Docker image is automated through a continuous deployment system.

Do Not Leak Your Secret Keys!

Most forges, including the popular GitHub and GitLab services, provide at least two ways to access private repositories: using HTTP authentication with personal access tokens or using SSH.

Most package managers support authentication by these means. It’s the case of virtually all package managers, including but not limited to NPM (JavaScript), Go Modules, pip (Python), and Composer (PHP). In this article, we’ll use Composer, but the discussed method also works with other packager managers.

Composer supports (among others) the following popular methods:

• Storing the personal access tokens provided by the forge in a file named auth.json
• Storing them in an environment variable named COMPOSER_AUTH
• Relying on SSH authentication

A good security practice is to prevent people who can access your Docker images from also having access to these secrets, and thus to the private repositories they grant access to. This means that the secrets should be used to download dependencies, but should not be accessible to people who have access to the Docker image or to the Git repository containing the source code.

Unfortunately, this is quite difficult to achieve with Docker. For performance reasons, Docker caches the result of each build step in what are called layers.

Anyone with access to an image can inspect all the layers created during the build process. Exploring the content of layers is very easy with tools like dive. That means that if you copy your tokens (e.g. the auth.json file for Composer) to download your private dependencies, then remove them from the image, they will still be accessible to anyone having access to the final image, through the layers!

Relying on environment variables or build args is not a better option: their values are also accessible to anyone who has access to the image!

Docker Build Secret Information

Docker 18.09 (2018) and Docker Compose 2.7.0 (2022) have introduced a new feature to solve this common problem: build secrets.

Build secrets allow files containing secrets to be mounted at build time, and guarantee that the content of these files will not be accessible in the final image.

They have also added a new subsystem designed specifically for downloading private dependencies: the ability to use the host system’s SSH access during builds.

This is very convenient because SSH has always been the preferred way to download private Git repositories, and is of course supported out of the box by the popular version control system.

Cloning a Private Git Repository Thanks to SSH Forwarding

Let’s create a Docker image containing a private Git repository but not the credentials needed to download it!

First, make sure that you’re able to download your private repository from your host:

• GitHub instructions
• GitLab instructions

Then, ensure that your private SSH key is added to the SSH agent (Docker will connect to it) by running:

ssh-add

You should now be able to build an image that contains your private repository:

# syntax=docker/dockerfile:1
FROM alpine

# OpenSSH client and Git are required dependencies to clone the repository
RUN apk add --no-cache openssh-client git

# Add the SSH public keys of the Git server to the known hosts (here, github.com)
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

# Clone the repository, notice the "--mount=type=ssh" option, which allow this instruction to use SSH forwarding
RUN --mount=type=ssh git clone --depth=1 '[email protected]:dunglas/my-private-repo.git'

Finally, hint Docker that it needs to connect to the default SSH agent of the host:

docker build --ssh default

Tadam! You have cloned a private Git repo without disclosing the secret key!

Downloading Private Composer Packages

Under the hood, Composer uses Git to download private repositories (it can also use other protocols such as HTTP, we’ll come back to that in the next section). Applying the same method we’ve seen previously is straightforward.

Here is a composer.json file referencing a private package stored on GitHub thanks to the repositories option:

{
  "require": {
    "dunglas/my-private-package": "dev-main"
  },
  "repositories": [
    {
      "type": "vcs",
      "url":  "[email protected]:dunglas/my-private-package.git"
    }
  ]
}

By the way, if you rely on private packages, consider subscribing to Private Packagist, a nice service created by the Composer team specially dedicated for this use case.

To copy this private repository into the vendor/ directory of the Docker image, use the following Dockerfile:

# syntax=docker/dockerfile:1
FROM php:alpine

RUN apk add --no-cache openssh-client git
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

# Install Composer
COPY --from=composer /usr/bin/composer /usr/bin/composer

# Copy the composer.json and composer.lock files
COPY composer.* .
# Install the dependencies and allow Docker to use the SSH credentials of the host
RUN --mount=type=ssh composer install --no-progress --no-interaction

If you already use the Symfony Docker skeleton or API Platform, you can adapt the provided Dockerfiles: add the required packages, add the SSH keys of your forge, and add `–mount=type=ssh` to the existing RUN instruction doing a composer install.

This method will also work with the package managers of other languages such as NPM.

Docker Compose

Docker Compose now natively supports SSH forwarding! Use the new --ssh default flag to let the builder use the SSH connection of the host:

docker compose build --ssh default

GitHub Actions and GitLab CI

Finally, you’ll probably want to build the Docker image directly into your CD pipeline.

To achieve this, the first step is to create deployment keys for your private Git repositories. Deploy keys are special SSH keys that grant read-only access to Git repositories.

First, generate a pair of public and private keys for each private repository. Then upload the public key on the forge and associate it with the private repository:

• Instructions for GitHub
• Instructions for GitLab

After that, store the private key as a CI secret:

• Instructions for GitHub
• Instructions for GitLab

Finally, each time you start a job building Docker images, copy the private key from the secret variable to SSH and start an SSH agent. To do so, I strongly recommend using the ssh-agent GitHub Action, which automates this process:

# .github/workflows/build-image.yml
jobs:
    build:
        steps:
            - uses: actions/checkout@v2

            - uses: webfactory/[email protected]
              with:
                  ssh-private-key: ${{ secrets.MY_PRIVATE_DEPLOY_KEY }}

            - run: docker compose build --ssh default

A Better Method for Composer

Ben Davies and Grzegorz Korba mentioned an even better method specifically tailored for Composer in the Twitter discussion that followed the initial publication of this article:

Very good, we mount a secret auth.json (https://t.co/amy1Yhft8X) file instead, which contains oauth tokens:
`–mount=type=secret,id=composer.auth,target=/build/auth.json`

— Ben Davies (@benjamindavies) August 9, 2022

The main idea is to generate a local auth.json file on the host and mount it as a secret when building the Docker image. With this method, if you use GitHub or GitLab, Composer will download Zip archives of your private packages over HTTPS, instead of using Git. As a result, it is no longer necessary to have the git and ssh commands installed in the container (the Docker image will be slightly smaller, and the attack surface reduced), and the packages will be faster to download.

However, because the auth.json file contains sensitive credentials, extra care must be taken to be sure to not leak it.

First, let’s create our composer.json file:

{
  "require": {
    "dunglas/private-package": "dev-main"
  },
  "repositories": [
    {
      "type": "vcs",
      "url": "https://github.com/dunglas/private-package.git"
    }
  ]
}

You may have noticed that we now use an HTTPS URL instead of a Git URL, the two can be used interchangeably but I prefer to make clear that I want to use HTTPS to download the package here.

Now, add auth.json to the .gitignore file to be sure to never store it in the Git repository:

# ...
vendor/
auth.json

Similarly, add it to the .dockerignore file, so it will never be copied into the Docker image:

# ...
vendor/
auth.json

Now that we have taken the necessary precautions to not leak the credentials it will contain, let’s generate the auth.json. Example with a GitHub token:

composer config --auth github-oauth.github.com ghp_<your_token>

GitLab and BitBucket tokens as well as plain old HTTP Basic authentication are also supported.

Run composer install to create a composer.lock file, and to add it in the Git repository. Thanks to the lock file, you’ll not need Git nor SSH in the image deployed in production (you may still need them in your development image, or on the host, to run composer update).

And here is the Dockerfile for a production-ready image:

# syntax=docker/dockerfile:1
FROM php:alpine

WORKDIR /srv/app

# Install Composer
COPY --from=composer /usr/bin/composer /usr/bin/composer

# Copy the composer.json and composer.lock files
COPY composer.* .

# Install the prod dependencies by allowing Docker to use the auth.json file of the host
RUN --mount=type=secret,id=composer_auth,dst=/srv/app/auth.json composer install --no-dev --no-scripts --no-autoloader --no-progress --no-interaction

# auth.json must not be copied into the final image
COPY . .

# Dump the autoloader
RUN composer dump-autoload --classmap-authoritative --no-dev

We use several tricks here:

1. auth.json is only mounted during the installation of the packages
2. composer.json and composer.lock are copied before the rest of the source code to prevent reinstalling the dependencies each time a file is modified
3. Development dependencies aren’t installed
4. We generate an optimized autoloader containing the list of all classes of the application after copying the entire project

To build this, image run: docker build --secret id=composer_auth,src=auth.json .

Docker Compose allows automating the mount of the local auth.json:

version: '3.9'

services:
  php:
    build:
      context: .
      secrets:
        - composer_auth

secrets:
  composer_auth:
    file: ./auth.json

To build using Docker Compose, just execute the usual command: docker compose build

For clarity, I created a GitHub project containing a working example.

Conclusion

You should now be able to safely build Docker images containing private code!

If you need help setting up fast and secure CI/CD pipelines, optimizing your Docker images, or creating platforms on top of Kubernetes, don’t hesitate to contact the great DevOps team at Les-Tilleuls.coop!

If you liked this post, consider sponsoring me on GitHub.