Solving the x509 problem when a docker-machine VirtualBox VM tries to access Gitlab

Solving the x509 problem when a docker-machine VirtualBox VM tries to access Gitlab

Problem:
Gitlab-runner created Virtualbox VM’s give an x509 error in Gitlab of

"Error response from daemon: Get https://gitlab.paradigm.local:5005/v2/: x509: certificate signed by unknown authority"

when gitlab is part of an internal AD domain with an internal PKI using an internal certification authority.
The error throws when a login attempt is made to the Gitlab registry by a runner VM using boot2docker.

Because of where this occurs in the ci process it cannot be an issue with the image subsequently used by the runner to run the ci job - a fact which was masked by originally using a standard image from docker hub which required no authentication to be pulled. (I created an internally certified node image and tried to pull that from the internal registry which lead me to twig to the real problem)

Issue:
As a result of the above the issue must be with boot2docker - a supposition borne out by the fact that a manual insertion of the CA.crt into the VM solved the problem.

What is needed therefore is a mechanism to either

a. automatically insert the CA.crt file at VM spool up time (not possible from everything I have read)
OR
b. modify the boot2docker.iso file used when docker-machine spools up the VMs on gitlab-runner’s request (possible and successfully done)

First attempt at a solution:

Get the repo for boot2docker

    git clone https://github.com/boot2docker/boot2docker

Open the Dockerfile in the cloned repo and copy the following code into it at line 306 (just before the first make command in the original Dockerfile)

    ENV REGISTRY <registry instance>
    ENV DOMAIN <registry domain>
    ENV PORT <registry port>
    ENV ROOTFS /rootfs

    # Install CA certificate
    RUN mkdir -p $ROOTFS/etc/docker/certs.d/${REGISTRY}:${PORT} ;\
        mkdir -p $ROOTFS/etc/docker/certs.d/${REGISTRY}.${DOMAIN}:${PORT}
    COPY CA.crt $ROOTFS/etc/docker/certs.d/${REGISTRY}:${PORT}/CA.crt
    COPY CA.crt $ROOTFS/etc/docker/certs.d/${REGISTRY}.${DOMAIN}:${PORT}/CA.crt

    RUN mkdir -p $ROOTFS/usr/local/share/ca-certificates/${DOMAIN}
    COPY CA.crt $ROOTFS/usr/local/share/ca-certificates/${DOMAIN}/CA.crt
    RUN echo "cat /usr/local/share/ca-certificates/${DOMAIN}/CA.crt >> /usr/local/etc/ssl/certs/ca-certificates.crt" >> $ROOTFS/etc/init.d/rcS   

Copy the CA.crt to the root of the cloned directory so that the copy commands above can access it

Then build the file

This takes some time and I discovered it builds with docker v19.03.0-rc3 (my version of the repo anyway).

This rc version actually causes a problem in that release candidates and beta releases cause docker-machine to believe the newly created boot2docker.iso file is out-of-date and when it is spooling up VMs for gitlab-runner it overwrites the carefully created and certified iso image with a downloaded one and all that hard work is negated.

Go to line 446 in the Dockerfile and replace the version environment variable found there with the latest release version number - found at

Take careful note of the exact number sequence including periods and then replace the number at line 446

NB this means that every release will require this solution to be rebuilt. Cache makes it pretty fast though - just need to keep an eye out for version bumps

Then

      docker build .

The outcome of the build is the creation of a new, unnamed image in the local docker images registry. Within that image a boot2docker.iso file has been created at /tmp/boot2docker.iso

Run the image

    docker run -d <image_id>

Copy the iso file from the running container (the container id is the first 8 characters of the id printed from the previous step)

    docker cp <container_id>:/tmp/boot2docker.iso /path/to_copy_to/on_runner_machine

gitlab-runner is assumed to be running as root on the runner machine. The new boot2docker.iso needs to replace the old one in the .docker directory under root’s home

In my case this old iso file was at

    /root/.docker/machine/cache/boot2docker.iso

su to root to gain access to the file

    su
    <password>

Then

    cd /root/.docker/machine/cache/
    mv boot2docker.iso boot2docker.orig

Note that the boot2docker.iso file was 42Mb or thereabouts and the new one was 48Mb (helps to distinguish between the two)

Then copy the new iso into the cache directory

    cp /path/to_copy_to/on_runner_machine/boot2docker.iso /root/.docker/machine/cache/boot2docker.iso

Remove any existing docker machines and let the gitlab-runner spool up new ones using the new iso

Voila, x509 problems gone!

Second (better) solution:

I wasn’t happy with slightly clunky nature with version number changes and the like so here is a much better, tighter and less heath-robinson method using a simple dockerfile

Create this Dockerfile
(note the specific version tag which gets us around the versioning issue from the first attempt above)

    FROM boot2docker/boot2docker:18.09.7
    ENV REGISTRY <registry instance>
    ENV DOMAIN <registry domain>
    ENV PORT <registry port>
    ENV WORKDIR /rootfs

    # Install CA certificate
    RUN mkdir -p $WORKDIR/etc/docker/certs.d/${REGISTRY}:${PORT} ;\
    mkdir -p $WORKDIR/etc/docker/certs.d/${REGISTRY}.${DOMAIN}:${PORT}
    COPY CA.crt $WORKDIR/etc/docker/certs.d/${REGISTRY}:${PORT}/CA.crt
    COPY CA.crt $WORKDIR/etc/docker/certs.d/${REGISTRY}.${DOMAIN}:${PORT}/CA.crt

    RUN mkdir -p $WORKDIR/usr/local/share/ca-certificates/${DOMAIN}
    COPY CA.crt $WORKDIR/usr/local/share/ca-certificates/${DOMAIN}/CA.crt
    RUN echo "cat /usr/local/share/ca-certificates/${DOMAIN}/CA.crt >> /usr/local/etc/ssl/certs/ca-certificates.crt" >> $WORKDIR/etc/init.d/rcS
    
    # Run the make iso file embedded in the original image
    RUN time make-b2d-iso.sh; \
    du -hs /tmp/boot2docker.iso
    
    # This console prints the various tool versions used in the final iso
    CMD ["cat", "/tmp/boot2docker.iso"]

Add your certificate authority certificate to the same folder as the above Docker file

Run

docker build .

From here on follow the instructions from the first attempt for extraction of the iso and its placement for use by docker-machine.

Hope this is useful for someone.

Regards
Ian Carson