Solving the x509 problem when a docker-machine VirtualBox VM tries to access Gitlab
Problem:
Gitlab-runner created Virtualbox VM’s give an x509 error in Gitlab of
"Error response from daemon: Get https://gitlab.paradigm.local:5005/v2/: x509: certificate signed by unknown authority"
when gitlab is part of an internal AD domain with an internal PKI using an internal certification authority.
The error throws when a login attempt is made to the Gitlab registry by a runner VM using boot2docker.
Because of where this occurs in the ci process it cannot be an issue with the image subsequently used by the runner to run the ci job - a fact which was masked by originally using a standard image from docker hub which required no authentication to be pulled. (I created an internally certified node image and tried to pull that from the internal registry which lead me to twig to the real problem)
Issue:
As a result of the above the issue must be with boot2docker - a supposition borne out by the fact that a manual insertion of the CA.crt into the VM solved the problem.
What is needed therefore is a mechanism to either
a. automatically insert the CA.crt file at VM spool up time (not possible from everything I have read)
OR
b. modify the boot2docker.iso file used when docker-machine spools up the VMs on gitlab-runner’s request (possible and successfully done)
First attempt at a solution:
Get the repo for boot2docker
git clone https://github.com/boot2docker/boot2docker
Open the Dockerfile in the cloned repo and copy the following code into it at line 306 (just before the first make command in the original Dockerfile)
ENV REGISTRY <registry instance>
ENV DOMAIN <registry domain>
ENV PORT <registry port>
ENV ROOTFS /rootfs
# Install CA certificate
RUN mkdir -p $ROOTFS/etc/docker/certs.d/${REGISTRY}:${PORT} ;\
mkdir -p $ROOTFS/etc/docker/certs.d/${REGISTRY}.${DOMAIN}:${PORT}
COPY CA.crt $ROOTFS/etc/docker/certs.d/${REGISTRY}:${PORT}/CA.crt
COPY CA.crt $ROOTFS/etc/docker/certs.d/${REGISTRY}.${DOMAIN}:${PORT}/CA.crt
RUN mkdir -p $ROOTFS/usr/local/share/ca-certificates/${DOMAIN}
COPY CA.crt $ROOTFS/usr/local/share/ca-certificates/${DOMAIN}/CA.crt
RUN echo "cat /usr/local/share/ca-certificates/${DOMAIN}/CA.crt >> /usr/local/etc/ssl/certs/ca-certificates.crt" >> $ROOTFS/etc/init.d/rcS
Copy the CA.crt to the root of the cloned directory so that the copy commands above can access it
Then build the file
This takes some time and I discovered it builds with docker v19.03.0-rc3 (my version of the repo anyway).
This rc version actually causes a problem in that release candidates and beta releases cause docker-machine to believe the newly created boot2docker.iso file is out-of-date and when it is spooling up VMs for gitlab-runner it overwrites the carefully created and certified iso image with a downloaded one and all that hard work is negated.
Go to line 446 in the Dockerfile and replace the version environment variable found there with the latest release version number - found at
Take careful note of the exact number sequence including periods and then replace the number at line 446
NB this means that every release will require this solution to be rebuilt. Cache makes it pretty fast though - just need to keep an eye out for version bumps
Then
docker build .
The outcome of the build is the creation of a new, unnamed image in the local docker images registry. Within that image a boot2docker.iso file has been created at /tmp/boot2docker.iso
Run the image
docker run -d <image_id>
Copy the iso file from the running container (the container id is the first 8 characters of the id printed from the previous step)
docker cp <container_id>:/tmp/boot2docker.iso /path/to_copy_to/on_runner_machine
gitlab-runner is assumed to be running as root on the runner machine. The new boot2docker.iso needs to replace the old one in the .docker directory under root’s home
In my case this old iso file was at
/root/.docker/machine/cache/boot2docker.iso
su to root to gain access to the file
su
<password>
Then
cd /root/.docker/machine/cache/
mv boot2docker.iso boot2docker.orig
Note that the boot2docker.iso file was 42Mb or thereabouts and the new one was 48Mb (helps to distinguish between the two)
Then copy the new iso into the cache directory
cp /path/to_copy_to/on_runner_machine/boot2docker.iso /root/.docker/machine/cache/boot2docker.iso
Remove any existing docker machines and let the gitlab-runner spool up new ones using the new iso
Voila, x509 problems gone!