I’ve been running a pipeline on a monthly schedule for a long time without any issues on GitLab.com’s shared runners. This pipeline keeps a number of Devuan Docker base images up-to-date in the face of security upgrades. The jobs rely on a docker:dind service and basically run debootstrap inside a container to create the tarball for the base images.
The last scheduled run on 2021-08-03 however failed on a mknod command that has been succeeding until then. The last successful run was on 2021-07-03. There have been no changes on my end between those two runs.
I’ve created an issue on my project with links to the various logs and additional information on what I’ve discovered and tried so far. Basically though,
docker run --rm --cap-add SYS_ADMIN --volume $PWD:/devuan --workdir /devuan $IMAGE ...
started to fail out of the blue with
/usr/sbin/debootstrap: 1456: /usr/sbin/debootstrap: cannot create /devuan/rootfs-jessie/test-dev-null: Permission denied
E: Cannot install into target '/devuan/rootfs-jessie' mounted with noexec or nodev
I’ve tried adding --cap-add MKNOD (which is normally enabled by default) and even --privileged to that command-line but to no avail. The error persists.
The command that triggers the error is mknod /devuan/rootfs-jessie/test-dev-null c 1 3. This is used to check that the target file system is sane.
For the record, the same docker command works without issues locally.
What gives? How do I make my pipelines pass again?