Container images are incorrectly read during CI since 2019-04-16

After 2019-04-16 gnutls CI jobs started failing with a very strange message:

./bootstrap: line 728: /builds/common/gnulib/gnulib-tool: No such file or directory

E.g., https://gitlab.com/gnutls/gnutls/-/jobs/197534342

This uses the image registry.gitlab.com/gnutls/build-images:buildenv-f29 which contains that file. The day before all CI jobs were run fine, and were able to read that missing file above.

The containers are generated in gitlab.com (as seen by the registry), and a regeneration of them did not address the issue. Even after regeneration the error persists.

By downloading the registry image I can verify that the reported as missing file is present. There seems to be an issue in how gitlab CI runners are reading the containers.

This also affects the wget project, with exactly the same error:

(btw. clearing the CI runner cache had no effect)

Neither clearing the pipeline caches nor re-creating the docker CI images makes a difference.

This also affects libidn2:
https://gitlab.com/libidn/libidn2/pipelines/57195291

The docker CI image are built like

WORKDIR /builds/common
RUN git clone https://git.savannah.gnu.org/git/gnulib.git

Using the resulting image from a CI pipeline reveals that /builds/common is simply empty - but only within Gitlab CI. As @nmav says, that issue came all of a sudden yesterday. Downloading the image from the Gitlab repository shows that it contains the expected /builds/common/gnulib directory.

Hello folks, just writing it down here as well so it’s visible to everyone in here https://gitlab.com/gitlab-org/gitlab-runner/issues/4174#note_161994927