Hello everyone 
as of yesterday, all of our CI / CD jobs are failing out of the blue. We did not change settings on the runner machine, nor on GitLab or any of our projects. Previously working pipelines all get stuck at the same point: we use docker-in-docker to build a docker container for deployment on Google Cloud Run. Every job fails after initializing an empty repo, so probably before pulling in the source code from GitLab:
Pulling docker image google/cloud-sdk ...
Using docker image sha256:3991d2b6213bee500d30c23c184d81f3ad26eb075e060d61dff69b5df6adfeea for google/cloud-sdk with digest google/cloud-sdk@sha256:de7eb24d1bc22cc7dc4ba4d1fedd0c2cf6a68418e5701668692b1d21293a8fe8 ...
Preparing environment
00:01
Running on runner-nbhwkxmq-project-35-concurrent-0 via gallus...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/datanauts/bannerfy-api/.git/
Created fresh repository.
This is my .gitlab-ci.yml:
# File: .gitlab-ci.yml
image: docker:latest
variables:
GIT_STRATEGY: clone
stages:
- deploy
deploy:
stage: deploy
image: google/cloud-sdk
services:
- docker:dind
script:
- echo "Deleting .gitignore"
- rm .gitignore
- cp $ENV_FILE .env
- cp $FIREBASE_ADMIN_JSON firebase-admin.json
- echo $GCP_SERVICE_KEY > gcloud-service-key.json # Google Cloud service accounts
- gcloud auth activate-service-account --key-file gcloud-service-key.json
- gcloud config set project $GCP_PROJECT_ID
- gcloud builds submit . --config=cloudbuild.yaml
only:
- release
Im in GitLab Version 14.8.2-ee and gitlab-runner:
gitlab-runner --version
Version: 14.8.2
Git revision: c6e7e194
Git branch: 14-8-stable
GO version: go1.17.7
Built: 2022-03-01T17:18:25+0000
OS/Arch: linux/amd64
This is the output of any job. Notice that the *** Warning has always occurred, even when everything worked nicely.
Running with gitlab-runner 14.8.2 (c6e7e194)
on Bannerfy API Deployment Runner nbHwkXMq
Preparing the "docker" executor
Using Docker executor with image google/cloud-sdk ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:c05a8c965912fe35df9cce58ddb7aed725fb34d81772761a4ef7a373fd3271c1 for docker:dind with digest docker@sha256:41bd636ca74163b65972d707fc43f6b955d9cc87d7fffaef76475b8a2eb9c323 ...
Waiting for services to be up and running...
*** WARNING: Service runner-nbhwkxmq-project-35-concurrent-0-990797c105006eb0-docker-0 probably didn't start properly.
Health check error:
service "runner-nbhwkxmq-project-35-concurrent-0-990797c105006eb0-docker-0-wait-for-service" timeout
Health check container logs:
Service container logs:
2022-03-17T22:09:29.442547090Z Generating RSA private key, 4096 bit long modulus (2 primes)
2022-03-17T22:09:30.721847353Z ......................................................................................................++++
2022-03-17T22:09:30.931153482Z ..................................++++
2022-03-17T22:09:30.931685425Z e is 65537 (0x010001)
2022-03-17T22:09:30.952699613Z Generating RSA private key, 4096 bit long modulus (2 primes)
2022-03-17T22:09:31.099361600Z ........................++++
2022-03-17T22:09:31.821369541Z ...........................................................................................................++++
2022-03-17T22:09:31.822241465Z e is 65537 (0x010001)
2022-03-17T22:09:31.863904122Z Signature ok
2022-03-17T22:09:31.865295592Z subject=CN = docker:dind server
2022-03-17T22:09:31.865815650Z Getting CA Private Key
2022-03-17T22:09:31.883194238Z /certs/server/cert.pem: OK
2022-03-17T22:09:31.887743155Z Generating RSA private key, 4096 bit long modulus (2 primes)
2022-03-17T22:09:32.735842262Z ................................................................................................................................++++
2022-03-17T22:09:33.823007624Z ........................................................................................................................................................................................++++
2022-03-17T22:09:33.823555463Z e is 65537 (0x010001)
2022-03-17T22:09:33.853023352Z Signature ok
2022-03-17T22:09:33.854470837Z subject=CN = docker:dind client
2022-03-17T22:09:33.854883520Z Getting CA Private Key
2022-03-17T22:09:33.870544651Z /certs/client/cert.pem: OK
2022-03-17T22:09:33.945369974Z ip: can't find device 'ip_tables'
2022-03-17T22:09:33.947519957Z ip_tables 32768 10 iptable_nat,iptable_filter
2022-03-17T22:09:33.947834000Z x_tables 40960 16 xt_MASQUERADE,ip6t_REJECT,xt_hl,ip6t_rt,ipt_REJECT,xt_LOG,xt_multiport,xt_comment,xt_limit,xt_addrtype,xt_tcpudp,xt_conntrack,ip6table_filter,ip6_tables,iptable_filter,ip_tables
2022-03-17T22:09:33.948670976Z modprobe: can't change directory to '/lib/modules': No such file or directory
2022-03-17T22:09:33.952890575Z mount: permission denied (are you root?)
2022-03-17T22:09:33.953326289Z Could not mount /sys/kernel/security.
2022-03-17T22:09:33.953446733Z AppArmor detection and --privileged mode might break.
2022-03-17T22:09:33.954875913Z mount: permission denied (are you root?)
*********
Pulling docker image google/cloud-sdk ...
Using docker image sha256:3991d2b6213bee500d30c23c184d81f3ad26eb075e060d61dff69b5df6adfeea for google/cloud-sdk with digest google/cloud-sdk@sha256:de7eb24d1bc22cc7dc4ba4d1fedd0c2cf6a68418e5701668692b1d21293a8fe8 ...
Preparing environment
00:01
Running on runner-nbhwkxmq-project-35-concurrent-0 via gallus...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/datanauts/bannerfy-api/.git/
Created fresh repository.
After this the runner is no longer responding / giving output and the job is terminated after 1 hour.
This behavior started yesterday evening, at around 5pm local time.
What we tried so far:
We started and configured a fresh machine for deployment, set up gitlab-runner and connected it to GitLab. The results were the same as on the old machine.
Any help would be highly appreciated