Auto DevOps Fails

GitLab CI/CD
1

Hello - I am running GitLab 15.10.3 (self-hosted)

System setup:

  • Ubuntu 20 VM using Omnibus installation via AWS using GitLab AMI
  • All-in-one setup - everything is running on this server (Docker and runners)

I am unable to get Auto DevOps working but CI/CD appears to work. I must have a configuration option wrong but after carefully reviewing the documentation around CI/CD and Auto DevOps, I can’t figure out what’s wrong.

Problems observed with Auto DevOps:

  • Warning: Service runner-nhyo6zgu-project-27-concurrent-0-110477ca3fc38451-docker-0 probably didn’t start properly
  • Error: Health check error: start service container: Error response from daemon: Cannot link to a non running container: /runner-nhyo6zgu-project-27-concurrent-0-110477ca3fc38451-docker-0 AS /runner-nhyo6zgu-project-27-concurrent-0-110477ca3fc38451-docker-0-wait-for-service/service (services.go:189:0s)
  • Error: Service container logs: mount: permission denied (are you root?)
  • Error: error during connect: Post “http://docker:2375/v1.24/auth”: dial tcp: lookup docker on 172.16.18.103:53: no such host
    • Note: 172.16.18.103 is our internal DNS server - I’m assuming it’s trying to do a DNS query for “docker” which fails
  • Warning: gl-auto-build-variables.env: no matching files. Ensure that the artifact path is relative to the working directory (/builds/james.small/auto-devops-test)

When I use CI/CD with an explicit .gitlab-ci.yml file, it works:

Running with gitlab-runner 15.10.1 (dcfb4b66)
  on GitLab-Docker-Local-Runner nhyo6ZGu, system ID: r_meoMYN5BfpiR
Preparing the "docker" executor
00:01
Using Docker executor with image ruby:2.7 ...
Pulling docker image ruby:2.7 ...
Using docker image sha256:a5f545f007b1f082e453d9088791fcd5146ea5cf053804bcd5a232252f70636c for ruby:2.7 with digest ruby@sha256:27322aa8ffad7a9640bc1108f398597fdab31d7ee6d745686b837bea53236b92 ...
Preparing environment
00:00
Running on runner-nhyo6zgu-project-20-concurrent-0 via f699884cff5c...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/james.small/ci-test/.git/
Checking out 9c010bf9 as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
Using docker image sha256:a5f545f007b1f082e453d9088791fcd5146ea5cf053804bcd5a232252f70636c for ruby:2.7 with digest ruby@sha256:27322aa8ffad7a9640bc1108f398597fdab31d7ee6d745686b837bea53236b92 ...
$ echo "Do a test here"
Do a test here
$ echo "For example run a test suite"
For example run a test suite
$ echo "Third line..."
Third line...
Cleaning up project directory and file based variables
00:00
Job succeeded

However, if I try to do something with Auto DevOps it fails:

Running with gitlab-runner 15.10.1 (dcfb4b66)
  on GitLab-Docker-Local-Runner nhyo6ZGu, system ID: r_meoMYN5BfpiR
Preparing the "docker" executor
00:03
Using Docker executor with image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.30.0 ...
Starting service docker:20.10.12-dind ...
Pulling docker image docker:20.10.12-dind ...
Using docker image sha256:1a42336ff683d7dadd320ea6fe9d93a5b101474346302d23f96c9b4546cb414d for docker:20.10.12-dind with digest docker@sha256:6f2ae4a5fd85ccf85cdd829057a34ace894d25d544e5e4d9f2e7109297fedf8d ...
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-nhyo6zgu-project-27-concurrent-0-110477ca3fc38451-docker-0 probably didn't start properly.
Health check error:
start service container: Error response from daemon: Cannot link to a non running container: /runner-nhyo6zgu-project-27-concurrent-0-110477ca3fc38451-docker-0 AS /runner-nhyo6zgu-project-27-concurrent-0-110477ca3fc38451-docker-0-wait-for-service/service (services.go:189:0s)
Service container logs:
2023-04-19T12:53:08.396933043Z ip: can't find device 'ip_tables'
2023-04-19T12:53:08.397709656Z ip_tables              32768  2 iptable_nat,iptable_filter
2023-04-19T12:53:08.397744699Z x_tables               53248  8 xt_conntrack,xt_MASQUERADE,iptable_nat,xt_addrtype,ip6table_filter,ip6_tables,iptable_filter,ip_tables
2023-04-19T12:53:08.398161684Z modprobe: can't change directory to '/lib/modules': No such file or directory
2023-04-19T12:53:08.400603327Z mount: permission denied (are you root?)
2023-04-19T12:53:08.400676272Z Could not mount /sys/kernel/security.
2023-04-19T12:53:08.400683012Z AppArmor detection and --privileged mode might break.
2023-04-19T12:53:08.401686269Z mount: permission denied (are you root?)
*********
Pulling docker image registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.30.0 ...
Using docker image sha256:7752aaac107bde593a659da30131e9589b9dfadf4b9f6c9debadaedb812206e8 for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.30.0 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:1cc43f962ea259f9ae49a43291fe916aed0b831d16bd364ac7b36fceefdb89e4 ...
Preparing environment
00:00
Running on runner-nhyo6zgu-project-27-concurrent-0 via f699884cff5c...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/james.small/auto-devops-test/.git/
Checking out eaa48337 as detached HEAD (ref is new-feature)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
Using docker image sha256:7752aaac107bde593a659da30131e9589b9dfadf4b9f6c9debadaedb812206e8 for registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image:v1.30.0 with digest registry.gitlab.com/gitlab-org/cluster-integration/auto-build-image@sha256:1cc43f962ea259f9ae49a43291fe916aed0b831d16bd364ac7b36fceefdb89e4 ...
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
$ /build/build.sh
Logging in to GitLab Container Registry with CI credentials...
error during connect: Post "http://docker:2375/v1.24/auth": dial tcp: lookup docker on 172.16.18.103:53: no such host
Uploading artifacts for failed job
00:01
Uploading artifacts...
WARNING: gl-auto-build-variables.env: no matching files. Ensure that the artifact path is relative to the working directory (/builds/james.small/auto-devops-test) 
ERROR: No files to upload                          
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

Steps to setup GitLab self-hosted instance:

Troubleshooting:

Any suggestions greatly appreciated,
–Jim

2

This is the documentation I was looking for:
GitLab - Use Docker to build Docker images

In particular, following this cleared up all the issues:
GitLab - Use Docker-in-Docker

My only caution - make sure you understand the security consequences of this.