Task fails to start with "remote error: tls: bad certificate driver=google"

GitLab CI/CD
, ,
1

Several weeks ago, we began to see this error when running tasks on a docker-machine autoscale configuration of gitlab-runner, on Google cloud.

I’m having difficulty even locating the certificate associated with the runner or docker-machine which is supposedly expired.

Is the error possibly a red herring? Can anyone suggest troubleshooting steps? Thanks very much.

Here is the service log obtained with “journalctl -f -u gitlab-runner” on the runner VM.

Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Checking for jobs... received                       job=8723052130 repo_url=https://gitlab.com/happyplace/root.git runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Processing chain                                    chain-leaf=[0xc000c33088 0xc000c33608 0xc000fe4108] context=certificate-chain-build resolve-full-chain=false
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Added job to processing list                        builds=1 job=8723052130 max_builds=3 project=3222143 repo_url=https://gitlab.com/happyplace/root.git time_in_queue_seconds=0
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Requeued the runner                                 builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Running with gitlab-runner 17.7.0 (3153ccc6)        job=8723052130 project=3222143 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]:   on gitlab-runner-gce-root-2 9db142ae, system ID: s_2ce7fa8dbcad  job=8723052130 project=3222143 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Preparing the "docker+machine" executor  job=8723052130 project=3222143 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Executing /usr/local/bin/docker-machine [docker-machine --bugsnag-api-token=no-report create --driver google --google-project=sunny-ship-215417 --google-machine-type=e2-standard-32 --google-machine-image=[REDACTED] --google-tags=gitlab-ci-slave --google-preemptible=true --google-zone=us-west2-c --google-use-internal-ip=true --google-disk-type=pd-ssd --google-disk-size=50 --engine-storage-driver=overlay2 runner-9db142ae-auto-scale-runner-1735148198-88722b19]
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Processing runner                                   builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring executor from provider                    builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Docker Machine Details                              creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=9db142ae time=2024-12-25 17:36:38.468688912 +0000 UTC m=+40.247842183 total=1 used=0
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=9db142ae total=1 used=0
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring job slot                                  builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring request slot                              builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Running pre-create checks...                        driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Check that the project exists  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Check if the instance already exists  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Checking for jobs...no content                      runner=9db142ae status=204 No Content
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: Creating machine...                                 driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:38 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Generating SSH Key  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:39 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Creating host...  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:39 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Opening firewall ports  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:39 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Creating instance  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Feeding runners to channel                          builds=1 max_builds=3
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Feeding runner to channel                           builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Processing runner                                   builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring executor from provider                    builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Docker Machine Details                              creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=9db142ae time=2024-12-25 17:36:40.936204425 +0000 UTC m=+42.715357701 total=1 used=0
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=9db142ae total=1 used=0
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring job slot                                  builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:40 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring request slot                              builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:41 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Waiting for Instance  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:41 gitlab-runner-root-2 gitlab-runner[14461]: Appending trace to coordinator...ok                 code=202 job=8723052130 job-log=0-236 job-status=running runner=9db142ae sent-log=0-235 status=202 Accepted update-interval=1m0s
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Feeding runners to channel                          builds=1 max_builds=3
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Feeding runner to channel                           builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Processing runner                                   builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring executor from provider                    builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Docker Machine Details                              creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=9db142ae time=2024-12-25 17:36:43.937204309 +0000 UTC m=+45.716357585 total=1 used=0
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: IdleCount is set to 0 so the machine will be created on demand in job context  creating=1 idle=0 idleCount=0 idleCountMin=0 idleScaleFactor=0 maxMachineCreate=0 maxMachines=0 removing=0 runner=9db142ae total=1 used=0
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring job slot                                  builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:43 gitlab-runner-root-2 gitlab-runner[14461]: Acquiring request slot                              builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:46 gitlab-runner-root-2 gitlab-runner[14461]: Feeding runners to channel                          builds=1 max_builds=3
Dec 25 17:36:46 gitlab-runner-root-2 gitlab-runner[14461]: Feeding runner to channel                           builds=1 max_builds=3 runner=9db142ae
Dec 25 17:36:49 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Uploading SSH Key  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:52 gitlab-runner-root-2 gitlab-runner[14461]: Waiting for machine to be running, this may take a few minutes...  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:52 gitlab-runner-root-2 gitlab-runner[14461]: Detecting operating system of created instance...   driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:36:52 gitlab-runner-root-2 gitlab-runner[14461]: Waiting for SSH to be available...                  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:00 gitlab-runner-root-2 gitlab-runner[14461]: Detecting the provisioner...                        driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:00 gitlab-runner-root-2 gitlab-runner[14461]: Provisioning with debian...                         driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:07 gitlab-runner-root-2 gitlab-runner[14461]: Copying certs to the local machine directory...     driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:08 gitlab-runner-root-2 gitlab-runner[14461]: Copying certs to the remote machine...              driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:10 gitlab-runner-root-2 gitlab-runner[14461]: Setting Docker configuration on the remote daemon...  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:13 gitlab-runner-root-2 gitlab-runner[14461]: Checking connection to Docker...                    driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: ERROR: Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "10.168.0.16:2376": remote error: tls: bad certificate  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: ERROR: You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: ERROR: Be advised that this will trigger a Docker daemon restart which might stop running containers.  driver=google name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=create
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: ERROR: Machine creation failed                      error=exit status 1 name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 time=36.777253112s
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: WARNING: Requesting machine removal                 lifetime=36.77765578s name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 now=2024-12-25 17:37:15.240033917 +0000 UTC m=+77.019187175 reason=Failed to create used=36.777656126s usedCount=0
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: WARNING: Stopping machine                           lifetime=36.778456265s name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 reason=Failed to create used=803.2µs usedCount=0
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: Stopping "runner-9db142ae-auto-scale-runner-1735148198-88722b19"...  name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=stop
Dec 25 17:37:15 gitlab-runner-root-2 gitlab-runner[14461]: (runner-9db142ae-auto-scale-runner-1735148198-88722b19) Waiting for instance to stop.  name=runner-9db142ae-auto-scale-runner-1735148198-88722b19 operation=stop

Configuration

Our config.toml

concurrent = 3
check_interval = 0
log_level = "debug"

[[runners]]
  name = "gitlab-runner-gce-root-2"
  url = "https://gitlab.com/"
  token = "[REDACTED]"
  executor = "docker+machine"
  environment = ["DOCKER_DRIVER=overlay2"]
  request_concurrency = 4

  [runners.docker]
    tls_verify = false
    image = "[REDACTED]"
    privileged = false
    disable_cache = true
    shm_size = 0

  [runners.cache]
    Type = "gcs"
    Path = "cache"
    Shared = false
    [runners.cache.gcs]
      CredentialsFile = "/home/gitlab-runner/[REDACTED]"
      BucketName = "[REDACTED]"

  [runners.machine]
    IdleCount = 0
    IdleTime = 60
    MachineDriver = "google"
    MachineName = "auto-scale-runner-%s"
    MachineOptions = [
      "google-project=[REDACTED]",
      "google-machine-type=e2-standard-32",
      "google-machine-image=https://www.googleapis.com/compute/v1/projects/[REDACTED]",
      "google-tags=gitlab-ci-slave",
      "google-preemptible=true",
      "google-zone=us-west2-c",
      "google-use-internal-ip=true",
      "google-disk-type=pd-ssd",
      "google-disk-size=50",
      "engine-storage-driver=overlay2"
    ]

Versions

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners

Versions

  • GitLab Runner

$ gitlab-runner -v
Version: 17.7.0
Git revision: 3153ccc6
Git branch: 17-7-stable
GO version: go1.23.2
Built: 2024-12-19T20:10:51+0000
OS/Arch: linux/amd64