Gitlab shared runner can not connect to Docker daemon and leaks secrets

GitLab CI/CD
, ,
#1

Hello!

I have two problems:
1) Pipeline running on shared runner throws error

"failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial tcp 172.17.0.3:2375: connect: connection refused"

I tried to restart pipeline several times, same error happens, sometimes error message differs little bit. That’s other error message I get

Cannot connect to the Docker daemon at tcp://docker:2375. Is the docker daemon running?

And here is step it is supposed to run


dockerize:
  stage: dockerize
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker build
      --build-arg DB_HOST=$PROD_DB_HOST
      --build-arg DB_USER=$PROD_DB_USER
      --build-arg DB_PASSWORD=$PROD_DB_PASSWORD
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  only:
    - develop
    - tags

It worked fine some days ago, but not it keeps throwing this error at build step



And other problem / question:

Sometimes when pipeline fails with this error it leaks all secrets into job history and I can’t find way how to clear job history. Someone suggested to delete whole project and make it again but that does not seem to be too good option (might cause all sorts of other issues)

Here is error message it throws sometimes

error during connect: Post http://docker:2375/v1.40/build?buildargs=%7B%DB_HOST%22%3A%22 ... all secrets leaked in here ... : context canceled


ERROR: Job failed: exit code 1

Both errors have happened on shared-runners-manager-4.gitlab.com and also with managers 3, 5, and 6

https://status.gitlab.com/ shows all green. Maybe there is something I should change in my CI settings to get it working again?

1 Like
#3

It seems docker updated their latest stable images and gitlab has not updated their runners yet, changing the images to be on the 18 major fixes these issues, eg:

image: docker:18-git

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

services:
  - docker:18-dind
4 Likes
#4

Nice!
It worked like charm!
Thank you!

Also for anyone who want’s to delete pipelines / jobs to get rid of exposed secrets, seems like only way to delete them currently is through API: https://docs.gitlab.com/ee/api/pipelines.html#delete-a-pipeline

#5

Link to related issue:

1 Like
Dind gitlab-runner service doesn't start up suddenly
#6

Thanks @ctorrisi the hotfix saves my day.:sweat_smile:

# .gitlab-ci.yml
variables:
  DOCKER_TLS_CERTDIR: ""
#7

@cheewai.lai Great to hear. The GitLab team are working on a more permanent fix and should resolve that issue once it is done.

Issue is being formally tracked on: https://gitlab.com/gitlab-com/gl-infra/production/issues/982

#8

You delete the job log using the delete icon at the top of the log. Looks like a tiny trashcan.

#9

You can delete an individual job’s log using the delete icon at the top of the log. There’s no need to go to the API unless you need to do bulk deletion.

1 Like