DIND Containers not accepting connections - shared runner

How to Use GitLab
,
1
  • What are you seeing, and how does it differ from what you expect to see?

All of our pipelines are failing due to one of the container we are running refusing to accept connections. No changes were made and previously working pipelines are now failing.

Best example of this is pipeline#118037545 (project #6672684) where there are 3 jobs in the testing stage. The first one was successfully ran under job #438058283 and then manually deployed without issue. Then after we began experiencing failures on other pipelines we decided to try re-running this same already successful job which gave us the same errors as our other pipelines. Waited some time and tried re-running again without success. See attached image.

Screen Shot 2020-02-14 at 7.17.55 PM
Screen Shot 2020-02-14 at 7.17.55 PM2616×1360 219 KB

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

I’ve tried updating the docker image to docker:19.03.1 from 19.03.0 as well as updating the dind service to docker:19.03.5-dind from docker:19.03.0-dind, as suggested in the recent threads with hanging docker issues, without any changes.

1 Like
2

Seeing what might be a similar issue here: tests have been failing for past 24 hours as unable to connect to postgres container.

1 Like
3

This has also just started happening for me (on GitLab’s hosted CI.)

Screen Shot 2020-02-17 at 2.31.52 PM
Screen Shot 2020-02-17 at 2.31.52 PM2550×1038 344 KB

My Rails app tests suddenly started failing a few days ago with the following error:

rake aborted!
 PG::ConnectionBad: could not translate host name "postgres" to address: Name or service not known

I haven’t changed my .gitlab-ci.yml for a long time. Relevant config:

rspec:
  stage: tests
  services:
    - postgres:latest
    - redis:latest
  ...
4

This was the issue: https://github.com/docker-library/postgres/issues/681

The official postgres Docker image was updated with some strict security settings. This was a breaking change that caused the CI builds to fail.

My workaround is to use an older tag for the service image:

  services:
    - postgres:9.6.16-alpine
    - redis:latest
5

Yes that was the same issue. If you do not wish to downgrade your image you can also set the environment variable POSTGRES_HOST_AUTH_METHOD=trust or use an explicit password POSTGRES_PASSWORD.