Since some time the CI pipeline fails often with access issues to APIs provided by GitLab. Example of those issues:
$ docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com WARNING! Using --password via the CLI is insecure. Use --password-stdin. Error response from daemon: Get "https://registry.gitlab.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
ERROR: Job failed: failed to pull image "registry.gitlab.com/3ker-grizzzel/candidate-api/app:de5061e6de3cf2506400024cfb5091dfbc017920" with specified policies [always]: Error response from daemon: Get "https://registry.gitlab.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) (manager.go:237:15s)
Downloading artifacts for test:data-migration (5167211972)... ERROR: Downloading artifacts from coordinator... forbidden id=5167211972 responseStatus=403 Forbidden status=403 Forbidden token=64_kzLnz FATAL: permission denied
Pipeline often fails if restarted immediately but works again if waiting a few hours.
Using custom CI runners setup with Docker+Machine on Hetzner. Has anyone seen this issue before?
Error in the request to Docker API seem to indicate a network connectivity issue. But it is strange that we see an authentication issue with the artifacts at the same time. Could it be that we run into some kind of rate limiting, which is handled differently between the services?
Would report as a bug. But I don’t have any idea how to reproduce it. It’s just happening from time to time.