GitLab.com CI stuck on docker build

GitLab.com job getting stuck during docker build

What are you seeing, and how does it differ from what you expect to see?

I have a project on gitlab.com which uses CI/CD to run tests.

I am using docker:dind to build my project and run unit tests.

The job was working fine, until today when it started getting stuck on part of the docker build phase:

 Removing intermediate container 34b9d9867e8c
  ---> f63242630279
 Step 11/21 : RUN mkdir /app
  ---> Running in 7fbef75e7fa6
Pulling docker image gitlab/gitlab-runner-helper:x86_64-003fe500 ...
00:00
 ERROR: Job failed: execution took longer than 1h0m0s seconds

The job is on a public project here: https://gitlab.com/LondonAppDev/recipe-app-api-devops/-/jobs/438001402

I expect the job should work as it did yesterday: https://gitlab.com/LondonAppDev/recipe-app-api-devops/-/jobs/436880414

What version are you on (Hint: /help) ? and are you using self-managed or gitlab.com?

I’m using gitlab.com with the public runners

What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

1. I have tried re-running a job which worked yesterday, and the re-run failed.
2. I was using my own custom image for the build (londonappdev/docker-with-compose:19) for the job, in-case this was the culprit, I tried using docker:19 directly and got the same issue.
3. I cleared the runner cache.
4. I checked the GitLab Status page for outages, and there are none which seem related to this issue.

Anyone have any ideas what might cause this? Is there something I am doing wrong, or could there be an outage?

Having the same issue did you find a fix? it just started happening

I haven’t confirmed a fix yet, however I found this post so it seems like this is a known issue:

I’m trying the suggested fix (adding -dind to the docker image tag) to see if this solves it.

Update: I can confirm that adding the following to my job fixed the issue:

  services:
    - docker:19.03.5-dind

This did indeed fix my issue thanks alot!

For the GitLab team or anyone wanting to re-produce this, here is a minimal project that demonstrates the issue: https://gitlab.com/LondonAppDev/sample-dind-build-failing/-/jobs/438236042

Thanks @londonappdev! It worked for me as well!

Thank you @londonappdev, fixed my issue

Can confirm the issue and the resolution. Thanks !

However this issue consumed all my CI minutes

Thanks.

Thank you, @londonappdev!

Working fine thanks!!!

Worked for me as well. Many thanks!

services:
  - docker:19.03.5-dind

Ran into the same issue and used the same solution. Thanks!

great, worked here too !

My build jobs are getting stuck at first RUN command of Dockerfile. It remains stuck for 20 minutes, never completes and the job is auto-marked as failed. It happens for every build. It is not a once-a-while issue. I have included the suggested line in .gitlab-ci.yml:

services
  - docker:19.03.6-dind

Any suggestions on what could be the problem because of which it remains stuck?

Screenshot_20210412_2300001379×855 94.5 KB

The suggested version was 19.03.5, not the 19.03.6, try it. Also check Job execution hangs when latest docker:dind (version 19.03.6 and "floating" ones) is used (#6697) · Issues · GitLab.org / gitlab-runner · GitLab