Pipeline stuck with fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz

GitLab CI/CD
,
1

I have a pipeline that builds a docker image with alpine as base image. But the pipeline is getting stuck in RUN apk update && apk add --no-cache git and the last output I am seeing is

fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz

and eventually the job is getting timed out.

I have tried several times but it is getting stuck in the same output. Any ideas?

3 Likes
2

I’m facing the same issue, any solution yet?

3

I have the same issue. But I’m also certain it has something to do with the ongoing network issues. So, you might want to wait for those issues to be resolved then check again.

https://status.gitlab.com

1 Like
4

You can try decreasing the default MTU in your .gitlab-ci.yml on gitlab.com:

image: docker:20
services:
  - name: docker:dind
    command: ["--mtu=1300"]

It helped my “red” pipeline become “green” again.

Other sources:

2 Likes
5

Here are some more temporarily workarounds that might help.

1 Like
6

As from the referenced ticket, the following worked for me:

build docker job:
  variables:
    FF_NETWORK_PER_BUILD: "true"
1 Like
7

Same for me…

8

Current Status

GitLab incident has been reported and a new issue about enforcing the proper MTU was filed:

Google Cloud incident about this issue:

  • Diagnosis: Some customers may be experiencing connection failures in Docker workflow to GCLB, GCB, or Fastly destinations such as debian.org, github.com with a timeout error.
  • Workaround: If you are impacted, please try adding an init container manifest into docker in docker deployment.This will ensures packets are sent with a proper MTU that will work with Fastly destinations.
  • Summary: Global: We have identified a Networking connectivity issue that impacts Docker workloads inside GKE and potentially GCE. Mitigation in progress. ETA - 00:00 PST, 29th Sept.
  • Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2021-09-29 00:00 US/Pacific. We will provide more information by Wednesday, 2021-09-29 00:30 US/Pacific.

9

plus 1 …

1 Like
10

I am dealing With the same issue, going to try the suggestion by @renestalder and see how it goes. Thanks!

11

Problem Resolved

by Google: The issue with Google Kubernetes Engine has been resolved for all affected users as of Wednesday, 2021-09-29 06:52 US/Pacific.

My failing test job using docker:dind got green after re-running the pipeline with no code change.