Pipeline stuck with fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz

iarkaroy · September 26, 2021, 3:55pm

I have a pipeline that builds a docker image with alpine as base image. But the pipeline is getting stuck in RUN apk update && apk add --no-cache git and the last output I am seeing is

fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz

and eventually the job is getting timed out.

I have tried several times but it is getting stuck in the same output. Any ideas?

hungphamvn · September 27, 2021, 2:56am

I’m facing the same issue, any solution yet?

renestalder · September 27, 2021, 2:03pm

I have the same issue. But I’m also certain it has something to do with the ongoing network issues. So, you might want to wait for those issues to be resolved then check again.

https://status.gitlab.com

prantlf · September 27, 2021, 2:18pm

You can try decreasing the default MTU in your .gitlab-ci.yml on gitlab.com:

image: docker:20
services:
  - name: docker:dind
    command: ["--mtu=1300"]

It helped my “red” pipeline become “green” again.

Other sources:

github.com/gliderlabs/docker-alpine

apk fetch hangs

opened 09:40AM - 17 Jul 17 UTC

closed 06:08AM - 05 Oct 18 UTC

dkirrane

question

fetch of the apk index just hangs. I hit this now on a Ubuntu server and Docker… for Windows ``` Step 1/14 : FROM maven:3.3.9-jdk-8-alpine ---> dd9d4e1cd9db Step 2/14 : RUN apk update && apk upgrade && apk add --no-cache --update ca-certificates bash wget curl tree libxml2-utils putty git && rm -rf /var/lib/apt/lists/* && rm -rf /var/cache/apk/* ---> Running in 536cbd484c36 fetch http://dl-cdn.alpinelinux.org/alpine/v3.5/main/x86_64/APKINDEX.tar.gz ``` Docker version 17.03.1-ce, build c6d412e

github.com/docker-library/docker

Setting a MTU for dind is difficult

opened 05:25PM - 27 Mar 18 UTC

closed 04:29PM - 01 May 18 UTC

neclimdul

question

https://github.com/moby/moby/issues/36659 isn't fixed for me yet, but assuming a…t some point it is setting the MTU is currently a bit tricky for the dind image in certain scenarios. In my case I want to run dind as a service to GitLab CI builds running in Kubernetes. For that to work I need to control the MTU of the docker bridge and I have limited control of how the service is started. Basically only really able to specify environment and and image (not able to interact with entrypoint). I think I have 2 options. 1) Extending the image and provide a mechanism for setting the MTU during startup using that image as a service. 2) Help this repo could have an environment variable for controlling the MTU. If 2, I'm willing to do the work writing and testing a PR given some direction on what sort of approach would be ideal. Or just testing someone else's PR. :-D

renestalder · September 27, 2021, 2:24pm

Here are some more temporarily workarounds that might help.

renestalder · September 27, 2021, 2:39pm

As from the referenced ticket, the following worked for me:

build docker job:
  variables:
    FF_NETWORK_PER_BUILD: "true"

OpakAlex · September 28, 2021, 10:38am

Same for me…

prantlf · September 29, 2021, 7:35am

Current Status

GitLab incident has been reported and a new issue about enforcing the proper MTU was filed:

Google Cloud incident about this issue:

Diagnosis: Some customers may be experiencing connection failures in Docker workflow to GCLB, GCB, or Fastly destinations such as debian.org, github.com with a timeout error.
Workaround: If you are impacted, please try adding an init container manifest into docker in docker deployment.This will ensures packets are sent with a proper MTU that will work with Fastly destinations.
Summary: Global: We have identified a Networking connectivity issue that impacts Docker workloads inside GKE and potentially GCE. Mitigation in progress. ETA - 00:00 PST, 29th Sept.
Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2021-09-29 00:00 US/Pacific. We will provide more information by Wednesday, 2021-09-29 00:30 US/Pacific.

uss.infrastructure · September 29, 2021, 2:10pm

plus 1 …

jpschafer · September 29, 2021, 2:37pm

I am dealing With the same issue, going to try the suggestion by @renestalder and see how it goes. Thanks!

prantlf · September 29, 2021, 4:46pm

Problem Resolved

by Google: The issue with Google Kubernetes Engine has been resolved for all affected users as of Wednesday, 2021-09-29 06:52 US/Pacific.

My failing test job using docker:dind got green after re-running the pipeline with no code change.