I’ll try to be a little brief, but it’s hard. This one I’m struggling with for a couple of days now.
I’m transitioning our GitLab runners on plain Docker servers to GitLab runners on Kubernetes. All on prem. This seems to work for all pipelines thus far, except for one thing: some connections to the outside world, specifically while building (
docker build) and only when spawned from one of our runners on one of our Kubernetes clusters (image
This is a piece of my gitlab-ci.yml file, stripped to its core:
image: singularityware/singularity:gitlab-2.6 stages: - build_docker variables: DOCKER_HOST: tcp://localhost:2375 DOCKER_TLS_CERTDIR: "" services: - docker:stable-dind build_docker: image: docker:stable stage: build_docker tags: - kubernetes script: - docker build --tag "$CI_REGISTRY_IMAGE/python_example/devel:$CI_PIPELINE_ID" --tag "$CI_REGISTRY_IMAGE/python_example/devel:latest" python/.
This is a piece of the Dockerfile, stripped to its core:
FROM python RUN pip install numpy RUN pip install matplotlib ENTRYPOINT [ "python", "/added/example.py"]
During the pipeline, this fails on the first step in the script, specifically at the second step from the Dockerfile, with:
Step 2/7 : RUN pip install numpy ---> Running in 36adeda1dbb0 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/numpy/
followed by a couple of more retries and an exit 1.
A few key points:
- This is only while spawned from the GitLab runner
- From the plain Docker servers: no building issues
- This is also with other
pippackages, or for instance with a
- All nodes, clusters, pods on this subnet have direct internet access
- When I lookup, ping or curl from the pipeline or the Dockerfile, this goes without a problem
- When I recreate the Docker situation without the runner involved, i.e. I spawn a
docker:dindpod on the cluster, get the
Dockerfilein there and do a
docker build, the build is successful
- All other lookups, build, curls, pings, pip installs, pip searches, etc, from any pod in the cluster: no problem at all
- This is why I want to emphasize once more that this problem only arises when working from the GitLab runner
What am I missing here? Any feedback is greatly appreciated!