Connection problems with kubernetes executer

Infrastructure as Code & Cloud Native
1

Hi all,

I’ll try to be a little brief, but it’s hard. This one I’m struggling with for a couple of days now.

I’m transitioning our GitLab runners on plain Docker servers to GitLab runners on Kubernetes. All on prem. This seems to work for all pipelines thus far, except for one thing: some connections to the outside world, specifically while building (docker build) and only when spawned from one of our runners on one of our Kubernetes clusters (image gitlab/gitlab-runner:alpine-v13.2.1).

This is a piece of my gitlab-ci.yml file, stripped to its core:

image: singularityware/singularity:gitlab-2.6
stages: 
  - build_docker
variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""
services:
  - docker:stable-dind
build_docker:
  image: docker:stable
  stage: build_docker
  tags:
    - kubernetes
  script:
    - docker build --tag "$CI_REGISTRY_IMAGE/python_example/devel:$CI_PIPELINE_ID" --tag "$CI_REGISTRY_IMAGE/python_example/devel:latest" python/.

This is a piece of the Dockerfile, stripped to its core:

FROM python
RUN pip install numpy 
RUN pip install matplotlib
ENTRYPOINT [ "python", "/added/example.py"]

During the pipeline, this fails on the first step in the script, specifically at the second step from the Dockerfile, with:

Step 2/7 : RUN pip install numpy
 ---> Running in 36adeda1dbb0
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/numpy/

followed by a couple of more retries and an exit 1.

A few key points:

  • This is only while spawned from the GitLab runner
  • From the plain Docker servers: no building issues
  • This is also with other pip packages, or for instance with a npm install
  • All nodes, clusters, pods on this subnet have direct internet access
  • When I lookup, ping or curl from the pipeline or the Dockerfile, this goes without a problem
  • When I recreate the Docker situation without the runner involved, i.e. I spawn a docker:dind pod on the cluster, get the Dockerfile in there and do a docker build, the build is successful
  • All other lookups, build, curls, pings, pip installs, pip searches, etc, from any pod in the cluster: no problem at all
  • This is why I want to emphasize once more that this problem only arises when working from the GitLab runner :wink:

What am I missing here? Any feedback is greatly appreciated!

2

##Correction
Sorry didn’t fully read all the way through looks like docker is working fine but the issue pip timing out during the building of the image. Only think I can think for timeouts would be firewall blocking it for some reason. May need to pass in your own pip.conf if you are using a proxy.