Networking troubles dind - pip/deb/npm dependencies

Hi,

We use gitlab-ci to build some python applications.
Unfortunally since 2 month this doesnt work well anymore.

Gitlab-CE installed with helm since 1year.
Hosted on AWS

Current versions :

Kubernetes 1.15.16 installed via kops 
network : calico
Gitlab-runner 12.5.0
Gitlab 12.5.4 
docker:19.03.5-dind

Versions tried :

gitlab 12.3-12.5 
Runner : 12.2-12.5
k8s: 1.13-1.15 

Explanation

We cant easily get pip package from pypi.org
We have multiple warnings and retry and then at the 4th/5th try, sometimes it works (but we have to wait until 5th try…).
Even in a “local” repository (pypiserver.ns.svc.cluster.local )
I tried everything i found on forum but nothing works.
(change mtu, change image version, change runner version, add long timeout for pip install… )
Oh, and i don’t know if this is related too but i got simmilar problem with NPM and debian packages…
And Obviously, everything is ok locally.

Here’s example :

Step 10/11 : RUN pip3 install --default-timeout=5 --no-cache-dir -r requirements.txt
 ---> Running in 2221753f970c
Looking in indexes: https://pypi.org/simple, https://admin:****@svc-pypisrv.ops.svc.cluster.local/simple
Collecting requests (from -r requirements.txt (line 1))
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f97823330b8>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9782333e48>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9782333dd8>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9782333f60>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f978233e0b8>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=5.0)")': /packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB) <<< Here the package is downloaded.
Collecting datetime (from -r requirements.txt (line 2))

CI :

variables:
   DOCKER_HOST: tcp://localhost:2375
   DOCKER_DRIVER: overlay2
   DOCKERFILE_PATH: ./Dockerfile
   DOCKER_TLS_CERTDIR: ""
build_dockerfile:
  stage: build
  image: docker:19.03.5-dind
  services:
    - docker:dind
    - name: docker:stable-dind
      command: [ "dockerd", "--host=unix:///var/run/docker.sock", "--host=tcp://0.0.0.0:2375", "--mtu=1440" ] # I tried with and without thoses lines 
  script:
    - docker login  ... 
    - build ... 
  ... 

I found that there is a problem with alpine dns but i don’t know if it’s related to my problem (and if this is it i dont know how to fix it with gitlab helm deployment).
And i even tried on a fresh gitlab install on another cluster and the problem was the same.

Thanks you for reading this and i hope you will help me to fix this …

if there is some informations missing please tell me

Looks like external connection errors. Most likely answer is that your outbound software or virtual firewall if blocking you.

1 Like

I think this is a calico trouble.
It works on GKE / EKS / kops with canal networking but not with calico.
I just try to deploy 2 k8s clusters with kops : One with calico and one with canal (works also with weavenet).
And as expected, the calico one have the same issue.

So you can reproduce the bug with a fresh k8s install with kops and calico networking and latest gitlab version installed with helm.

Someone can try the following config ?
Deploy kops cluster with calico
install gitlab helm chart
build dind python Dockerfile with pip requirements

1 Like