Hi,
We use gitlab-ci to build some python applications.
Unfortunally since 2 month this doesnt work well anymore.
Gitlab-CE installed with helm since 1year.
Hosted on AWS
Current versions :
Kubernetes 1.15.16 installed via kops
network : calico
Gitlab-runner 12.5.0
Gitlab 12.5.4
docker:19.03.5-dind
Versions tried :
gitlab 12.3-12.5
Runner : 12.2-12.5
k8s: 1.13-1.15
Explanation
We cant easily get pip package from pypi.org
We have multiple warnings and retry and then at the 4th/5th try, sometimes it works (but we have to wait until 5th try…).
Even in a “local” repository (pypiserver.ns.svc.cluster.local )
I tried everything i found on forum but nothing works.
(change mtu, change image version, change runner version, add long timeout for pip install… )
Oh, and i don’t know if this is related too but i got simmilar problem with NPM and debian packages…
And Obviously, everything is ok locally.
Here’s example :
Step 10/11 : RUN pip3 install --default-timeout=5 --no-cache-dir -r requirements.txt
---> Running in 2221753f970c
Looking in indexes: https://pypi.org/simple, https://admin:****@svc-pypisrv.ops.svc.cluster.local/simple
Collecting requests (from -r requirements.txt (line 1))
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5.0)")': /simple/requests/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f97823330b8>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9782333e48>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9782333dd8>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9782333f60>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f978233e0b8>, 'Connection to svc-pypisrv.ops.svc.cluster.local timed out. (connect timeout=5.0)')': /simple/requests/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=5.0)")': /packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB) <<< Here the package is downloaded.
Collecting datetime (from -r requirements.txt (line 2))
CI :
variables:
DOCKER_HOST: tcp://localhost:2375
DOCKER_DRIVER: overlay2
DOCKERFILE_PATH: ./Dockerfile
DOCKER_TLS_CERTDIR: ""
build_dockerfile:
stage: build
image: docker:19.03.5-dind
services:
- docker:dind
- name: docker:stable-dind
command: [ "dockerd", "--host=unix:///var/run/docker.sock", "--host=tcp://0.0.0.0:2375", "--mtu=1440" ] # I tried with and without thoses lines
script:
- docker login ...
- build ...
...
I found that there is a problem with alpine dns but i don’t know if it’s related to my problem (and if this is it i dont know how to fix it with gitlab helm deployment).
And i even tried on a fresh gitlab install on another cluster and the problem was the same.
Thanks you for reading this and i hope you will help me to fix this …