Gitlab runner hangs infrequently on Python's `pip install` or `pip download`

robert.meyer · May 5, 2019, 11:33am

Hi,

I use gitlab runner with docker in docker. Unfortunately, our runners sometimes (and infrequently and un-reproducibly) hang or get stuck upon the following commands while building the docker container:

RUN pip download --no-cache-dir -r requirements.txt -d /artifacts

or

RUN pip install --no-cache-dir --no-index --find-links=/tmp/artifacts /tmp/artifacts/*

It hangs there for an hour until the build times out. The only solution is to restart the build process (and then it usually completes within just 3-5 minutes!).

The gitlab yaml looks (kinda) like this following minimal example:

 - docker:dind
before_script:
  - docker info

build:
  stage: build
  image: docker:1.11
  tags:
    - docker
  script:
    - ./configure
    - make build
    - make push

and make build simply does
docker build $(IMAGE_NAME) -t $(IMAGE_NAME_SHORT) -f Dockerfile .

Any idea what could be the cause for this or where to first dig into to identify the problem?

Is this most likely a pypi issue?
Or is this a docker issue?
Or is this a gitlab issue?

Thanks!

robert.meyer · May 6, 2019, 9:57am

Unfortunately, this keeps happening, anyone any idea?

ghostsquad · May 7, 2019, 5:52am

Maybe this might help?

iamliamc · May 7, 2019, 12:55pm

Wanted to add attention to this post, we are currently having significant issue on many of our builds due to timeouts after hanging on pip installations.

The problem is sporadic, but also a frequent kind of sporadic in our cases.

Lots of this:

WARNING: Retrying 
(Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): 
Read timed out. (read timeout=15)",)':

Among other things. Builds work fine locally, no changes in requirements.txt simple Docker command:

RUN pip install -r requirements.txt
RUN pip install --no-cache-dir -vvv -r requirements.txt

Also tried all sorts of stuff about adding indexes explicitly to pip but that did nothing.

ghostsquad · May 7, 2019, 5:14pm

@iamliamc maybe this would help you? Tldr: change the timeout

https://stackoverflow.com/questions/43298872/how-to-solve-readtimeouterror-httpsconnectionpoolhost-pypi-python-org-port

ghostsquad · May 7, 2019, 5:19pm

Some other things that come to mind:

When this happens, can you identify which runner (if you have multiple) that this occurs on?
Can you do some other internet connectivity tests before this command (like ping, etc) to see if there’s a problem there?

iamliamc · May 7, 2019, 5:40pm

Hey @ghostsquad - thanks for the link. Yeah, I should’ve been more explicit, like the original poster my runner times out after 1 hr (the default), the stack trace above doesn’t actually alway happen, nor is it necessarily the place where the pip install command hangs. We did try increasing it to 360 seconds but hasn’t quite solved it.

Regarding your second point.
The job includes pulling an ubuntu image, cloning repositories and sub-repositories, and often the job is able to reach pypi and download and install multiple packages and then hangs. So we know it has network connection (but at somepoint get’s wonked out).

ghostsquad · May 8, 2019, 4:27am

@iamliamc docker in docker runner? Or other? Have you tried multiple varieties to see if it’s specific to the runner?

Can you encode your job as a script and run it 100+ times (in the same docker image in your pipeline) without the using Gitlab runner? Maybe that might help isolate whether this is a runner problem, script problem, network problem, etc.

robert.meyer · May 8, 2019, 1:01pm

This keeps happening even with --default-timeout=999.
Here’s the beginning of the gitlab runner log:

Running with gitlab-runner 11.11.0-rc2 (7f58b1ec)
  on docker-auto-scale fa6cab46
Using Docker executor with image docker:1.11 ...
Starting service docker:dind ...

robert.meyer · May 11, 2019, 11:52am

For now it seems to work again. Can someone from gitlab comment on this? Could it have been network issues on gitlab’s site or issues on PyPi’s site? Thanks!

iamliamc · May 16, 2019, 2:10pm

Yeah I’ve got to agree I’ve no longer had this problem this week and never came to an actual conclusion or fix on my end… we did upgrade to the newest gitlab. Thanks for all the comments and collaboration on possible fixes.

hij3200 · June 3, 2020, 10:18pm

One year later, and it is keep hanging in python:3.6-alpine3.11

hij3200 · June 3, 2020, 10:54pm

I resolved it with this accepted answer

I wrote in my docker-compose.yml file:

       version: '3.4'
       services:
            image_name:
               build:
                   context: .
                   network: host

dansolo · July 4, 2020, 1:26am

Sorry guys, but I’m getting this issue to in Gitlab.

Like @robert.meyer said, infrequently.

I’m trying to build a simple Python application and in some builds Runner take timeout from py.org in pip install. My image base is python:3.7.8-slim.

The Runner is running on Kubernetes on AWS. About internet connection I don’t have any problems.

Using GitLab Community Edition 13.1.2

Down is an example:

Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-1) ...

[343](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L343) Setting up libgnutls28-dev:amd64 (3.6.7-4+deb10u4) ...

[344](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L344) Setting up libmariadb-dev (1:10.3.22-0+deb10u1) ...

[345](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L345) Setting up libmariadb-dev-compat:amd64 (1:10.3.22-0+deb10u1) ...

[346](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L346) Setting up default-libmysqlclient-dev:amd64 (1.0.5) ...

[347](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L347) Processing triggers for libc-bin (2.28-10) ...

[348](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L348) WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/mysqlclient/

[349](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L349) WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/mysqlclient/

[350](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L350) WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/mysqlclient/

[351](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L351) Collecting mysqlclient==1.4.6

[352](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L352)  Downloading mysqlclient-1.4.6.tar.gz (85 kB)

[353](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L353) Collecting Flask==1.1.2

[354](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L354)  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)

[355](https://gitlab.biqueirabr.com.br/gitlab/front/-/jobs/16#L355) Collecting Flask-RESTful==0.3.8

Some builds pass ok and others crash out of by timeout.