Pipeline Timeout Due to gitlab-helper Image pull

GitLab CI/CD
, ,
1

Hello,
we have self hosted gitlab instance ver [12.10.14-ee].
And we have experienced weird behaviour from few of our pipelines…
On the Apr. 16th 8am GMT, few of our CI/CD jobs started to pull some image toward the end of its run.
Pulling docker image gitlab/gitlab-runner-helper:x86_64-577f813d
ERROR: Job failed: execution took longer than 1h0m0s seconds
Do you have any idea why it happened? It happened only for scheduled job of one pipeline others were OK.
.gitlab-ci.yml was basicly just:

image: python:3.8
variables:
PIP_CACHE_DIR: “$CI_PROJECT_DIR/.cache/pip”
GIT_CLEAN_FLAGS: -ffdx -e venv/ -e .cache/pip

  • ls -a
  • python -V # Print out python version for debugging
  • if [ -d “venv/bin” ]; then
    source venv/bin/activate
    else
    pip install virtualenv
    virtualenv venv
    source venv/bin/activate
    fi
  • pip install -r requirements.txt

prod:
tags:
- runner_tag
stage: production
only:
- schedules
script:
- python script_name.py
Thank you for you help.

2

Hi @alfons_mucha
for Docker executor the helper image should be stored locally. You can check docker images to see if it’s there and the tarball that GitLab Runner is using to import the image in case it’s missing is at /usr/lib/gitlab-runner/helper-images.
If it is only for scheduled pipeline make sure you don’t have any cleanup jobs that would remove the image from Docker in the middle of a run.

3

Hello, thank you.
still, it’s interesting that it was pulling the image until one hour timeout.
Yeah, I can see the gitlab/gitlab-runner-helper image in docker images.
What do you mean by clean up jobs? AFAIK there should be nothing like that, if you have any more tips, how to look more into it, I would appreciate it, thank you.