Local GitLab runner freezes while Shared GitLab.com runner succeeds

I am developing a Python program using a conda environment. I configured the following .gitlab-ci.yml file for it:

image: continuumio/miniconda3:latest

before_script:
  # Update conda and create environment, which is then activated.
  - conda update -vvv -y -c conda-forge conda
  - conda env create -f helpers/NAME.yml
  - source activate NAME
  # Correct installation.
  - conda install -q -y gsl=2.2.1

pages:
  script:
    # Install make.
    - apt-get update
    - apt-get install -q -y build-essential
    # Install Spinx-related packages.
    - conda install -q -y sphinx sphinx_rtd_theme
    # Create documentation.
    - cd REPO/doc
    - sphinx-apidoc -o source/ ../REPO --force --separate
    - make html
    # Transfer documentation to public pages folder.
    - mv build/html/ ../../public/
  artifacts:
    paths:
      - public
#   only:
#     - master

Running this script with a shared GitLab runner that is supplied with GitLab.com works and the documentation is generated and placed in the public folder.

For future unit tests (which take much longer), I want to provide a local runner on a Win 10 machine in my network. For this, I installed the gitlab-runner.exe and Docker Desktop. I successfully registered the runner with the project on GitLab.com.

The runner is using the following config.toml configuration file:

concurrent = 1
check_interval = 0
log_level = "info"

[session_server]
  session_timeout = 1800

[[runners]]
  name = "NAME"
  url = "https://gitlab.com"
  token = "TOKEN"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

The problem is now that the local runner freezes during the execution of the above script without producing any error messages and I am at a loss on how to debug it. What I have is

  1. The log of the script that is shown on the Job page on GitLab.com; and
  2. The console output of the gitlab-runner.exe on the local machine.

Regarding 1., I see

e[0KRunning with gitlab-runner 11.10.0 (3001a600)
...
e[32;1mChecking out COMMIT_HASH as BRANCH_NAME...e[0;m
...
e[0Ke[32;1m$ conda update -vvv -y -c conda-forge condae[0;m
DEBUG conda.gateways.logging:set_verbosity(148): verbosity set to 3
...
...
...
TRACE conda.gateways.disk.update:rename(52): renaming /opt/conda/share/doc/openssl/html/man3/OSSL_STORE_LOADER_new.html => /opt/conda/share/doc/openssl/html/man3/OSSL_STORE_LOADER_new.html.c~
TRACE conda.core.path_actions:execute(1041): renaming share/doc/openssl/html/man3/OSSL_STORE_LOADER_set_close.html => share/doc/openssl/html/man3/OSSL_STORE_LOADER_set_close.html.c~
TRACE conda.gateways.disk.update:rename(52): renaming /opt/conda/share/doc/openssl/html/man3/OSSL_STORE_LOADER_set_close.html => /opt/conda/share/doc/openssl/html/man3/OSSL_STORE_LOADER_set_close.html.c~
TRACE conda.core.path_actions:execute(1041): renaming share/doc/openssl/html/man3/OSSL_STORE_LOADER_set_ctrl.html => share/doc/openssl/html/man3/OSSL_STORE_LOADER_set_ctrl.html.c~

where it abruptly stops without reaching the - conda env create -f helpers/NAME.yml line.

Regarding 2., I see

C:\GitLab-Runner>gitlab-runner.exe --debug run
Runtime platform                                    arch=amd64 os=windows pid=14116 revision=3001a600 version=11.10.0Starting multi-runner from C:\GitLab-Runner\config.toml ...  builds=0
Checking runtime mode                               GOOS=windows uid=-1
Configuration loaded                                builds=0
...
Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=TOKEN
Feeding runners to channel                          builds=0
Checking for jobs... received                       job=203033130 repo_url=REPO_URL.git runner=TOKEN
...
Attaching to container HASH ...  job=203033130 project=6249897 runner=TOKEN
Starting container HASH ...  job=203033130 project=6249897 runner=TOKEN
Waiting for attach to finish HASH ...  job=203033130 project=6249897 runner=TOKEN
Waiting for container HASH ...  job=203033130 project=6249897 runner=TOKEN
Appending trace to coordinator... ok                code=202 job=203033130 job-log=0-10348 job-status=running runner=TOKEN sent-log=1801-10347 status=202 Accepted
Appending trace to coordinator... ok                code=202 job=203033130 job-log=0-19445 job-status=running runner=TOKEN sent-log=10348-19444 status=202 Accepted
...
Appending trace to coordinator... ok                code=202 job=203033130 job-log=0-933150 job-status=running runner=TOKEN sent-log=241860-933149 status=202 Accepted
Submitting job to coordinator... ok                 code=200 job=203033130 job-status= runner=TOKEN 
Submitting job to coordinator... ok                 code=200 job=203033130 job-status= runner=TOKEN 

where it seems that the switch from Appending trace to coordinator to Submitting job to coordinator happens around the time when it gets stuck.
After this, 1. is not updated with any further information and 2. is stuck in a Submitting job to coordinator loop.

Does anyone know:

  1. What the reason for the failure with a local runner could be (when the same script works with a shared runner)?
  2. What I could do to debug this problem?

Thanks and all the best,
Thomas

As pointed out by Rekovni on StackOverflow (see link), the problem was that the combination of a GitLab Runner and a Docker image does not work on a Windows machine.
Using the runner in a virtual machine with a Linux client solved the problem.

1 Like

I’m having the same issue running a local runner on a windows server registered as shell instead of docker. Any good solutions to get around this issue? I’m unable to use a gitlab.com runner or change the OS of the server the runner is used on.