Gitlab Runner doesn't update status in webinterface

Hi there,

Not sure if this is the right place to post this, but I’ll give it a shot.

I came across an issue today with a private runner I’m trying to deploy on a privately hosted Gitlab instance. As the title says: The runner doesn’t update its status in the webinterface. To be precise: When I start a CI Job, the runner is triggered and picks up a job. The web-terminal shows me what it’s doing but at some point it seems to “lose connection” and just waits indefinitely.

The point at which it loses connection seems to be random. Sometimes it will show me more output, sometimes less. But the runner on my PC always executes the container successfully.

However, I can verify that the runner has indeed executed the job by looking at the logs if my docker containers (Yeah, the executor is docker, btw). And the job has succeeded, too. It just doesn’t seem to send that information to the web interface…

  • What are you seeing, and how does that differ from what you expect to see?

I’m seeing the “loading” symbol in the “Jobs” section of my project, and it doesn’t proceed. It stays there forever.

  • Consider including screenshots, error messages, and/or other helpful visuals

Well, I’m not allowed to do that yet. :frowning:

Here’s the output of the web console:

1 Running with gitlab-runner 13.2.2 (a998cacd)
2  on is2_api:fh-precision RMLMGWz8
3 Preparing the "docker" executor
4 Using Docker executor with image ghdl/vunit:mcode-master ...
5 Pulling docker image ghdl/vunit:mcode-master ...
... <-- Becomes stuck here.
  • What version are you on? Are you using self-managed or GitLab.com?

    • GitLab (Hint: /help): GitLab Enterprise Edition 13.2.3-ee
    • Runner (Hint: /admin/runners): Self-managed, using my personal PC as runner
  • Add the CI configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml)

image: "ghdl/vunit:mcode-master"

variables:

stages:
  - build
  - test

elaborate design:
  stage: build
  script:
    - echo "Starting Elaboration!"
    - python3 test_i2s_clock_generator.py --elaborate
    - echo "Elaboration done"

compile tests:
  stage: build
  script:
    - echo "Starting compilation!"
    - python3 test_i2s_clock_generator.py --compile
    - echo "Compilation done"

run tests:
  stage: test
  script:
    - python3 test_i2s_clock_generator.py

Here’s what I added to the /etc/gitlab-runner/config.toml under [runners.docker]:

    dns = ["10.131.128.2"]
    dns_search = [""]
  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

I have tried:

  • Using shell as executor
  • Restarting the gitlab-runner
  • Restarting the Jobs/Pipelines
  • Simplyfying .gitlab-ci.yml

Thanks for taking the time to be thorough in your request, it really helps! :blush:

Sure thing.

Here’s more info in case that’s relevant:

  • Host OS: Debian 10 (Kernel 4.19.0-10)
  • Docker Version: 19.03.12, build 48a66213fe
  • Container Image: Here

I’m connected to the Gitlab server through a VPN client. Thus the change in the dns section in my config.toml file, otherwise the container wouldn’t resolve the hostname of the self-hosted gitlab instance.

I did take a look into the communication with Wireshark and that shows me that Server and Client communicate with TLS 1.3 (which is fine and expected). However, about every 2nd packet in Wireshark is either a TCP duplicate ACK, TCP Retransmission or and Out-of-Order TCP packet. It looks very messy…

Forgot to mention: Here’s the output of docker logs for the container that executed the python script:

$ echo "Starting Elaboration!"
Starting Elaboration!
$ python3 test_i2s_clock_generator.py --elaborate
... # Stuff from Python, but that succeeded.
$ echo "Elaboration done"
Elaboration done