please help me.
I installed a gitlab-runner (shell) instance on a server behind a corporate firewall and proxy. Proxy environment variables as well as self-signed certificates are set correctly. The runner is registered in our self-managed GitLab instance that is hosted outside our infrastructure.
sudo gitlab-runner verify I get this status message:
Runtime platform: arch=amd64 os=linux pid=33986 revision=7178588d version=15.5.1 Running in system-mode. Verifying runner... is alive runner=<my_runner_id>
When I check the status (
sudo service gitlab-runner status) of the service runner seems to have problems querying our GitLab instance:
gitlab-runner.service - GitLab Runner Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2022-11-21 13:31:35 CET; 4h 24min ago Main PID: 927 (gitlab-runner) Tasks: 14 (limit: 38353) Memory: 43.8M CPU: 2.116s CGroup: /system.slice/gitlab-runner.service └─927 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner Nov 21 17:51:36 <my> gitlab-runner: WARNING: Checking for jobs... failed runner=<my_runner_id> status='couldnt execute POST against https://<my_gitlab>/api/v4/jobs/request: Post "https://<my_gitlab>/api/v4/jobs/request": dial tcp <my_gitlab_instance_ip>:443: i/o timeout'
It seems like the runner cannot connect regularly to the endpoint.
- Asked our GitLab instance provider if there is anything wrong with their firewall. Didn’t help.
- Checked if I can reach the GitLab server in different ways from the behind-the-proxy-server where the gitlab-runner is installed: I can.
curl --globoff --header "PRIVATE-TOKEN: <my_token>" "https://<my_gitlab>/api/v4/projects/<project_id>/jobs"will produce a status code of 200. Didn’t help to improve my understanding of the problem.
- Tried various advanced configurations in the
config.tomlsuch as TLS_* or setting the proxy environment variables again. Didn’t help.
- Install a second gitlab-runner outside our company’s infrastructure where there is no proxy or firewall or whatsoever. Everything works just fine. The runner is picking up jobs as expected. Soo… did help (?!).
- Fetch jobs sucessfully manually running
sudo gitlab-runner run. That… did help (?!).
sudo gitlab-runner --debug runwill produce the following output:
5.2. Starting a job while having the runner in
[session_server].listen_address not defined, session endpoints disabled builds=0 Feeding runners to channel builds=0 Starting worker builds=0 worker=0 Dialing: tcp <my_proxy>:<my_proxy_port> ... Checking for jobs... nothing runner=<my_runner_id> Feeding runners to channel builds=0 Checking for jobs... nothing runner=<my_runner_id>
--debug run-mode will produce the following:
Checking for jobs... received job=<my_job_id> repo_url=<my_repo> runner=<my_runner_id> Processing chain chain-leaf=[0xc00068a000 0xc00068a580 0xc00068c100 0xc0004a3600] context=certificate-chain-build resolve-full-chain=true Certificate doesnt provide parent URL: exiting the loop Issuer=T-TeleSec GlobalRoot Class 2 IssuerCertURL= Serial=1 Subject=T-TeleSec GlobalRoot Class 2 context=certificate-chain-build Failed to requeue the runner builds=1 runner=<my_runner_id> Running with gitlab-runner 15.5.1 (7178588d) job=<my_job_id> project=<my_project_id> runner=<my_runner_id> on <my_runner_name> <my_runner_id> job=<my_job_id> project=<my_project_id> runner=<my_runner_id> Resolving secrets job=<my_job_id> project=<my_project_id> runner=<my_runner_id> Preparing the "shell" executor job=<my_job_id> project=<my_project_id> runner=<my_runner_id> Shell configuration: command: bash # etc etc Job succeeded duration_s=1.074760487 job=<my_job_id> project= <my_project_id> runner=<my_runner_id> Feeding runners to channel builds=0 Checking for jobs... nothing runner=<my_runner_id> Feeding runners to channel builds=0 # etc etc
What is going on??
I assume that the runner is making requests to this endpoint in a way where it cannot get across either our proxy/firewall infrastructure or something else. Unfortunately, I’m not really able to identify a solution. And looking at the output of 5.2 there is something going on with certificates (?).
- self-hosted GitLab instance: v.15.5 outside company infrastructure
- My system where I want my runner to work:
- Ubuntu 22.04 LTS, linux/amd64
- gitlab-runner v.15.5.1
- plenty of RAM and CPU
Did anyone of you encounter a similar problem or can point me in a direction where I might get a better grasp. Is there more detail logs I could explore?
Thank you in advance for your help!
All the best,