Hi everyone,

please help me.

The problem

I installed a gitlab-runner (shell) instance on a server behind a corporate firewall and proxy. Proxy environment variables as well as self-signed certificates are set correctly. The runner is registered in our self-managed GitLab instance that is hosted outside our infrastructure.

When I sudo gitlab-runner verify I get this status message:

Runtime platform:     arch=amd64 os=linux pid=33986 revision=7178588d 
version=15.5.1
Running in system-mode.                                                                              
Verifying runner... is alive      runner=<my_runner_id>

When I check the status (sudo service gitlab-runner status) of the service runner seems to have problems querying our GitLab instance:

gitlab-runner.service - GitLab Runner
  Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
  Active: active (running) since Mon 2022-11-21 13:31:35 CET; 4h 24min ago
  Main PID: 927 (gitlab-runner)
    Tasks: 14 (limit: 38353)
   Memory: 43.8M
      CPU: 2.116s
   CGroup: /system.slice/gitlab-runner.service
           └─927 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

Nov 21 17:51:36 <my> gitlab-runner[927]: WARNING: Checking for jobs... failed                runner=<my_runner_id> 
status='couldnt execute POST against https://<my_gitlab>/api/v4/jobs/request: Post "https://<my_gitlab>/api/v4/jobs/request": dial tcp <my_gitlab_instance_ip>:443: i/o timeout'

It seems like the runner cannot connect regularly to the endpoint.

What I tried so far

1. Asked our GitLab instance provider if there is anything wrong with their firewall. Didn’t help.
2. Checked if I can reach the GitLab server in different ways from the behind-the-proxy-server where the gitlab-runner is installed: I can. curl --globoff --header "PRIVATE-TOKEN: <my_token>" "https://<my_gitlab>/api/v4/projects/<project_id>/jobs" will produce a status code of 200. Didn’t help to improve my understanding of the problem.
3. Tried various advanced configurations in the config.toml such as TLS_* or setting the proxy environment variables again. Didn’t help.
4. Install a second gitlab-runner outside our company’s infrastructure where there is no proxy or firewall or whatsoever. Everything works just fine. The runner is picking up jobs as expected. Soo… did help (?!).
5. Fetch jobs sucessfully manually running sudo gitlab-runner run. That… did help (?!).
   5.1. Running sudo gitlab-runner --debug run will produce the following output:

   [session_server].listen_address not defined, session endpoints disabled  builds=0
   Feeding runners to channel                          builds=0
   Starting worker                                     builds=0 worker=0
   Dialing: tcp <my_proxy>:<my_proxy_port> ...   
   Checking for jobs... nothing                        runner=<my_runner_id>
   Feeding runners to channel                          builds=0
   Checking for jobs... nothing                        runner=<my_runner_id>
   

   5.2. Starting a job while having the runner in --debug run-mode will produce the following:

   Checking for jobs... received                    job=<my_job_id>
   repo_url=<my_repo> runner=<my_runner_id>
   Processing chain                                 chain-leaf=[0xc00068a000 0xc00068a580 0xc00068c100 0xc0004a3600] context=certificate-chain-build resolve-full-chain=true
   
   Certificate doesnt provide parent URL:           exiting the loop  Issuer=T-TeleSec GlobalRoot Class 2 IssuerCertURL=[] Serial=1 Subject=T-TeleSec GlobalRoot Class 2 context=certificate-chain-build
   Failed to requeue the runner                     builds=1 runner=<my_runner_id>
   
   Running with gitlab-runner 15.5.1 (7178588d)     job=<my_job_id> project=<my_project_id> runner=<my_runner_id>
   on <my_runner_name> <my_runner_id>               job=<my_job_id> project=<my_project_id> runner=<my_runner_id>
   Resolving secrets                                job=<my_job_id> project=<my_project_id> runner=<my_runner_id>
   Preparing the "shell" executor                   job=<my_job_id> project=<my_project_id> runner=<my_runner_id>
   Shell configuration: command: bash
   # etc etc 
   
   Job succeeded                                    duration_s=1.074760487 job=<my_job_id> project=    <my_project_id> runner=<my_runner_id>
   Feeding runners to channel                       builds=0
   Checking for jobs... nothing                     runner=<my_runner_id>
   Feeding runners to channel                       builds=0
   # etc etc
   

What is going on??
I assume that the runner is making requests to this endpoint in a way where it cannot get across either our proxy/firewall infrastructure or something else. Unfortunately, I’m not really able to identify a solution. And looking at the output of 5.2 there is something going on with certificates (?).

Versions

• self-hosted GitLab instance: v.15.5 outside company infrastructure
• My system where I want my runner to work:
  • Ubuntu 22.04 LTS, linux/amd64
  • gitlab-runner v.15.5.1
  • plenty of RAM and CPU

Did anyone of you encounter a similar problem or can point me in a direction where I might get a better grasp. Is there more detail logs I could explore?

Thank you in advance for your help!
All the best,
sweet_banana

Hi @sweet_banana

if you are behind proxy that does not require authentication, you can alter GitLab-Runner systemd config to add the required environment variables
Create systemd dir sudo mkdir /etc/systemd/system/gitlab-runner.service.d
Create file /etc/systemd/system/gitlab-runner.service.d/http-proxy.conf
Edit the file and add

[Service]
Environment="HTTP_PROXY=http://YOUR_PROXY:YOUR_PROXY_PORT/"
Environment="HTTPS_PROXY=http://YOUR_PROXY:YOUR_PROXY_PORT/"

Systemd stuff:

sudo systemctl daemon-reload
sudo systemctl restart gitlab-runner

@balonik

AWESOME! That’s it!! Thank you so much for your help!

(I named the file env.conf as to the official documentation: The system services of GitLab Runner | GitLab