Error Gitlab-Runner: OpenSSL SSL_connect: Connection reset by peer in connection

Infrastructure as Code & Cloud Native
,
1

I am hoping to get my gitlab runner working but I keep getting error messages.
The Gitlab-Runner is behind a HAProxy by PfSense which is set to listen on port 3128.

Would love to know your thoughts :smiley:

Gitlab Deploy Output 
Running with gitlab-runner 14.5.2 (e91107dd)
  on <Gitlab-Runner>
Preparing the "docker" executor
Using Docker executor with image florianweinrich/laravel-deploy:7.4 ...
Pulling docker image florianweinrich/laravel-deploy:7.4 ...
Using docker image sha256:a0182e679d8eb4a926873fdae75c602989d3cb73cd7ad4c2042221e736ef7515 for florianweinrich/laravel-deploy:7.4 with digest florianweinrich/laravel-deploy@sha256:14adaf25ff270f87b463f8a71c93e738646d016fa907b97e2d99a5470132be23 ...
Preparing environment
00:01
Running on runner-qern8q8n-project-13-concurrent-0 via Gitlab-runner...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/erp/connector/.git/
fatal: unable to access 'https://<gitlab hostname>/erp/connector.git/': OpenSSL SSL_connect: Connection reset by peer in connection to <gitlab hostname>:443 
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1```
Gitlab Runner config.toml 
concurrent = 1
check_interval = 0

[session_server]
 session_timeout = 1800

[[runners]]
  name = "My-Gitlab-Runner"
  url = "https://<gitlab hostname>/"
  token = "<token>"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "ruby:2.6"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    extra_hosts=["localhost:172.18.1.23"]
2

Here’s the docs for running a Gitlab runner behind a proxy: Running GitLab Runner behind a proxy | GitLab

Your other alternatives would require using export http_proxy=“x.x.x.x:3128” in bashrc or bash_profile so that whenever a session starts, the proxy environment variable is set.

3 Likes
3

I downloaded CNTLM.

Here is the config of CNTLM

Don’t think its doing anything with the username, password and domain. So I will leave that out
Proxy [Public Ip of Proxy]:3128
Listen 127.0.0.1:3128

I sadly don’t get the other option. I tried putting it into the command prompt but other than just being able to write text, I didn’t get anywhere. I also saw a different Variant on the site you linked which I followed.

/etc/systemd/system/gitlab-runner.service.d/http-proxy.conf

[Service]
Environment=“HTTP_PROXY=http://[Public Proxy Ip]:3128/”
Environment=“HTTPS_PROXY=http://[Public Proxy Ip]:3128/”

The Status of that. Wasn't even able to pick up the deploy.

gitlab-runner.service - GitLab Runner
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/gitlab-runner.service.d
└─http-proxy.conf
Active: active (running) since Thu 2022-01-13 14:45:41 CET; 1min 15s ago
Main PID: 331 (gitlab-runner)
Tasks: 10 (limit: 2359)
Memory: 42.6M
CGroup: /system.slice/gitlab-runner.service
└─331 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

Jan 13 14:46:27 Gitlab-runner gitlab-runner[331]: WARNING: Checking for jobs
 failed runner=QERn8Q8N status=couldn’t execute POST against https://[Gitlab URL]
Jan 13 14:46:30 Gitlab-runner gitlab-runner[331]: WARNING: Checking for jobs
 failed runner=QERn8Q8N status=couldn’t execute POST against https://[Gitlab URL]
Jan 13 14:46:33 Gitlab-runner gitlab-runner[331]: WARNING: Checking for jobs
 failed runner=QERn8Q8N status=couldn’t execute POST against https://[Gitlab URL]

//Next try

Added the following lines to /etc/gitlab-runner/config.toml like the website said

pre_clone_script = "git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY"pre_clone_script = "git config --global http.proxy $HTTP_PROXY; git conf$
environment = [“https_proxy=http://:3128”, “http_proxy=http://[Proxy Public Ip]:3128”, “HTTPS_PROXY=[Proxy Public Ip]:3128”, “HTTP_PROXY=[Proxy Public Ip]:3128”]

But these stopped the gitlab runner from activating.

The Status of when gitlab runner was unable to activate

gitlab-runner.service - GitLab Runner
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/gitlab-runner.service.d
└─http-proxy.conf
Active: activating (auto-restart) (Result: exit-code) since Thu 2022-01-13 14:42:56 CET; 1min 48s ago
Process: 330 ExecStart=/usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner (code=e
Main PID: 330 (code=exited, status=1/FAILURE)

4

On the site I realized that the proxy defined by the private ip add. I have changed it and instead of the gitlab-runner failing it looked like it works.
But on the Gitlab site I just saw that it didn’t even pick up the job. I have no idea on what I should look into next. Made sure that no firewall is blocking it and everything.

Status on when its not even picking up the Job

gitlab-runner.service - GitLab Runner
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/gitlab-runner.service.d
└─http-proxy.conf
Active: active (running) since Fri 2022-01-14 11:26:02 CET; 17s ago
Main PID: 1458 (gitlab-runner)
Tasks: 9 (limit: 2359)
Memory: 11.0M
CGroup: /system.slice/gitlab-runner.service
└─1458 /usr/bin/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

Jan 14 11:26:02 Gitlab-runner systemd[1]: Started GitLab Runner.
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]: Runtime platform arch=amd64 os=linux pid=1458 revision=e91107dd version=14.5.2
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]: Starting multi-runner from /etc/gitlab-runner/config.toml
 builds=0
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]: Running in system-mode.
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]:
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]: Configuration loaded builds=0
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]: listen_address not defined, metrics & debug endpoints disabled builds=0
Jan 14 11:26:02 Gitlab-runner gitlab-runner[1458]: [session_server].listen_address not defined, session endpoints disabled builds=0

Without the /etc/systemd/system/gitlab-runner.service.d/http-proxy.conf it can pick up the job just fine.
The question that puzzles me the most is why can it pick up a job but not continue to execute it. If it picks up the job, doesn’t it mean it can connect just fine?

Status of the same Job but on the Runner itself

Jan 14 11:32:04 Gitlab-runner gitlab-runner[1498]: Checking for jobs
 received job=2048 repo_url=https://[Gitlab URL]/erp/connector.git runner=QERn8Q8D
Jan 14 11:34:10 Gitlab-runner gitlab-runner[1498]: WARNING: Job failed: exit code 1
Jan 14 11:34:10 Gitlab-runner gitlab-runner[1498]: duration_s=125.789835713 job=2048 project=13 runner=QERn8Q8D
Jan 14 11:34:10 Gitlab-runner gitlab-runner[1498]: WARNING: Failed to process runner builds=0 error=exit code 1 executor=docker runner=QERn8Q8D

Also noticed that nobody sees what I type in <> So some things may have looked wierd when I wanted to hide the public ips

5

When pinging to gitlab i get: 64 bytes from static.[ip reversed].clients.your-server.de ([ip]): icmp_seq=1 ttl=54 time=25.9 ms

Not sure if that means anything. But all I see is that it is able to connect to gitlab, take the job and then the connection gets refused for some reason.

PfSense just acts like a router going out. Going in I told the HAProxy that anything that goes to port 3128 will go to the gitlab-runner server. Is that even being used or only the gitlab-runner contacting gitlab and gitlab never contacts the runner. In that regards the proxy shouldn’t even play a role.
It definitely feels like it as I see the runner messaging gitlab if it has a new job.