Gitlab-runner connection refused

Hi, whenever I try to run a pipeline on gitlab it fails with the following error:
Fetching changes with git depth set to 50... Initialized empty Git repository in /builds/lukas/test/.git/ Created fresh repository. fatal: unable to access 'http://<ip>:999/lukas/test.git/': Failed to connect to <ip> port 999: Connection refused ERROR: Job failed: exit code 1

On my system I have installed docker and use the docker executer with the alpine:latest image. I did also try to use the shell executor and get the same issue so it seems not to have to do with docker.
Connection refused for me sounds like a firewall problem…? It might be from importance that the gitlab service runns on the same machine as the gitlab-runner. It does not make a difference if it runs on another machine though.
My operating system is CentOS 7.

Has anyone an idea what I can do to solve this problem?

Is GitLab running also under Docker?
Is a git clone or fetch with the same repos. on the machine working?

In the CI file set these variables which will make the git call more verbose:

variables:
  GIT_CURL_VERBOSE: 1
  GIT_TRACE: 1

Thanks for your answer! Actually git clone does NOT work on the machine, so it must be a firewall issue, doesn’t it?
What would be possible problems? Port 8080 is the only open port and http is the only allowed protocol. I cloned via http so it should not be the protocol…?

GitLab pipelines using only HTTP or HTTPS depending on your setup. So port 80 or 443 must be open in your firewall. Port 8080 is not relevant from my POV.

You should first get git clone directly on the host running were your shell executor should run.

Gitlab is setup using port 999 on the local machine but port 8080 in the internet (forwarded 8080 to 999). When I try to clone from 127.0.0.1:999 it works just fine but if I try to clone from the public ip address on port 8080 it fails with the same error.
Can I configure from which ip address it clones? Or do you know anything else that might cause this issue (e.g. firewalld) @nightman68

Are you able accessing the GitLab Web frontend via the Internet URL / IP?

I am accessing the GitLab Web frontend via Internet IP.

Then you have to use the IP for a git clone, e.g. git clone http(s)://IP_ADDRESS:8080/REPOS-NAME.git - is this working?

No that is why I think it is a firewall problem.
When I use “ip” in the following i mean the public ip which I sucessfully can use the GitLab Web fronted over.
Cloning from ip:8080 does not work.
Cloning from ip:999 does not work.
Cloning from 127.0.0.1:999 does work.

This is a bit confusing, sorry. When you’re able accessing the Web frontend externally you see the URL for git clone on the project home page, top right. When this is identical with the one in your Web browser for the GitLab Web frontend it must work - it’s the same network route!

That’s true, very confusing, i don’t get why either.
The clone command that is shown on the top right works from any machine except my machine that gitlab runs on.
And yes, the clone command has the same ip address & port number as the web frontend.

I’m pretty sure now that the firewall is the problem because curl to the public ip address also doesn’t work from this machine, while using curl on the localhost ip address works.

Have you tried to set the 2 env. var. below in the shell before clone with the Internet IP?

export GIT_CURL_VERBOSE=1
export GIT_TRACE=1
git clone ...

I just tried, same error @nightman68

Yes but you should see a lot of message related to the network traffic etc.

BTW: is the clone working from these other machines with external IP?

Yes it is working from these other maschines with external IP.
I do get more info:

grafik842×216 5.97 KB

In the masked URL from the clone command is there the port 999 present? It is in the error message included…

When you configure the external URL to listen on 8080 you must use this one.

Have you tried to move your $HOME/.gitconfig file away that it is re-created?

I did not want to bother you by explaining why 8080 and 999 is used as port.
I use internal port forwarding from port 8080 to 999 because port 8080 is alread used by another service. So publically I have to use port 8080 or 999 (because 999 is also forwarded, most of the users use port 8080) but internally I have to use 999.
I just added the --verbose option to the command - it still doesn’t show more output… I do not have a .gitconfig at all…
But maybe I used curl to collect trace information - and I do not even copy them in because there is nothing more than already known… it says failed to connect and connection refused

It must be a networking problem not a git problem.

The final solution was to just change the /etc/gitlab-runner/build/config.toml file. A property ‘clone_url’ can be added to change the url that should be used to clone from git.
https://docs.gitlab.com/runner/configuration/advanced-configuration.html#how-clone_url-works

Setting this property to the local address of the machine was the solution. 127.0.0.1 though did not work because this is executed in a docker machine. Instead use the local address of the real machine, e.g. 192.168.1.10

@lukaswindisch21 Nice solution!
I know some time is passed by, but, could you please explain how to get that ip address (192.168.1.10)?