ssh: connect to host port 22: Connection refused

Hey, I have my own V-Server on which only gitlab should run. So I followed the installation of Gitlab for Ubuntu 22.04 and everything worked fine. Then I added an SSH key and wanted to push a repo and then the following error:

git push --set-upstream origin master
ssh: connect to host port 22: Connection refused
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The port of my V-Server is 23, I changed it because I thought that git uses port 22 and so my server cannot use it at the same time, but then I get the message above. If I change the port of my V-Server back to 22 and then enter the command again. Then the terminal asks me for a password for the git user. What can I do about this?

Here is my config

external_url ''
letsencrypt['enable'] = false
nginx['ssl_certificate'] = "/etc/letsencrypt/live/"
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/"
gitlab_rails['gitlab_shell_ssh_port'] = 22