Gitlab runs with docker-compose and there is a docker registry and two runners.
Currently I use haproxy as proxy, with SSL offloading taking place in gitlab (haproxy runs in tcp mode).
The two runners have to be able to access a service that provides secrets to images running in the CI/CD pipelines of some projects.Therefore I configured gitlab, the runners and said service to share a common network, so I have an entry:
network mode: gitlab_network in the config.toml of the runners.
The problem I encounter is, that the IPs logged by the nginx inside gitlab are always equal to the IP of the docker-container of gitlab itself. This is, because the original IPs seem not to be passed on in tcp mode.
I found out about proxy_protocol and configured nginx according to the documentation here.
I added this in the haproxy backend configuration:
server server1 127.0.0.1:xxx send-proxy-v2
and this in the omnibus config, as documented:
nginx['proxy_protocol'] = true
nginx['real_ip_trusted_addresses'] = [ "IP_OF_THE_PROXY/32"]
This does lead to the external IPs being correctly logged, when gitlab is accessed.
However, the runners now cannot reach their destination url. In the logs I see error messages like this:
gitlab-runner_1 | Dialing: tcp gitlab.mydomain.tld:443 ...
gitlab_1 |
gitlab_1 | ==> /var/log/gitlab/nginx/gitlab_error.log <==
gitlab_1 | 2023/03/31 06:37:55 [error] 687#0: *44082 broken header: "�"��[���_�'�9�A�Jƕbj���Vt'� K�Q���@!X�;��Ʌ�Ƈe�*o̷A�Q/�&�+�/�,�0̨̩� ��
gitlab-runner_1 | WARNING: Checking for jobs... failed runner=G7ZWES2_ status=couldn't execute POST against https://gitlab.mydomain.tld/api/v4/jobs/request: Post "https://gitlab.mydomain.tld/api/v4/jobs/request": read tcp 192.168.16.3:36282->192.168.16.2:443: read: connection reset by peer
gitlab_1 | ���/5�" while reading PROXY protocol, client: 192.168.16.3, server: 0.0.0.0:443
^CERROR: Aborting.
So it looks like the runner tries to access nginx 1. directly, 2. without using proxy_protocol.
Is there a way to solve this by somehow reconfiguring the runners?
Or are there any other suggestions concerning the setup as a whole?
Any hints are welcome!
Below my GITLAB_OMNIBUS_CONFIG:
external_url "https://${DOMAIN}"
letsencrypt['enable'] = false
gitlab_rails['smtp_enable'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.tld'
gitlab_rails['smtp_address'] = "mail.mydomain.tld"
gitlab_rails['smtp_port'] = 25
nginx['proxy_protocol'] = true
nginx['listen_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/${DOMAIN}.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/${DOMAIN}.key"
nginx['http2_enabled'] = false
nginx['proxy_set_headers'] = {
"Host" => "${DOMAIN}",
}
nginx['real_ip_trusted_addresses'] = [ 'myip/32' ]
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
gitlab_rails['omniauth_providers'] = [
{
"name" => "github",
"app_id" => "xyz",
"app_secret" => "xyz",
"args" => { "scope" => "user:email" }
}
]
gitlab_rails['packages_enabled'] = true
gitlab_rails['lfs_enabled'] = true
gitlab_rails['registry_port'] = ${REGISTRYPORT}
gitlab_rails['registry_host'] = "${DOMAIN}"
gitlab_rails['backup_keep_time'] = 86400
gitlab_rails['gitlab_ssh_host'] = 'gitlab.mydomain.tld'
gitlab_rails['gitlab_shell_ssh_port'] = mysshport
registry['enable'] = true
registry_external_url "https://${DOMAIN}:${REGISTRYPORT}"
registry_nginx['redirect_http_to_https'] = true
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/${DOMAIN}.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/${DOMAIN}.key"
puma['worker_processes'] = 2
puma['per_worker_max_memory_mb'] = 2024
sidekiq['max_concurrency'] = 10
postgresql['shared_buffers'] = "500MB"
prometheus['listen_address'] = '0.0.0.0:${PROMETHEUSPORT}'
And the config.toml for the runner:
concurrent = 2
check_interval = 0
log_level = "debug"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "cicd"
url = "https://gitlab.mydomain.tld"
id = 18
token = "token"
token_obtained_at = 2023-03-09T13:51:01Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
tls_verify = false
image = "docker:20.10.23"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/srv/gitlab-runner/shared:/shared_cache:rw", "/cache", "/var/run/docker.sock:/var/run/docker.sock"]
shm_size = 0
network_mode = "gitlab_network"