Gitlab behind proxy: how to log external IP addresses

Gitlab runs with docker-compose and there is a docker registry and two runners.

Currently I use haproxy as proxy, with SSL offloading taking place in gitlab (haproxy runs in tcp mode).

The two runners have to be able to access a service that provides secrets to images running in the CI/CD pipelines of some projects.Therefore I configured gitlab, the runners and said service to share a common network, so I have an entry:
network mode: gitlab_network in the config.toml of the runners.

The problem I encounter is, that the IPs logged by the nginx inside gitlab are always equal to the IP of the docker-container of gitlab itself. This is, because the original IPs seem not to be passed on in tcp mode.

I found out about proxy_protocol and configured nginx according to the documentation here.

I added this in the haproxy backend configuration:

server server1 send-proxy-v2

and this in the omnibus config, as documented:

nginx['proxy_protocol'] = true
nginx['real_ip_trusted_addresses'] = [ "IP_OF_THE_PROXY/32"]

This does lead to the external IPs being correctly logged, when gitlab is accessed.

However, the runners now cannot reach their destination url. In the logs I see error messages like this:

gitlab-runner_1  | Dialing: tcp gitlab.mydomain.tld:443 ...          
gitlab_1         | 
gitlab_1         | ==> /var/log/gitlab/nginx/gitlab_error.log <==
gitlab_1         | 2023/03/31 06:37:55 [error] 687#0: *44082 broken header: "�"��[���_�'�9�A�Jƕbj���Vt'� K�Q���@!X�;��Ʌ�Ƈe�*o̷A�Q/�&�+�/�,�0̨̩�	��
gitlab-runner_1  | WARNING: Checking for jobs... failed                runner=G7ZWES2_ status=couldn't execute POST against https://gitlab.mydomain.tld/api/v4/jobs/request: Post "https://gitlab.mydomain.tld/api/v4/jobs/request": read tcp> read: connection reset by peer
gitlab_1         | ���/5�" while reading PROXY protocol, client:, server:
^CERROR: Aborting.

So it looks like the runner tries to access nginx 1. directly, 2. without using proxy_protocol.

Is there a way to solve this by somehow reconfiguring the runners?

Or are there any other suggestions concerning the setup as a whole?

Any hints are welcome!


external_url "https://${DOMAIN}"
        letsencrypt['enable'] = false
        gitlab_rails['smtp_enable'] = true
        gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.tld'
        gitlab_rails['smtp_address'] = "mail.mydomain.tld"
        gitlab_rails['smtp_port'] = 25
        nginx['proxy_protocol'] = true
        nginx['listen_https'] = true
        nginx['ssl_certificate'] = "/etc/gitlab/ssl/${DOMAIN}.crt"
        nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/${DOMAIN}.key"
        nginx['http2_enabled'] = false
        nginx['proxy_set_headers'] = {
          "Host" => "${DOMAIN}",
        nginx['real_ip_trusted_addresses'] = [ 'myip/32' ]
        nginx['real_ip_header'] = 'X-Forwarded-For'
        nginx['real_ip_recursive'] = 'on'
        gitlab_rails['omniauth_providers'] = [
            "name" => "github",
            "app_id" => "xyz",
            "app_secret" => "xyz",
            "args" => { "scope" => "user:email" }
        gitlab_rails['packages_enabled'] = true
        gitlab_rails['lfs_enabled'] = true
        gitlab_rails['registry_port'] = ${REGISTRYPORT}
        gitlab_rails['registry_host'] = "${DOMAIN}"
        gitlab_rails['backup_keep_time'] = 86400
        gitlab_rails['gitlab_ssh_host'] = 'gitlab.mydomain.tld'
        gitlab_rails['gitlab_shell_ssh_port'] = mysshport
        registry['enable'] = true
        registry_external_url "https://${DOMAIN}:${REGISTRYPORT}"
        registry_nginx['redirect_http_to_https'] = true
        registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/${DOMAIN}.crt"
        registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/${DOMAIN}.key"
        puma['worker_processes'] = 2
        puma['per_worker_max_memory_mb'] = 2024
        sidekiq['max_concurrency'] = 10
        postgresql['shared_buffers'] = "500MB"
        prometheus['listen_address'] = '${PROMETHEUSPORT}'

And the config.toml for the runner:

concurrent = 2
check_interval = 0
log_level = "debug"
shutdown_timeout = 0

  session_timeout = 1800

  name = "cicd"
  url = "https://gitlab.mydomain.tld"
  id = 18
  token = "token"
  token_obtained_at = 2023-03-09T13:51:01Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
    MaxUploadedArchiveSize = 0
    tls_verify = false
    image = "docker:20.10.23"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/srv/gitlab-runner/shared:/shared_cache:rw", "/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
    network_mode = "gitlab_network"

You need to have haproxy set the X-Forwarded-For header (haproxy docs).