[gitlab][ce][kas] Agent behind proxy

How to Use GitLab Self-managed
,
1

Abstract

I tried to setup gitlab-kas on my gitlab server, as well as the related gitlab-agent on my kubernetes cluster. The gitlab-kas seems to work fine, but the gitlab agent on kubernetes cannot contact the gitlab api endpoints.

My gitlab is running the packaged nginx, but is ALSO behind my own nginx. I suspect that I got some configurations wrong; since other things in gitlab are working correctly.

Question: How do I need to setup my external nginx as well as the gitlab-kas package to make everything work smoothly?

Architecture

  1. Schema

    image
    image1448×653 42.3 KB

  2. Basic flow

[request] -> [external-nginx-lb: 10.1.100.68] -> [gitlab-nginx: 10.1.0.20] <-> [external postgres: 10.1.0.90]

I have an external nginx in front of everything. If you wonder why; it is due to the fact that I have a ton of services running, and that I need to loadbalance requests to those services from a single loadbalancer.

This external nginx redirects every queries to git.mydomain.com to the gitlab server running on my network. After this, the gitlab internal nginx redirect those queries to the correct path.

Here is what works:

  • Access the login / post login page :white_check_mark:
  • git related activites (pull, push, clone…) :white_check_mark:
  • Access to settings / admin pages / projects / groups … :white_check_mark:
  • Create ressources (groups, repo…) or use internal web-ide :white_check_mark:

What I try to configure

  • Gitlab kas to connect the gitlab agent to my own kubernetes cluster :x:

Configuration

Here is my current external nginx configuration:

upstream gitlab {
    server 10.1.0.20:80;  # Port 80 as GitLab's internal NGINX is listening on 80
}

server {
    listen 80;
    listen 443 ssl http2;
    server_name git.mydomain.com;

    ssl_certificate /etc/letsencrypt/live/git.mydomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/git.mydomain.com/privkey.pem;

    location /-/kubernetes-agent/ {
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://gitlab/-/kubernetes-agent/;

        # Additional configuration to handle WebSocket
        proxy_http_version 1.1;
    }

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://gitlab;
    }

    # Additional configuration to handle WebSocket
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}

# Map to ensure the $connection_upgrade variable is properly set
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

And my current gitlab-ce configuration:

external_url 'https://git.mydomain.fr'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "<redacted>"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "<redacted>"
gitlab_rails['smtp_password'] = "<redacted>"
gitlab_rails['smtp_domain'] = "mydomain.fr"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_pool'] = false
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@mydomain.fr'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@mydomain.fr'
gitlab_rails['gitlab_default_theme'] = 2
gitlab_rails['gitlab_default_projects_features_issues'] = true
gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
gitlab_rails['gitlab_default_projects_features_wiki'] = true
gitlab_rails['gitlab_default_projects_features_snippets'] = true
gitlab_rails['gitlab_default_projects_features_builds'] = true
gitlab_rails['gitlab_default_projects_features_container_registry'] = false
gitlab_rails['webhook_timeout'] = 5
gitlab_rails['trusted_proxies'] = ['10.1.100.68']
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '::1/128', '10.0.0.0/8']
gitlab_rails['shutdown_blackout_seconds'] = 10
gitlab_rails['object_store']['enabled'] = true
gitlab_rails['object_store']['connection'] = {<redacted>}
gitlab_rails['object_store']['storage_options'] = {}
gitlab_rails['object_store']['proxy_download'] = false
gitlab_rails['object_store']['objects']['artifacts']['bucket'] = 'gitlab-artifacts'
gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = 'gitlab-external-diffs'
gitlab_rails['object_store']['objects']['lfs']['bucket'] = 'gitlab-lfs'
gitlab_rails['object_store']['objects']['uploads']['bucket'] = 'gitlab-uploads'
gitlab_rails['object_store']['objects']['packages']['bucket'] = 'gitlab-packages'
gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = 'gitlab-dependency-proxy'
gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = 'gitlab-terraform-state'
gitlab_rails['object_store']['objects']['ci_secure_files']['bucket'] = 'gitlab-ci-secure-files'
gitlab_rails['object_store']['objects']['pages']['bucket'] = 'gitlab-pages'
gitlab_rails['impersonation_enabled'] = true
gitlab_rails['disable_animations'] = true
gitlab_rails['usage_ping_enabled'] = true
gitlab_rails['backup_keep_time'] = 604800
gitlab_rails['backup_upload_connection'] = {<redacted>}
gitlab_rails['backup_upload_remote_directory'] = 'gitlab-backups'
gitlab_rails['backup_multipart_chunk_size'] = 104857600
gitlab_rails['env'] = {
    "SKIP" => "db,uploads,repositories,builds,artifacts,lfs,registry,pages"
}
gitlab_rails['gitaly_token'] = '<redacted>'
gitlab_rails['rack_attack_git_basic_auth'] = {
  'enabled' => true,
  'ip_whitelist' => ["127.0.0.1","10.0.0.0/8"],
  'maxretry' => 5,
  'findtime' => 60,
  'bantime' => 3600
}

gitlab_rails['initial_root_password'] = "<redacted>"
gitlab_rails['initial_shared_runners_registration_token'] = "<redacted>"

# DB
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"
gitlab_rails['db_database'] = "gitlab_db"
gitlab_rails['db_username'] = "user"
gitlab_rails['db_password'] = "<redacted>"
gitlab_rails['db_host'] = "10.1.0.90"
gitlab_rails['db_port'] = 6432
gitlab_rails['db_sslmode'] = "require"
gitlab_workhorse['enable'] = true


user['shell'] = "/bin/bash"
puma['enable'] = true
sidekiq['enable'] = true
postgresql['enable'] = false
redis['enable'] = true

# NGINX
web_server['external_users'] = ['www-data']
nginx['enable'] = true
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['listen_addresses'] = ['0.0.0.0']
nginx['real_ip_trusted_addresses'] = ['10.1.100.68']
nginx['real_ip_header'] = 'X-Real-IP'
nginx['real_ip_recursive'] = 'on'
logrotate['enable'] = true

manage_accounts['enable'] = true
gitlab_pages['enable'] = false
pages_nginx['enable'] = false

# CI
gitlab_ci['gitlab_ci_all_broken_builds'] = true
gitlab_ci['gitlab_ci_add_pusher'] = true
gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'

# KAS
gitlab_rails['gitlab_kas_enabled'] = true
gitlab_rails['gitlab_kas_external_url'] = 'ws://git.mydomain.fr/-/kubernetes-agent/'
gitlab_rails['gitlab_kas_internal_url'] = 'grpc://localhost:8153'
gitlab_kas['enable'] = true
gitlab_kas['listen_websocket'] = true
mattermost['enable'] = false
prometheus['enable'] = true
<prometheus-configs>
storage_check['enable'] = false
storage_check['target'] = 'unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
storage_check['log_directory'] = '/var/log/gitlab/storage-check'
letsencrypt['enable'] = false
gitlab_rails['dependency_proxy_enabled'] = false

Notes

I tried various configurations setups:

  • use unix socket as recommended in the gitlab doc here
  • Use or not use the location /-/kubernetes-agent in my external nginx
  • Try adding/removing gitlab_kas["gitlab_url"] = "http(s)://git.mydomain.com" (cannot find the link where someone suggested to add this) in the gitlab.rb configuration file
  • Try various configurations with/without wss / ws with/without unix, with/without the external nginx virtual server block for kubetrnetes agent.

Description

When running the gitlab agent on my kubernetes cluster, I got one of those error logs in my kubernetes cluster, depending on the configuration setup I tried :

  • A timeout with this logs :
{"level":"error","time":"2024-05-20T11:27:40.420Z","msg":"Unable to check if CRD is installed","mod_name":"flux","k8s_group":"source.toolkit.fluxcd.io","error":"unable to get CRD source.toolkit.fluxcd.io/v1, Resource=gitrepositories: customresourcedefinitions.apiextensions.k8s.io \"gitrepositories.source.toolkit.fluxcd.io\" is forbidden: User \"system:serviceaccount:gitlab-runner:gitlab agent\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"}                                                            
{"level":"info","time":"2024-05-20T11:27:40.420Z","msg":"Flux is not installed, skipping module. A restart is required for this to be checked again","mod_name":"flux"}                                              
{"level":"info","time":"2024-05-20T11:27:40.423Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":":8080"} # <=== Timeout here
  • A websocket error
{"level":"info","time":"2024-05-20T08:59:59.210Z","msg":"Observability endpoint is up","mod_name":"observability","net_network":"tcp","net_address":":8080"}
{"level":"error","time":"2024-05-20T08:59:59.255Z","msg":"Failed to register agent pod. Please make sure the agent version matches the server version","mod_name":"agent_registrar","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to WebSocket dial: expected handshake response status code 101 but got 426\""}
{"level":"error","time":"2024-05-20T09:00:16.933Z","msg":"Failed to register agent pod. Please make sure the agent version matches the server version","mod_name":"agent_registrar","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to WebSocket dial: expected handshake response status code 101 but got 426\""}

Versions

  • Self-managed
  • Gitlab-ce
  • 17.0.0 (also had the same problem on 16.11.0)

Already checked resources

What doc / issues / forum I already looked upon to try to solve my problem :

  1. Need help setting up gitlab-kas behind proxy
  2. KAS setup is broken in v16.11.1, v16.10.4, and v16.9.6 upgrade (#458462) · Issues · GitLab.org / GitLab · GitLab
  3. Install the GitLab agent server for Kubernetes (KAS) | GitLab
  4. websocket - Gitlab Kubernetes Agent behind Nginx Reverse Proxy - Stack Overflow
  5. Reddit - Dive into anything
  6. nginx - Gitlab CE - Cannot connect with Gitlab Kubernetes Agent Server - Stack Overflow
  7. Troubleshooting docs: Self-managed GitLab instances.
2

Note:

I also found this logs in the gitlab-kas/current logs:

2024-05-20_17:23:10.36839 {"level":"error","time":"2024-05-20T17:23:10.368Z","msg":"AgentInfo()","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://git.mydomain.com/api/v4/internal/kubernetes/agent_info\": dial tcp <public-ip>:443: i/o timeout"}
3

Hello eveyone,

I just figured out what my problem was.

I have a firewall that handle inter-vlan communications, device-to-device communication, port filtering …
I had setup those rules:

  • ALLOW: nginx (vlan 1)-> gitlab (vlan 2) for http, https, and ssh
  • ALLOW: gitlab (vlan 2)-> nginx (vlan 1) for http, https, and ssh`
  • ALLOW: related, established
  • BLOCK: everything else

I was going mad over my problem, and figured I would re-check step by step every part of my network communication to be sure all my FW rules where fine.

To simplify, I kept the rules, but remove the filtering on the specific HTTP, HTTPS and SSH port. To my surprise, my kubectl logs that was running at the time started to show a handfull of logs that were not here at that rate before.

I realised that I was filtering websocket ports KAS was trying to run on… Now I just need to open the correct port for KAS in the port filtering and everything will work !

1 Like