Problem to solve
Gitlab docker behind Traefik through Cloudflare tunnel
Gitlab isn’t loading when I try to log in through my domain, and I get an Error 422 when I try to log in locally
Steps to reproduce
Install gitlab docker using docker-compose.yml:
GitLab docker-compose.yml:
services:
gitlab:
image: gitlab/gitlab-ce:latest
container_name: gitlab
restart: always
init: true
hostname: lab.**DOMAIN**
environment:
GITLAB_HOME: '/mnt/cloud/gitlab'
GITLAB_OMNIBUS_CONFIG: |
web_server['username'] = 'git'
external_url 'https://lab.**DOMAIN**'
nginx['enable'] = true
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['error_log_level'] = "debug"
nginx['proxy_set_headers'] = {
"X-Forwarded-Proto" => "https",
"X-Forwarded-Ssl" => "on"
}
gitlab_rails['gitlab_shell_ssh_port'] = 2224
letsencrypt['enable'] = false
ports:
- '8929:80'
- '9443:443'
- '2224:22'
labels:
- traefik.enable=true
- traefik.http.routers.gitlab_insecure.entrypoints=web
- traefik.http.routers.gitlab_insecure.rule=Host(`lab.**DOMAIN**`)
- traefik.http.routers.gitlab.entrypoints=websecure
- traefik.http.routers.gitlab.rule=Host(`lab.**DOMAIN**`)
- traefik.http.services.gitlab.loadbalancer.server.port=80
- traefik.docker.network=traefik-proxy
- traefik.tcp.routers.gitlab-ssh.entrypoints=ssh
- traefik.tcp.routers.gitlab-ssh.service=gitlab-ssh-svc
- traefik.tcp.services.gitlab-ssh-svc.loadbalancer.server.port=22
volumes:
- '/etc/gitlab/ssl:/home/*username*/gitlab:ro'
- '$GITLAB_HOME/config:/etc/gitlab'
- '$GITLAB_HOME/logs:/var/log/gitlab'
- '$GITLAB_HOME/data:/var/opt/gitlab'
networks:
- traefik_proxy
shm_size: '2g'
networks:
traefik_proxy:
external: true
driver: bridge
Put gitlab.yml into traefik’s dynamic settings:
Traefik gitLab.yml file:
http:
routers:
gitlab:
rule: "Host(`lab.**DOMAIN**`)"
entryPoints:
- "https"
service: "gitlab"
middlewares:
- gitlab-redirect
tls:
certResolver: "letsencrypt"
middlewares:
gitlab-redirect:
redirectScheme:
scheme: https
# permanent: true
services:
gitlab:
loadBalancer:
servers:
- url: "http://gitlab:80"
tcp:
routers:
gitlab-ssh:
entryPoints: ["ssh"]
rule: "HostSNI(`*`)"
service: "gitlab-ssh"
services:
gitlab-ssh:
loadBalancer:
servers:
- address: "gitlab:22"
here’s the Traefik docker-compose.yml just in case it’s needed:
Traefik Docker-compose.yml:
services:
traefik:
image: traefik:latest
container_name: traefik
restart: unless-stopped
command:
- "--log.level=INFO"
- "--api.insecure=true"
- "--api.dashboard=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=true"
- "--entrypoints.web.address=:80"
- "--entrypoints.websecure.address=:443"
- "--entrypoints.websecure.http.tls=true"
- "--providers.file.directory=/etc/traefik/dynamic"
- "--providers.file.watch=true"
environment:
- CLOUDFLARE_DNS_API_TOKEN=[[REDACTED]]
ports:
- "1080:80"
- "1443:443"
- "1888:8080"
networks:
- traefik_proxy
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /etc/traefik/static:/etc/traefik/static:ro
- /etc/traefik/dynamic:/etc/traefik/dynamic:ro
- /etc/ssl/certs:/etc/ssl/certs:ro
networks:
traefik_proxy:
external: true
driver: bridge
Here’s the message I get from curl -lv domain inside and outside of the container:
root@lab:/# curl -lv lab.**DOMAIN**
* Host lab.**DOMAIN**:80 was resolved.
* IPv6: (none)
* IPv4: 172.18.0.5
* Trying 172.18.0.5:80...
* Connected to lab.**DOMAIN** (172.18.0.5) port 80
* using HTTP/1.x
> GET / HTTP/1.1
> Host: lab.**DOMAIN**
> User-Agent: curl/8.11.1-DEV
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 302 Found
< Server: nginx
< Date: Fri, 14 Feb 2025 19:19:37 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: no-cache
< Content-Security-Policy:
< Location: https://lab.**DOMAIN**/users/sign_in
< Permissions-Policy: interest-cohort=()
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Gitlab-Meta: {"correlation_id":"01JM2YDXD95CJVH4GJX73XBXRZ","version":"1"}
< X-Permitted-Cross-Domain-Policies: none
< X-Request-Id: 01JM2YDXD95CJVH4GJX73XBXRZ
< X-Runtime: 0.034121
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
< Strict-Transport-Security: max-age=63072000
< Referrer-Policy: strict-origin-when-cross-origin
<
* Connection #0 to host lab.**DOMAIN** left intact
<html><body>You are being <a href="https://lab.**DOMAIN**/users/sign_in">redirected</a>.</body>
user@server: curl -lv lab.**DOMAIN**
* Trying 2606:4700:3037::ac43:d6c4:80...
* Connected to lab.**DOMAIN** (2606:4700:3037::ac43:d6c4) port 80 (#0)
> GET / HTTP/1.1
> Host: lab.**DOMAIN**
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Date: Fri, 14 Feb 2025 19:22:19 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< cf-cache-status: DYNAMIC
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=%2Fb1YVhovvdKuqIxqtYW1wdKJTwSep3zuD8jj3xAte3EO%2FuuKGzDfsNtFbn0NVLPYKms0CHLrQYrQlwzxo1mRJrUJJj92d%2BkYB%2FvSk35aPv9NxQlRgYceT946HPG%2Bx7S6Rz5nmGL%2FMZ7jaqGay1grjIWyJRE%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 911f6d1d7d747ad6-SJC
< alt-svc: h3=":443"; ma=86400
< server-timing: cfL4;desc="?proto=TCP&rtt=23967&min_rtt=23967&rtt_var=11983&sent=1&recv=3&lost=0&retrans=0&sent_bytes=0&recv_bytes=85&delivery_rate=0&cwnd=250&unsent_bytes=0&cid=0000000000000000&ts=0&x=0"
<
* Connection #0 to host lab.**DOMAIN** left intact
and here’s what I get when I try to log into the site:
My domain:
from server:port:
I’ve tried setting the timezone in the gitlab container by changing the value in /etc/timezone and /etc/localtime to America/Los_Angeles since I’m in PST, but this is the message I get when I use gitlab:env:info --trace:
** Invoke gitlab:env:info (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
rake aborted!
ArgumentError: Invalid Timezone: UTC
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/core_ext/time/zones.rb:84:in `find_zone!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/railtie.rb:93:in `block in <class:Railtie>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:32:in `instance_exec'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:32:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:61:in `block in run_initializers'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:228:in `block in tsort_each'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:431:in `each_strongly_connected_component_from'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:349:in `block in each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:347:in `each'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:347:in `call'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:347:in `each_strongly_connected_component'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:226:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/3.2.0/tsort.rb:205:in `tsort_each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/initializable.rb:60:in `run_initializers'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/application.rb:372:in `initialize!'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/opt/gitlab/embedded/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38:in `require'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/application.rb:348:in `require_environment!'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/railties-7.0.8.7/lib/rails/application.rb:506:in `block in run_tasks_blocks'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/helpers.rake:7:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:25:in `load'
/opt/gitlab/embedded/bin/rake:25:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/cli.rb:455:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/cli.rb:35:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/cli.rb:29:in `start'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/exe/bundle:28:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/bundler-2.5.11/exe/bundle:20:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => environment
I don’t know if it’s a timezone issue, I don’t understand why my traefik reverse proxy isn’t redirecting requests from my domain to gitlab, and I don’t know what else might be causing the failure to log in directly to the server through the network. I’ve cleared cookies and site data, I’ve done a password reset in case that was the issue, and I’ve tried logging in through other devices and browsers to no avail, I either get a blank page or a 404 error.