502 Bad Gateway - Unraid | Cloudflare | NGINX

Heya folks, I hope you can help me out in my situation, as I must be missing something obvious that’s not part of the documentation. I’ve followed a few different documentations (linked below) and forum entries, but I keep getting a 502 Bad Gateway whenever I try to set the ‘external_url’ to fix gitlab linking to an internal (inaccessible) IP for tasks.

My Setup

CloudFlare domain with Origin SSL Certificate
Unraid Server with an NGINX docker and GitLab-CE docker.
GitLab-CE is set to listen to ports 7380 (HTTP) and 7343 (HTTPS)
NGINX forwards subdomain git.mydomain.org → 192.168.178.20 : 7380 (Unraid IP and docker port)
This setup, in itself, works when external_url is not set

My Issue

Password Recovery E-Mails have the URL http://unraid:9080
Tasks linked under in Issues have the URL unraid:9080/projectname/-/work_items/25

This means that half the time when I click something in gitlab, it will try to go to the internal (!) address within the docker rather than the actual domain name I’m accessing the page from.

When I set external_url to any of the following (to my understanding correct) values, it breaks with a 502 Bad Gateway error. Some say to include the port, some do it without. I tried them all and they all don’t work.

http://git.mydomain.org
http://git.mydomain.org:7380
https://git.mydomain.org
https://git.mydomain.org:7343

Configuration

NGINX

Nginx FWD500×544 14.8 KB

SYSTEM

GitLab SSL folder permissions

gitlab.rb

external_url 'https://git.mydomain.org'

# PORTS & ROUTING
nginx['ssl_certificate'] = "/etc/gitlab/ssl/mydomain.org.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mydomain.org.key"
# nginx['redirect_http_to_https'] = true #tried w+w/o
# nginx['redirect_http_to_https_port'] = 7343 # tried w+w/o
nginx['listen_addresses'] = ['*', '[::]']
nginx['listen_port'] = 7380
nginx['listen_https'] = false
nginx['referrer_policy'] = 'same-origin'
web_server['external_users'] = ['www-data']
puma['port'] = 7380

# SERVICES
nginx['enable'] = false #tried w+w/o
letsencrypt['enable'] = false
# puma['enable'] = false #tried w+w/o
gitlab_kas['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
node_exporter['enable'] = false
redis_exporter['enable'] = false
postgres_exporter['enable'] = false

# RAILS
gitlab_rails['smtp_enable'] = true
gitlab_rails['incoming_email_enabled'] = false
gitlab_rails['terraform_state_enabled'] = false

Things I’ve Tried

Various different attempts described on the Forum as well as the Documentations, all of which seem to have some variation on what to do.

Setting only the ‘external_url’, disabling nginx, disabling puma, setting gitlab_rails[‘internal_api_url’], setting web_server[‘external_users’] = [‘www-data’], nginx[‘redirect_http_to_https’], specifying the ports … all in various combinations.
Whatever I tried, the result was always the same: 502 bad gateway

Docu: Nginx
Docu: Manual SSL for Nginx

Forum: File permissions and setting cert path
Forum: Cert chmod check and [‘www-data’] for my system
Forum: Recommended changing Puma Port

Some errors I get in the log (it’s mostly the blue spam-text from scraping so it’s difficult to see actual errors and warnings):

# recurring about every minute or so
==> /var/log/gitlab/gitlab-workhorse/current <==
redis: 2024/07/15 11:08:15 pubsub.go:168: redis: discarding bad PubSub connection: EOF
{"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"","time":"2024-07-15T11:08:15Z"}

# when doing gitlab-ctl reconfigure
==> /var/log/gitlab/postgresql/current <==
2024-07-15_11:09:25.57112 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2024-07-15_11:09:25.57116 FATAL:  Peer authentication failed for user "gitlab"
2024-07-15_11:09:25.57118 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
2024-07-15_11:09:25.63809 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2024-07-15_11:09:25.63812 FATAL:  Peer authentication failed for user "gitlab"
2024-07-15_11:09:25.63813 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"
2024-07-15_11:09:25.70327 LOG:  no match in usermap "gitlab" for user "gitlab" authenticated as "root"
2024-07-15_11:09:25.70330 FATAL:  Peer authentication failed for user "gitlab"
2024-07-15_11:09:25.70331 DETAIL:  Connection matched pg_hba.conf line 70: "local   all         all                               peer map=gitlab"

If you have specific commands you’d like me to run for more specific logs, let me know.
Status seems fine, it’s not in a restart-loop

# gitlab-ctl status
run: gitaly: (pid 21026) 4027s; run: log: (pid 4091) 43722s
run: gitlab-exporter: (pid 322) 44774s; run: log: (pid 313) 44774s
run: gitlab-workhorse: (pid 21470) 3941s; run: log: (pid 314) 44774s
run: logrotate: (pid 22709) 1574s; run: log: (pid 308) 44774s
run: nginx: (pid 22182) 3840s; run: log: (pid 22181) 3840s
run: postgresql: (pid 319) 44774s; run: log: (pid 312) 44774s
run: puma: (pid 21464) 3942s; run: log: (pid 21463) 3942s
run: redis: (pid 316) 44774s; run: log: (pid 309) 44774s
run: sidekiq: (pid 20957) 4045s; run: log: (pid 310) 44774s
run: sshd: (pid 36) 44784s; run: log: (pid 35) 44784s

Generally, the external_url, should be how you want to access it, whether that is over http and port 8380 or https on port 7343 is up to you. You mention you have an nginx container/server in front of Gitlab? Did you ensure to configure the proxy headers, since nginx is working as a reverse proxy. For example:

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Forwarded-Host $host;
      proxy_set_header X-Forwarded-Server $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_pass http://unraid:7380;
   }

you may also need additional settings in gitlab.rb as per these Gitlab nginx docs: NGINX settings | GitLab but I would see how you go first when nginx is providing the headers as above from nginx to Gitlab. My example above is set to redirect nginx over http to port 7380, so change accordingly on whether your external url uses 7380 or https on 7343.

If you use external_url without the port, it will want to use 80 or 443 by default.

Hey @iwalker , thanks a lot for the quick reply. I really appreciate it.
Unfortunately, it didn’t seem to do anything yet.

I’ve set the headers via nginx’ advanced config for the host which now sets the proxy_pass to http://192.168.178.20:7380, as all Hosts, internally, communicate on HTTP beyond nginx.

I also made sure I’ve set external_url 'http://git.mydomain.org:7380' .
It still gives the error, so I attemted to set the headers from the gitlab.rb file as well as per the link you kindly provided.

The entirety (minus SMTP and log adjustments) of my gitlab.rb is below so the actual state is clear, as I may have forgotten parts

external_url 'http://git.mydomain.org:7380'
web_server['external_users'] = ['www-data']
nginx['ssl_certificate'] = "/etc/gitlab/ssl/mydomain.org.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mydomain.org.key"
nginx['referrer_policy'] = 'same-origin'
nginx['listen_port'] = 7380
nginx['listen_https'] = false
puma['port'] = 7380

nginx['proxy_set_headers'] = {
  "Host" => "$http_host_with_default",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "http", #https
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
}

#services
nginx['enable'] = false
letsencrypt['enable'] = false
gitlab_kas['enable'] = false
alertmanager['enable'] = false
node_exporter['enable'] = false
redis_exporter['enable'] = false
postgres_exporter['enable'] = false

#rails
gitlab_rails['smtp_enable'] = true
gitlab_rails['incoming_email_enabled'] = false
gitlab_rails['terraform_state_enabled'] = false

Note that the bundled nginx is set to disabled but some values are still configured. However, I tried enabling the bundled nginx and either way it still gives the 502.

Could you kindly take a look at my config and confirm if…

1. bundled nginx should be disabled when you use an external instance
2. any ‘nginx’ related changes in gitlab.rb are unused when bundled nginx is disabled
3. puma['port'] = 7380 (http) should be set when bundled nginx is disabled

in this documentation it also mentions having to ‘download the correct web server configuration’. Is this meant to be installed somewhere on my non-bundled nginx or inside gitlab someplace?

1. Bundled nginx doesn’t have to be disabled. If you do however, you will need to look in the Gitlab documentation to make sure everything that the bundled Nginx did is now configured correctly on your external nginx. Thus, if you leave the bundled nginx running, all you need to worry about is forwarding from your external nginx to the internal/bundled one on Gitlab.
2. If you do disable nginx in gitlab.rb, then none of the configuration is used.
3. No, puma runs on port 8080 by default, therefore if you disable bundled nginx, ports 7380 and 7343 no longer exist. In which case as per the Gitlab nginx documentation, and the expanded nginx.config as already mentioned in point 1 will need to be configured on your external nginx. A lot more work.

If it was me, I would leave nginx working on Gitlab using ports 7380 and 7343 (https) and use the external nginx to just redirect to those ports on the Gitlab nginx. Otherwise it gets more complicated.

Once you get the basic nginx redirecting from one to the other, only at this point would I think of disabling the bundled nginx if I felt it would give me benefits of some sort.

First, let’s forget about HTTPS and port 7343 for now. Let’s just configure it to work on port 7380 with normal http. Then later think about SSL if you want https → https between your external NGINX and Gitlab nginx. Try the following config options in gitlab.rb:

external_url 'http://gitlab.example.com:7380'

nginx['listen_port'] = 7380
nginx['proxy_protocol'] = true
nginx['proxy_set_headers'] = {
 "Host" => "$http_host_with_default",
 "X-Real-IP" => "$remote_addr",
 "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
 "X-Forwarded-Proto" => "http",
# "X-Forwarded-Ssl" => "on",
# "Upgrade" => "$http_upgrade",
# "Connection" => "$connection_upgrade"
}

all those options exist in your gitlab.rb just search for them, rather than just copy/paste what I put above. You will notice 3 options are hashed, because they are related to SSL which we are not using right now. Once you get it working on http, then you can try https by changing your external_url and changing http to https, and changing the port from 7380 to 7343. Then, unhash those three options, and you will also need to add:

nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"
nginx['listen_https'] = 7343

Gitlab nginx will still listen on 7380 and 7343. In theory at this point, any connections to http on 7380 should automatically redirect to https on port 7343 although we don’t need that, since your external NGINX is working on port 80/443 anyway so it’s irrelevant.

Again, only think about enabling those SSL options if you need to, and only after it works first on normal http.

I personally haven’t tried the above, but it should work. I would have to make a server and installation to physically test it, but I’m pretty sure it should work based on the Gitlab documentation.

Hey again. Thanks a ton for your answer - I can see you put a lot of effort into it and it cleared up many things I was unsure about.

I’m fine having only HTTP on GitLab (internally) if that works at least, as my structure here is as such:
Cloudflare <==HTTPS==> Nginx <==HTTP==> Host

As per the suggestion I have left bundled nginx enabled, GitLab ports remain the same: 7380 (http) + 7343 (https)

With the nginx sections and external_url commented out in gitlab.rb the external nginx can connect just fine to GitLab, but the only (major!) issue I have is that clicking Tasks goes to an invalid IP as per my original post.

Now, with external_url set and the remaining setup as prescribed, it still produces a 502 Bad Gateway error. Note, this is both with and without the previously mentioned location/{} block added to the external nginx.

# nginx

git.mydomain.org => http://192.168.178.20:7380
# below same config as all other hosts
'mydomain.org' Origin Certificate
> Force SSL
> HTTP/2 Support
> HSTS

# gitlab.rb

external_url 'http://git.mydomain.org:7380'

nginx['listen_port'] = 7380
nginx['proxy_protocol'] = true
letsencrypt['enable'] = false #nil
web_server['external_users'] = ['www-data']

nginx['proxy_set_headers'] = {
  "Host" => "$http_host_with_default",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "http",
#  "X-Forwarded-Ssl" => "on",
#  "Upgrade" => "$http_upgrade",
#  "Connection" => "$connection_upgrade"
}

Other things I’ve tried:

• disabling HSTS / ForceSSL etc on the ext. nginx config (shouldn’t be necessary but I tried anyway)
• This topic although here, too, no luck

Do you happen to have any more pointers or where I could look for more info that may be emitted by gitlab?

By the way, something I noticed that may also be relevant:

I am unable to access GitLab directly via IP (192.168.178.20:7380) nor git.mydomain.org are accessible if I have either

• external_url set
• nginx['listen_port'] = 7380 set
• nginx['proxy_protocol'] = true set

Which is odd, especially for the port as it’s the same port I connect to if I were to disable it (at least that is my assumption)

Edit: For shits’n’giggles I set the nginx listen port to 9080 (the docker internal listen port)
With that but without external_url set, I’m able to connect both via IP and domain.
So that’s cool and a good direction.

In hindsight that makes sense, as, why would GitLab, inside a docker, listen to the external port outside the docker?
(nginx ([gitlab 9080]<>7380 docker))

I proceeded to set external_url 'http://git.mydomain.org' (without port) and for the first time, I’m actually getting log output (albeit only errors) when trying to connect via IP or domain:

#via domain - 172.18.0.1 is the nginx docker
2024/07/15 20:21:03 [error] 2777#0: *10 broken header: "GET / HTTP/1.1" while reading PROXY protocol, client: 172.18.0.1, server: 0.0.0.0:9080
2024/07/15 20:21:03 [error] 2778#0: *11 broken header: "GET /favicon.ico HTTP/1.1" while reading PROXY protocol, client: 172.18.0.1, server: 0.0.0.0:9080

#via IP - 192.168.178.120 is my current client ip
2024/07/15 20:23:47 [error] 2781#0: *17 broken header: "GET / HTTP/1.1" while reading PROXY protocol, client: 192.168.178.120, server: 0.0.0.0:9080
2024/07/15 20:23:49 [error] 2769#0: *18 broken header: "GET / HTTP/1.1" while reading PROXY protocol, client: 192.168.178.120, server: 0.0.0.0:9080

Omg I think I got it working…

fkinfinally718×358 34.4 KB

I haven’t tested it extensively yet, but the part that didn’t work before appears fixed.
For others that may have the same issue in a similar setup (Unraid , Nginx docker , GitLab docker) and want to use their Domain for GitLab …

image704×249 26 KB

external_url 'https://git.mydomain.org' #no port. use https
letsencrypt['enable'] = false

nginx['listen_addresses'] = ['*', '[::]']
nginx['listen_port'] = 9080 #internal docker http port
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
  "Host" => "$http_host_with_default",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
}

As for nginx advanced config (location /{} block) - I didn’t add any configuration there.
I hope this is of help to someone

What were the port parameters for running Gitlab on Docker(Unraid)? It would seem your published ports are -p 9080:9080, if that is the case then that would explain a lot of things. Had the ports been published as -p 9080:7380 then it would have forwarded correctly to the Gitlab config or even -p 7380:7380. I’m guessing, but that would suggest what is going on here.

For clarity, the reason why you had URL’s with http://unraid were due to the lack of passing proxy headers. So once a lot of that had been sent, that would have been what fixed that issue.

I think even if I had of done my own lab with this, I wouldn’t have used Docker myself, and thus would probably have not replicated your problem anyway. I missed that bit before, which is why it never came to my mind to think of the ports being used for exposing those ports.

For those reading, also check my previous post for a fix in the same situation.

The port was published as 9080 as the internal port and 7380 as the external one. The docker was not set up as 9080:9080.
What threw me off was that I was setting an nginx port inside the docker but that should obviously point at the internal one, not external.

Either way, it seems that was the primary issue I had with enabling the external_url.
I went ahead and marked your last message as the solution as you helped out a lot in clearing up other parts that I was unsure of. Thanks a ton

Ah so you are saying external docker port was 7380 and internal was 9080? Then -p 7380:9080 in which case yes, Gitlab.rb configured with 9080 and external nginx with 7380. The other alternative, to keep it simple is make both ports identical, so either 7380:7380 or 9080:9080 where possible. Obviously if Unraid has lots of services running, and 9080 is already in use, then not much choice.

Yeah, that can get confusing but glad it’s all working