Hi,
just made an upgrade to latest version and Gitlab portal is not responding.
Have not touched any config files for years, because all was set and running as it should

All services are running, except NGINX (even when i do service restart, nginx looks like is starting):

➜ ~ sudo gitlab-ctl status
run: alertmanager: (pid 2063) 1038s; run: log: (pid 1873) 1084s
run: crond: (pid 2031) 1046s; run: log: (pid 1859) 1084s
run: gitaly: (pid 2057) 1039s; run: log: (pid 1870) 1084s
run: gitlab-exporter: (pid 1941) 1060s; run: log: (pid 1852) 1085s
run: gitlab-kas: (pid 1909) 1081s; run: log: (pid 1832) 1085s
run: gitlab-workhorse: (pid 2016) 1052s; run: log: (pid 1854) 1085s
run: grafana: (pid 1930) 1068s; run: log: (pid 1833) 1085s
run: logrotate: (pid 1969) 1058s; run: log: (pid 1853) 1085s
run: mattermost: (pid 2030) 1046s; run: log: (pid 1858) 1084s
down: nginx: 0s, normally up, want up; run: log: (pid 1855) 1085s
run: node-exporter: (pid 2011) 1053s; run: log: (pid 1856) 1085s
run: postgres-exporter: (pid 2037) 1044s; run: log: (pid 1869) 1084s
run: postgresql: (pid 2244) 1020s; run: log: (pid 1890) 1082s
run: prometheus: (pid 2108) 1032s; run: log: (pid 1874) 1084s
run: puma: (pid 1923) 1071s; run: log: (pid 1876) 1084s
run: redis: (pid 2221) 1026s; run: log: (pid 1889) 1083s
run: redis-exporter: (pid 2056) 1039s; run: log: (pid 1871) 1084s
run: registry: (pid 1922) 1072s; run: log: (pid 1831) 1085s
run: sidekiq: (pid 2058) 1039s; run: log: (pid 1872) 1084s

But from Tail i see some problems:

==> /var/log/gitlab/nginx/current <==
2022-03-25_09:51:14.17645 nginx: [emerg] duplicate location “/.well-known/acme-challenge/” in /var/opt/gitlab/nginx/conf/gitlab-http.conf:173
^[

or

==> /var/log/gitlab/gitaly/current <==
{“address”:“localhost:9236”,“level”:“info”,“msg”:“starting prometheus listener”,“time”:“2022-03-25T09:32:47.352Z”}
{“level”:“warning”,“msg”:"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/ruby.0 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = “transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.0: connect: connection refused”. Reconnecting…",“pid”:2080,“system”:“system”,“time”:“2022-03-25T09:32:48.162Z”}
{“level”:“warning”,“msg”:"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = “transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory”. Reconnecting…",“pid”:2080,“system”:“system”,“time”:“2022-03-25T09:32:48.162Z”}
{“level”:“info”,“msg”:“maintenance: daily scheduled”,“pid”:2080,“scheduled”:“2022-03-25T12:00:00+01:00”,“time”:“2022-03-25T09:32:48.497Z”}
{“level”:“info”,“msg”:“PID 2666 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile”,“supervisor.args”:[“bundle”,“exec”,“bin/ruby-cd”,"/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby",“2080”,"/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],“supervisor.name”:“gitaly-ruby.0”,“time”:“2022-03-25T09:32:59.509Z”}
{“level”:“info”,“msg”:“PID 2665 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile”,“supervisor.args”:[“bundle”,“exec”,“bin/ruby-cd”,"/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby",“2080”,"/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],“supervisor.name”:“gitaly-ruby.1”,“time”:“2022-03-25T09:32:59.510Z”}
{“level”:“warning”,“msg”:"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = “transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory”. Reconnecting…",“pid”:2080,“system”:“system”,“time”:“2022-03-25T09:33:03.164Z”}

I tried to manuallyt remove all the files from gitlab ssl folder and try to run Reconfigure to generate new certs. But that is failing.
Manually put the still valid cert/key back to SSL folder and Reconfigure is running smoothly. But still:

Hmmm… can’t reach this page

gitlab.domain.com refused to connect.

Try:

• Checking the connection
• Checking the proxy and the firewall

ERR_CONNECTION_REFUSED

I have the same error and find a workwaround:

docker exec -it gitlab bash

vi /var/opt/gitlab/nginx/conf/gitlab-http.conf

Go to line 173 and delete the 3 lines above as they are the same as line 150

location /.well-known/acme-challenge/ {
root /var/opt/gitlab/nginx/www/;
}

Save the file and wait a moment for gitlab to start ok.

Can you please share your /etc/gitlab/gitlab.rb file (after redacting secrets from it) ?

Similar discussion: Gitlab not accesible after Upgrade to 14.9.1 (#6750) · Issues · GitLab.org / omnibus-gitlab · GitLab

I reverted VHDX back to the state before update.
Let me colelct .rb and ngnix config from both states

.rb fiels are the same
What has changed was realy only nginx files: gitlab-http.conf, gitlab-mattermost-http.conf and gitlab-registry.conf

Gitlab-http.conf

non-functional copy
vs
functional copy

40: location /.well-known/acme-challenge/ {
vs
40: location /.well-known {

150: location /.well-known/acme-challenge/ {
151: root /var/opt/gitlab/nginx/www/;
152: }
vs
150:
151:
152:

gitlab-mattermost-http.conf has only these line in non-functional (functional has no lines like these)

103: location /.well-known/acme-challenge/ {
104: root /var/opt/gitlab/nginx/www/;
105: }

gitlab-registry.conf has only these line in non-functional (functional has no lines like these)

44: location /.well-known/acme-challenge/ {
45: root /var/opt/gitlab/nginx/www/;
46: }

This was probably the reason. But why it added/changed these line to one instance and not the the second one?

I encountered the same issue after upgrading from 14.8.2 to 14.9.2. My gitlab-http.conf had muliple .well-known/acme-challenge routes.

I found that my gitlab.rb config had letsencrypt['enable'] = true set. After commenting this out and restarting gitlab, gitlab started up and was available again. The gitlab-http.conf nginx file still had a acme-challenge route. It is poorly formatted, which makes me some what suspicious. Note that the odd indentation here is intentionally pasted

  location /.well-known/acme-challenge/ {
 root /var/opt/gitlab/nginx/www/; 
}

} ## end HTTPS server

my SSL was going to end so i started to update/upgrade my gitlab to 14.10, first i run reconfigure to have fresh lets encrypt SSL, this wen fine
Upon upgrading to 14.10 i run into DB migration problem, resolved by running rake db migration and special lengthy command found in error output. This went fine after all.
Then already mentioned gitlab-http.conf problem

i commented 3 last lines (location and root) and started gitlab

This was few days ago

Today gitlab was not running again. First thing i opened /var/opt/gitlab/nginx/conf/gitlab-http.conf and saw my commented lines got uncommented.

Why? Who/What made it?

Changes shouldnt be made in individual config files. It should be done in /etc/gittlab/gitlab.rb and then using gitlab-ctl reconfigure to apply it to the various config files.

what exactly, to have those duplicated lines be off

Hi guys,

Thanks for the thread.
I had the same issue, and I found why the reconfigure add both ACME config in the nginx.conf.

If you set

letsencrypt['enable'] = false

it will remove the duplicata BUT your gitlab won’t work anymore with letsencrypt, so it’s not the solution for me.

So I looked for another config in the gitlab.rb file, and found this line :

nginx['custom_gitlab_server_config'] = "location /.well-known/acme-challenge/ {\n root /var/opt/gitlab/nginx/www/; \n}\n"

Do not ask me why it’s here … I’m pretty sure I never added it, but I just commented it … and TA DA !!

Gitlab works well , and I can reconfigure my gitlab instance without changing manually the nginx conf file

I wonder if my config will be changed after next update
not making any changes to gitlab.rb now

I too had this issue.
I remember adding that line quite a while ago when the letsencypt integration was new.
I followed this tutorial which is now deprecated, which describes adding the custom_gitlab_server_config line.

Possibly others did too hence we have this issue.
I confirm, removing the line stops the duplicate config being added.

made the same, made upgrade to 15.0.2 and all went just fine
thank you

My nginx was failing to start after each update from 14 to 14.10 to 15.0 to 15.5 that I did.

I recall that the custom_gitlab_server_config was required to enable Let’s Encrypt at one point and was listed in various tutorials.

https://www.digitalocean.com/community/tutorials/how-to-secure-gitlab-with-let-s-encrypt-on-ubuntu-16-04
https://github.com/VojtechMyslivec/letsencrypt-gitlab

My nginx was failing after updates and I had to discover why using gitlab-ctl tail nginx and seeing duplicate location "/.well-known/acme-challenge/" in /var/opt/gitlab/nginx/conf/gitlab-http.conf which I was removing by hand before finding this discussion/solution. Thanks!