GitLab Webinterface not reachable after update to 15.1.0

Hi! I am making this post, because I can’t seem to find the issue with my GitLab installation.

It worked fine before the update on 15.1.0.

I updated from 15.0.1 to 15.0.2 and then from .2 to .3 and from there to 15.1.0. I even made a backup, used gitlab-ctl cleanse, purged the package and restored my backup. It still shows the same issues. Git itself via commandline works fine though.

My GitLab is running as a container inside of Proxmox. There is another container that runs OpenResty as a Loadbalancer for SSL offloading. Starting a Python webserver on the GitLab container and curling it works fine, also it worked fine before the updates even with GitLab 15.
So I do very much think that it is an issue with GitLab.

The current state is as follows:

# gitlab.rb
external_url 'https://anonymized'
gitlab_rails['lfs_enabled'] = true
sidekiq['concurrency'] = 5
nginx['listen_addresses'] = ["192.168.128.103"]
nginx['redirect_http_to_https'] = false
nginx['listen_https'] = false
nginx['listen_port'] = 80
nginx['real_ip_header'] = 'X-Forwarded-For'
nginx['real_ip_recursive'] = 'on'
letsencrypt['enable'] = false

root@gitlab:~# sudo gitlab-rake gitlab:env:info

System information
System:		Ubuntu 20.04
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.7
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	15.1.0
Revision:	6bea4379525
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	13.6
URL:		https://git.cle.ar.it
HTTP Clone URL:	https://git.cle.ar.it/some-group/some-project.git
SSH Clone URL:	git@git.cle.ar.it:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.7.4
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

root@gitlab:~# gitlab-ctl status
run: alertmanager: (pid 10214) 623s; run: log: (pid 318) 2712s
run: gitaly: (pid 10229) 622s; run: log: (pid 369) 2712s
run: gitlab-exporter: (pid 11833) 447s; run: log: (pid 352) 2712s
run: gitlab-kas: (pid 11802) 448s; run: log: (pid 322) 2712s
run: gitlab-workhorse: (pid 11814) 448s; run: log: (pid 363) 2712s
run: grafana: (pid 10270) 620s; run: log: (pid 350) 2712s
run: logrotate: (pid 10280) 620s; run: log: (pid 328) 2712s
run: nginx: (pid 10300) 620s; run: log: (pid 367) 2712s
run: node-exporter: (pid 10304) 620s; run: log: (pid 354) 2712s
run: postgres-exporter: (pid 10314) 619s; run: log: (pid 324) 2712s
run: postgresql: (pid 10402) 619s; run: log: (pid 315) 2712s
run: prometheus: (pid 10411) 618s; run: log: (pid 365) 2712s
run: puma: (pid 11852) 440s; run: log: (pid 356) 2712s
run: redis: (pid 10427) 617s; run: log: (pid 330) 2712s
run: redis-exporter: (pid 10435) 617s; run: log: (pid 313) 2712s
run: sidekiq: (pid 11795) 449s; run: log: (pid 346) 2712s

So as far as I have figured out, the request is handled by GitLabs nginx in /var/opt/gitlab/nginx/conf/gitlab-http.conf here:

  location / {
    proxy_cache off;
    proxy_pass  http://gitlab-workhorse;
  }

Which is an upstream to a socket in /var/opt/gitlab/nginx/conf/nginx.conf:

  upstream gitlab-workhorse {
    server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket;
  }

A local curl -I 192.168.128.103:80 never gets answered and never times out. If I go via OpenResty the upstream connect will run into a timeout and show a 504.

I can not see any severe issues in gitlab-ctl tail and only found this in gitlab-workhorse logs but I don’t think this is the issue:

root@gitlab:~# grep -i ERROR /var/log/gitlab/gitlab-workhorse/current 
[...]
{"error":"keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory","level":"error","msg":"","time":"2022-06-24T17:15:26+02:00"}
[...]
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-06-24T17:15:42+02:00","uri":""}

Can anyone please help me debug this further?

Hi
i have same problem but aftre uncomment this line “nginx[‘worker_processes’] = 4” in gitlab config file fixed this issue.

Thank you, this indeed was the fix for me as well. I am slightly confused how this can be a problem. Do you think opening an issue over at Issues · GitLab.org / GitLab · GitLab?

Must be something unique to your installation. I have two installations, one was 15.0.2 and the other was 15.0.3 and both upgraded perfectly fine. Also, my gitlab.rb:

cat gitlab.rb | grep -i worker_processes | grep nginx
# nginx['worker_processes'] = 4

I didn’t need to uncomment it, but all works fine. I run a normal VM rather than docker install, so perhaps it’s something unique to docker?

I think this issue for lxc container at proxmox only .

Same problem on Proxmox. After uncomment nginx[‘worker_processes’] = 4 it works.

I’m having similar issue. Worked until 14.xx. After a upgrade to 15.1, the web interface is unreachable

RHEL 7
gitlab-ee (No enterprise support)

I also hit the same issue after upgrading to 15.1.0.
Because I am on a single core vm I set nginx['worker_processes'] = "auto" which lets nginx autodect how many worker it should start. But it still fixes the issue as already mentioned.

It seems something is not right with the default nginx config from the gitlab 15.1.0 release.

@mstinsky then you are running with incorrect specs: Installation system requirements | GitLab

This documentation states:

therefore, if you run with specifications smaller than what is set, then it’s obvious it’s not going to work. The Gitlab 15.1.0 release configuration is perfectly fine, since when I have a VM that has 4cpu and the appropriate amount of ram, all works fine.

My gitlab install was working fine since years and it keeps on working fine with nginx['worker_processes'] = "auto" set.

I did not look into the nginx configuration directly but after the upgrade from 15.0.x to 15.1.x something 100% changed otherwise it does not make sense that setting worker_processes to auto manually fixes this issue.

In the end sure the new default configuration might not be compatible with my setup anymore but its working again which is fine for me

For a single cpu, you may wish to look at the Raspberry PI instructions, you can disable other items as well to make it use less cpu/ram. Unless of course you already have done this. I have done it for disabling things that I’m not using, and saves using unnecessary resources.

Thanks for flagging. Reported in nginx: 504 Gateway Time-out after upgrade to 15.1: worker_processes is set to zero (#6895) · Issues · GitLab.org / omnibus-gitlab · GitLab

Just to confirm, setting the worker_processes to auto works for me also. I have LXC running with 4Gigs of ram and 4 cores. Saw no issues upgrading to 15.3.2.