GitLab CE: Upgrade omnibus from 15.0.2 to 15.1.2 - Nginx not responding

I’ve got a GitLab Omnibus install running on Ubuntu 20.04, which was running version 15.0.2.

I followed the Upgrading GitLab instructons to upgrade (namely: take a full backup, check for background migrations (both commands returned 0), run sudo apt-get update && sudo apt-get install gitlab-ee) to update to version 15.1.2 (the current version at the time of writing).

All of the commands seemed to run fine, with no errors or warnings reported. The upgrade script shows Upgrade complete! and all the services are showing as run:. However, when I try to access the web pages, my web browser (and I’ve tried several - Chrome, Firefox, Edge) just show the loading progress spinner indefinitely without any kind of timeout (I’ve let it run all morning).

This is my second attempt at this upgrade - my previous attempt was on Monday and I ended up rolling back a snapshot of the VM that runs the service which I’d taken before attempting the upgrade.

I’m at a loss as to what on earth’s going wrong, as checking the GitLab specific status and checking the Nginx logs don’t show anything wrong.

If I use curl to talk directly to the rails socket (sudo curl --unix-socket /var/opt/gitlab/gitlab-workhorse/sockets/socket, then I see a correct response being printed out, so it definitely seems like something is funky with Nginx.

I’ve also fully disabled the UFW firewall just in case it was causing problems, and have tried restarting things with sudo gitlab-ctl restart.

1 Like

Also worth noting:

  • /var/log/gitlab/nginx/gitlab_error.log is empty
  • /var/log/gitlab/nginx/gitlab_access.log only shows requests from before the attempted upgrade

I’ve just changed to using a non-bundled version of Nginx and now everything works fine. I’m using v1.22.0 from the official packages

Certainly seems to me like the version of nginx shipped with the 15.1.2-ee package for ubuntu might be slightly broken :slight_smile: The version that was bundled was v1.20.1, which is a legacy version now and there have been several security and bug fixes since then.

At least my install is now back up and running for now though.

It might just have needed:

gitlab-ctl reconfigure
gitlab-ctl restart

failing that, sometimes you need to do:

systemctl restart gitlab-runsvdir

as that can also sometimes cause issues. Failing that a reboot. I use same version, nginx works fine that is bundled with it and I have 15.1.2.

Yeah, unfortunately I tried both reconfigure and restart (including a full systemctl poweroff followed by a clean boot). I never knew about systemctl restart gitlab-runsvdir though, which may have helped.

I am running the server as an LXC Container, which shouldn’t cause any problems but may be worth noting just in case.

Ah yeah, then the problem is you need to uncomment the worker_processes:

# nginx['worker_processes'] = 4

remove the comment hash, run reconfigure, and nginx will be fine. There are some posts on here about it already relating with LXC under proxmox: GitLab Webinterface not reachable after update to 15.1.0 - #2 by samansohani

So it seems when using LXC you need to uncomment this.

As for gitlab-runsvdir, it would have just saved you the reboot, after you rebooted, this service was restarted anyway.

1 Like

Aaaah, that might just be the missing part of the puzzle!

I’ll give that a go and see if it works - if so, I’ll go back to using the bundled version as it does make life much easier.

Thanks @euantorano and @iwalker for the solution.
After days of banging my head with this issue your post gave me the solution.
I’m using a container under proxmox as well.

Again, thanks!