As part of our Gitlab upgrades we were upgrading from 16.1.2 to 16.5.2 using the Gitlab omnibus tool.
We followed this path 16.1.2 -> 16.1.5 -> 16.3.6 -> 16.5.2 for the upgrade
But in versions 16.3.6 & 16.5.2 we were faced with a 503 Error in Gitlab after the upgrade finished. On running gitlab-ctl status we found out some services to be down. Once restarted, everything was working fine.
Isn’t the restart already included in upgrade? Any reason why we had to do a gitlab-ctl restart explicitly?
You shouldn’t need to. I never had these problems and I upgrade immediately when a new Gitlab version is released. So I’ve never seen that with 16.3.6 or 16.5.2. But then I prob also had 16.4 releases running also between these two versions.
The only question or thing I can think of is, before the upgrade did you stop Gitlab? If so this would cause problems. Gitlab doesn’t need to be stopped before upgrade, since it will stop the running services itself when it starts the upgrade. It does need at least postgresql running and some others for migrations, etc to work during the upgrade.
Otherwise, that scenario you experienced shouldn’t happen.
I’ve seen various errors like this. Simply put, sometimes things go wrong. I’ve had weird service errors or migrations not run, etc.
It mainly is why they put "if something isn’t working, run gitlab-ctl restart before doing anything else.
I’d usually recommend checking service status with gitlab-ctl status after an upgrade and then checking DB migrations with gitlab-rake db:migrate:status
In my experience those are usually the go-tos to fix thing. If a service refuses to start, then just do gitlab-ctl tail <service-name> to get logs as to why it’s not running right.