ERROR: relation "services" does not exist after upgrade to 14.2.0

i updated from 13.12 to 14.0.7 and i thought the migrations was completed and everything was fine and then stop the container and re run it with 14.2 but it does not coming up and i reverted to 14.0.7 again and this time gave me the 500 error with message.

ActionView::Template::Error (PG::UndefinedTable: ERROR: relation "services" does not exist

and i didn’t backup unfortunately.

There should be a backup under /var/opt/gitlab/backups which was taken during upgrade process.

thanks for the reply,
but backup wasn’t taken
and i open the issue

that solved my problem.

1 Like

The whole concept of ‘background’ migration is very risky when it is not made clear to the (admin) user. Judging by these forums I’m clearly not the only one who jumped the 14.0.x → 14.2 upgrade too early and ended up with a broken instance and potentially lost data.

Presumably ‘background’ migration is intended as an optimisation to get the instance up as soon as possible. I put it to you that most users will prefer safety of migration than absolute minimisation of downtime. If power users need background migration, can it be a startup option, and the default be to complete migration during startup?

At least a banner can inform users of the incomplete migration; otherwise this is dangerous design.

Hi @chris-hatton

There’s a difference between upgrading and not waiting for the background migrations to finish, than jumping a version - in your instance 14.0.x to 14.2.x without installing the latest 14.1.x release first.

My install for example (I’ve written about it here), I used to upgrade regularly. I got to 12.9.3 and then it didn’t upgrade anymore. I was doing the usual apt-get update && apt-get upgrade which would normally refresh the repo data, and then it would see the update and download. I then decided to refresh the repo by running the command from the install docs, which sorted this out. I then went to do the update and upgrade commands again, and saw that 13.x was available. I was about to do this, but then decided perhaps I should look at the Gitlab documentation before I actually do it. Lucky that I did, since otherwise I wouldn’t have followed the upgrade path and would have broken my installation.

To update from 12.9.3 to the latest 13.x release of that time meant me going through a whole load of updates and manual apt-get install commands for specific versions, and between each version waiting for background migrations to complete before going to the next one.

In your instance, the banner wouldn’t help, since you jumped from 14.0.x to 14.2.x and missed an important update in between.

I think the best thing, and this isn’t unique to Gitlab, but I highly recommend reading any documentation before making updates of a system purely because we don’t know if it’s going to work or not. Especially taking a backup/snapshot beforehand to revert back to in case of problems being experienced.

Little off-topic, but OpenStack I attempted to upgrade once and it broke badly. Purely because there are a ton of config files to work through later to get all the related services back online again. Gitlab as an upgrade is perfectly fine provided that the documentation is reviewed.

I now check regularly for updates, and have been pretty much applying every single update that has been released. So I was on 14.0.6 before going to 14.1.0, 14.1.1, 14.1.2, 14.1.3, 14.2.0, 14.2.1 - but since I was checking regularly, it just meant a standard apt-get update && apt-get upgrade. Had I not done it for weeks/months, I would have checked the documentation first to ensure what the upgrade path was before doing something that would break my system.

Ideally, I think a pre-install script would be able to ensure that the existing install is on at least 14.1.x before upgrading to 14.2.x. That way it could save a few broken installs at least as it would stop it jumping a certain required minimum Gitlab version.