GitLab Omnibus Container 'stuck' on version 12.8.2-ee.0

Hello GitLab Forum,

we are currently using a GitLab Omnibus container for our internal development.
Sadly said Omnibus hasn’t seen a maintenance in a long time, so it hasn’t been upgraded in a long time.
Recently my team was assigned the responsibility of said GitLab Omnibus instance.
One of the first things we wanted to do is upgrade the Omnibus container but the problem is that the upgrades from version 12 to 13 are failing every time.
The container is currently running on the 12.8.2-ee.0 Docker Image.
We tried upgrading to 12.10.14-ee.0 and afterwards tried upgrading from there as recommended in the GitLab Docs upgrade path for the enterprise edition.
The problem after upgrading from 12.10.14-ee.0 or 12.10.13-ee.0 to 13.0.14-ee.0 is always the same:
The users are unable to pull or push code and the Web Editor vanishes from the UI.

Component Versions:
GitLab 12.8.2-ee (72ed810d281)

GitLab Shell 11.0.0

GitLab Workhorse v8.21.0

GitLab API v4

Ruby 2.6.5p114

Rails 6.0.2

PostgreSQL 11.7

Does anyone have an idea what we are doing wrong? Or what I could try in order to upgrade the Container to a newer version? Sadly my experience with GitLab administration is almost nonexistent.

Are you running GitLab in a docker container or not? In one place you write

I have no experience running GitLab in docker, so if you are, I can’t help.

If you’re running an omnibus installation or some OS, please tell us which OS, and provide us with the complete interaction users have with GitLab when they are unable to pull/push code.

I’m running an Ominbus GitLab as a Docker Container (as far as I know Omnibus GitLab is always a (docker) Container)?
I don’t think that my issues are related to how the GItLab is running but more related to Background migrations not working or components running on the wrong version?

The setup is as follows: CentOS7 vm hosts → Docker Container contains → GitLab Omnibus Instance.

When running the psql query from the docs to check for background Migrations it always returns ‘0’ if that helps.

I will get back to you soon for a detailed error description regarding pushing and pulling, since it will take some time to reproduce the error, as the testing environment is currently on version 12

Update:
I was able to reproduce the error when pushing:

$ git push
info: detecting host provider for 'http://InternalGitLabUrl.com/'...
fatal: unable to access 'http://InternalGitLabUrl.com/group/subgroup/project/subproject.git/': The requested URL returned error: 500

Cloning seems to work, so I assume that pulling should work too.

What I recognized during upgrading is that postgre SQL is not updated by the version change of the docker image. Could the version that maybe is to ‘new’ of the postgre SQL cause the migrations to fail/not start at all?

We use omnibus GitLab and run it on some servers (nowadays virtual, when I took over it was still on a physical), no containers and no docker in sight. While I can imagine how to make a hybrid approach (but my idea would just be silly), I think it’s pretty much an either/or-situation; either you use omnibus GitLab or you use docker images. On

you either select “Official Linux package”, that gives an omnibus installation, or you select “Docker” from “Other official, supported installation methods” (below “Kubernetes Deployments” and “Supported cloud”).

I’m not saying that the problem is related to how you run GitLab, but it does affect who can help, I have no idea how the docker images work, or how to upgrade such an installation.

If your installation is based on docker images, I would suggest a new thread that doesn’t have “Omnibus” in the title, possibly mentioning it’s an installation based on docker images. With “Omnibus” in the title (like this thread), you attract some people (like me) who believe we know something about omnibus installations, and you make people who knows something about docker installations skip the thread, because:

  1. they won’t be able to help anyway
  2. they think you will get better help from people that know about omnibus installations