Backup succeeded with errors

I’m running Gitlab on OpenSUSE and using the gitlab-backup script with “create” as the sole parameter. It runs and returns a success code, but when I examine the log file, I see this:
2023-12-17 10:00:25 UTC – Dumping database …
Dumping PostgreSQL database gitlabhq_production … 2023-12-17 10:00:26 UTC – Dumping database failed: Failed to create compressed file ‘/var/opt/gitlab/backups/db/database.sql.gz’ when trying to backup the main database:

  • host: ‘/var/opt/gitlab/postgresql’
  • port: ‘5432’
  • database: ‘gitlabhq_production’
    As stated, there is no file at /var/opt/gitlab/backups/db. In fact there is no directory. If I manually create it, it is removed.
    I’m running it in a cron job, but if I run it manually, I get the same results. And I get this on the command line:
    pg_dump: error: invalid column numbering in table “epic_user_mentions”

Usually that directory will be created and removed automatically. Once backups have completed the only contents of /var/opt/gitlab/backups is the tar files.

What’s the permissions for your /var/opt/gitlab/backups directory? Mine look like this:

root@gitlab:~# ls -lha /var/opt/gitlab/ | grep backups
drwx------  2 git               root       4.0K Dec 13 20:49 backups

drwx------ 2 git root 4096 Dec 20 23:50 backups

What version of Gitlab exactly?

GitLab Enterprise Edition v16.6.2-ee

After several gitlab updates, the backup is no longer returning success.
It’s still failing with exactly the same message, however.
When I try to install the latest gitlab version, the update fails and it reports that I need to upgrade postgres.
When I run sudo gitlab-ctl pg-upgrade, it fails and the error in the log file is:
pg_dump: error: invalid column numbering in table “epic_user_mentions”

I really need help with this.
I can’t upgrade gitlab because pg-upgrade fails.
I can’t upgrade postgres because of the column numbering error.
And gitlab isn’t working now and I don’t have access to my source control.