Postgresql wont start -> FATAL: could not access status of transaction

After a powerdown crash gitlab server wont come back up. gitlab-ctl tail postgresql spawn this log message:

2022-05-19_12:21:46.60893 LOG:  starting PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2022-05-19_12:21:46.60977 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"  
2022-05-19_12:21:46.69933 LOG:  database system was shut down at 2022-05-19 11:45:18 GMT                                                                                                                                                                                                                                     
2022-05-19_12:21:46.69953 FATAL:  could not access status of transaction 979
2022-05-19_12:21:46.69954 DETAIL:  Could not open file "pg_multixact/offsets/0000": No such file or directory.
2022-05-19_12:21:46.70005 LOG:  startup process (PID 9664) exited with exit code 1
2022-05-19_12:21:46.70006 LOG:  aborting startup due to startup process failure
2022-05-19_12:21:46.71676 LOG:  database system is shut down

Before this it complained about missing pg_ directories which i recreated and error message about this when away.

Had to run fsck due to the loss of power, Some files might have gone corrupted.

I am not sure how to interpret this erros message(s), hope someone can give some help to get my gitlab server backonline?

Regards

Thor

I would suggest you look at restoring your system from a backup. This link explains how to do it.:

Basically, I would install the same version of Gitlab that was used when the last backup was taken and let it configure an empty installation so that the services are running. Then copy the gitlab.rb and gitlab-secrets.json files to /etc/gitlab on the new server and reconfigure. At this point you are then ready to restore your last backup which should be under /var/opt/gitlab/backups.

The link explains the restore process in more detail.

1 Like

Ok, I understand it that serious. I see the backups in the mentioned directory.

Thanks

Thor

Is there a way to find out which version is last when server not running? I updates via Ubuntu apt.

Thor

Hi,

Yes it’s in the filename, here is an example from mine:

1652911310_2022_05_19_14.10.2_gitlab_backup.tar

my version is 14.10.2 so the last few digits. You can also check/verify that against your system if you didn’t upgrade it recently:

dpkg -l | grep -i gitlab

the version will also be shown in the package installed. Assuming you didn’t upgrade in between the last backup and the crash, then you have the same version installed most likely so they would then match. So for example:

dpkg -l | grep -i gitlab
ii  gitlab-ce                              14.10.2-ce.0                                                               amd64        GitLab Community Edition (including NGINX, Postgres, Redis)

If you did upgrade in between the last backup and the crash, then obviously those versions will not match in that instance.

1 Like

Ok also found a file version-manifest.txt in /opt/gitlab saying gitlab-ce 14.8.2, and dpkg command output is 14.8.2-ce.0 so it seems like a match. However last tar backyp file says 14.8.1

Thanks again.

The command sudo gitlab-backup restore BACKUP=... takes very long time with no feedbacks exept a message:

Be sure to stop Puma, Sidekiq, and any other process that connects to the database before proceeding. For Omnibus installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations

And

Before restoring the database, we will remove all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed.

Then no messages, program seems to run from htop but no CPU usage. Is supposed to be like that? It has run for hours.

How big is the backup file? Mine are about 5gb as we don’t have a lot of repos, or ones that are huge, so it’s relatively quick.

Mine is about 2.6 MB. It was on a LVM mapped drive (rather big). Never restored anything it seems, get message about file is extracted and I see the extracted file in the ./backups/db directory. After that just the message as descbed above in red fonts. Then all “quiet”, no reports or errors. Made the restore run for hours…

The gitlab server is up and running and I followed the instruction on Gitlab wepages. (Back up and restore GitLab | GitLab)

BTW: The initial file size of the first bacup of the clean installation is about 10.6 MB

I would say a 2.6MB backup is small, unless your repos don’t have much in them. You could check/verify this with:

du -sh /var/opt/gitlab/git-data

based on the size of this, we can get an idea of whether your backups actually worked properly, or whether something is wrong with them. If so, that would explain why the backup is taking so long to restore because it’s invalid, or corrupted or something.

1008M is the current dh output. Guess something is wrong. I did not configured the old server, not sure what has happened. No regular backup was configured. My guess is has been backuped during upgrades.

I have a copy of the old git-data directory and it reports 992MB so clearly something is wrong with the backup.