After a powerdown crash gitlab server wont come back up. gitlab-ctl tail postgresql spawn this log message:
2022-05-19_12:21:46.60893 LOG: starting PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2022-05-19_12:21:46.60977 LOG: listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2022-05-19_12:21:46.69933 LOG: database system was shut down at 2022-05-19 11:45:18 GMT
2022-05-19_12:21:46.69953 FATAL: could not access status of transaction 979
2022-05-19_12:21:46.69954 DETAIL: Could not open file "pg_multixact/offsets/0000": No such file or directory.
2022-05-19_12:21:46.70005 LOG: startup process (PID 9664) exited with exit code 1
2022-05-19_12:21:46.70006 LOG: aborting startup due to startup process failure
2022-05-19_12:21:46.71676 LOG: database system is shut down
Before this it complained about missing pg_ directories which i recreated and error message about this when away.
Had to run fsck due to the loss of power, Some files might have gone corrupted.
I am not sure how to interpret this erros message(s), hope someone can give some help to get my gitlab server backonline?
I would suggest you look at restoring your system from a backup. This link explains how to do it.:
Basically, I would install the same version of Gitlab that was used when the last backup was taken and let it configure an empty installation so that the services are running. Then copy the gitlab.rb and gitlab-secrets.json files to /etc/gitlab on the new server and reconfigure. At this point you are then ready to restore your last backup which should be under /var/opt/gitlab/backups.
The link explains the restore process in more detail.
Yes it’s in the filename, here is an example from mine:
1652911310_2022_05_19_14.10.2_gitlab_backup.tar
my version is 14.10.2 so the last few digits. You can also check/verify that against your system if you didn’t upgrade it recently:
dpkg -l | grep -i gitlab
the version will also be shown in the package installed. Assuming you didn’t upgrade in between the last backup and the crash, then you have the same version installed most likely so they would then match. So for example:
dpkg -l | grep -i gitlab
ii gitlab-ce 14.10.2-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis)
If you did upgrade in between the last backup and the crash, then obviously those versions will not match in that instance.
Ok also found a file version-manifest.txt in /opt/gitlab saying gitlab-ce 14.8.2, and dpkg command output is 14.8.2-ce.0 so it seems like a match. However last tar backyp file says 14.8.1
The command sudo gitlab-backup restore BACKUP=... takes very long time with no feedbacks exept a message:
Be sure to stop Puma, Sidekiq, and any other process that connects to the database before proceeding. For Omnibus installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations
And
Before restoring the database, we will remove all existing tables to avoid future upgrade problems. Be aware that if you have custom tables in the GitLab database these tables and all data will be removed.
Then no messages, program seems to run from htop but no CPU usage. Is supposed to be like that? It has run for hours.
Mine is about 2.6 MB. It was on a LVM mapped drive (rather big). Never restored anything it seems, get message about file is extracted and I see the extracted file in the ./backups/db directory. After that just the message as descbed above in red fonts. Then all “quiet”, no reports or errors. Made the restore run for hours…
I would say a 2.6MB backup is small, unless your repos don’t have much in them. You could check/verify this with:
du -sh /var/opt/gitlab/git-data
based on the size of this, we can get an idea of whether your backups actually worked properly, or whether something is wrong with them. If so, that would explain why the backup is taking so long to restore because it’s invalid, or corrupted or something.
1008M is the current dh output. Guess something is wrong. I did not configured the old server, not sure what has happened. No regular backup was configured. My guess is has been backuped during upgrades.
I have a copy of the old git-data directory and it reports 992MB so clearly something is wrong with the backup.
I had a very similar issue, I got plenty of FATAL: the database system is starting up.
Amongst other errors causing my instance not to start correcttly i found this one,
Essentially saying delete / rename these files
/var/opt/gitlab/postgresql/.s.PGSQL.5432
/var/opt/gitlab/postgresql/.s.PGSQL.5432.lock
once i did that my instance resolved itself.
Hope it helps someone looking when their instance crashes…
Or power is inadvertantly lost.