Replace this template with your information
Describe your question in as much detail as possible:
Today, I used yum update command on my CentOS box. One of the updated components was Gitlab. It is now updated to Gitlab CE 13.0.6 (gitlab-ce-13.0.6-ce.0.el7.x86_64). However, now that it is updated, the service was unable to start.
-
What are you seeing, and how does it differ from what you expect to see?
I’m seeing error 502 when accessing Gitlab page instead of the Gitlab content.
-
Consider including screenshots, error messages, and/or other helpful visuals
-
What version are you on (Hint: /help) ? and are you using self-managed or gitlab.com?
13.0.6-CE. I’m using a self-managed Gitlab.
-
What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
Tried to run gitlab-ctl pg-upgrade and gitlab-ctl reconfigure but no luck due to the database was not able to start.
To avoid data loss that I’m afraid, I tried to do backup using the gitlab-backup command. It failed as well due to the database issue.
Dumping PostgreSQL database gitlabhq_production … pg_dump: [archiver (db)] connection to database “gitlabhq_production” failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”?
Please help on what to do to fix this issue urgently.
Thank you
To save your data, you can try to downgrade gitlab-ce to make a backup.
A little documentation of yum downgrade:
https://access.redhat.com/solutions/29617
Thanks for the response.
Although not so straight-forward, I managed to downgrade the Gitlab to latest of v12. It is up and running now.
Thank you
You’re welcome !
It dosn’t solve your problem, but now you can restore a working intance of gitlab and have a backup file.
If someone knows how to solve this, tell us !
Hi,
it seemed that your database migration did not happen. It would be worthwhile to know which version you had been using before to get an idea of missing migration runs.
The steps from 12.x involve to at least upgrade to the latest 12.10.x release first, and then do the upgrade to 13.x. There is a mechanism in place which will prevent a major jump though - https://docs.gitlab.com/omnibus/update/#mandatory-upgrade-paths-for-version-upgrades
Please show the output of
yum info gitlab-ce
and share the package repository configuration from /etc/yum.repos.d as well.
Cheers,
Michael
Background details: PostgreSQL relies on a binary structure for the data storage requiring migrations between major versions. Since GitLab 13.0 requires at least PostgreSQL 11.x as backend version, and 12.8 provided 11.7 already with doing the migrations.
Hi,
Please find the output of the command yum info gitlab-ce below.
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Loading mirror speeds from cached hostfile
* base: ossm.utm.my
* epel: sg.fedora.ipserverone.com
* extras: centos.mirror.angkasa.id
* updates: ossm.utm.my
Installed Packages
Name : gitlab-ce
Arch : x86_64
Version : 12.10.11
Release : ce.0.el7
Size : 1.7 G
Repo : installed
From repo : gitlab_gitlab-ce
Summary : GitLab Community Edition (including NGINX, Postgres, Redis)
URL : https://about.gitlab.com/
License : MIT
Description : GitLab Community Edition (including NGINX, Postgres, Redis)
Available Packages
Name : gitlab-ce
Arch : x86_64
Version : 13.0.6
Release : ce.0.el7
Size : 695 M
Repo : gitlab_gitlab-ce/x86_64
Summary : GitLab Community Edition (including NGINX, Postgres, Redis)
URL : https://about.gitlab.com/
License : MIT
Description : GitLab Community Edition (including NGINX, Postgres, Redis)
Here is the list of my /etc/yum.repos.d:
[root@git ~]# ls -l /etc/yum.repos.d/
total 68
-rw-r--r--. 1 root root 1664 Apr 8 06:01 CentOS-Base.repo
-rw-r--r--. 1 root root 1309 Apr 8 06:01 CentOS-CR.repo
-rw-r--r--. 1 root root 649 Apr 8 06:01 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 314 Apr 8 06:01 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 630 Apr 8 06:01 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Apr 8 06:01 CentOS-Sources.repo
-rw-r--r--. 1 root root 7577 Apr 8 06:01 CentOS-Vault.repo
-rw-r--r--. 1 root root 616 Apr 8 06:01 CentOS-x86_64-kernel.repo
-rw-r--r--. 1 root root 1050 Sep 18 2019 epel.repo
-rw-r--r--. 1 root root 1149 Sep 18 2019 epel-testing.repo
-rw-r--r--. 1 root root 773 Aug 23 2019 gitlab_gitlab-ce.repo
-rw-r--r--. 1 root root 773 Aug 23 2019 gitlab_gitlab-ee.repo
-rw-r--r--. 1 root root 669 May 2 2019 ius-archive.repo
-rw-r--r--. 1 root root 591 May 2 2019 ius.repo
-rw-r--r--. 1 root root 669 May 2 2019 ius-testing.repo
-rw-r--r--. 1 root root 821 Feb 21 12:48 runner_gitlab-runner.repo
And here is the content of the gitlab_gitlab-ce.repo.
[root@git ~]# cat /etc/yum.repos.d/gitlab_gitlab-ce.repo
[gitlab_gitlab-ce]
name=gitlab_gitlab-ce
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/7/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
[gitlab_gitlab-ce-source]
name=gitlab_gitlab-ce-source
baseurl=https://packages.gitlab.com/gitlab/gitlab-ce/el/7/SRPMS
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey
https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey/gitlab-gitlab-ce-3D645A26AB9FBD22.pub.gpg
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
I was under the assumption that since the update comes from the yum, it is safe to proceed. It is not the case though eventually. Right now, I’m running the latest version of v12 (12.10.11). Prior to this, I was running version 12.9.2-EE. I should have been more careful the next time.
Thank you. Let me know if you need further information.
If you were receiving the 502 bad gateway error after the upgrade, one thing I found I needed to do was give Gitlab 13 more compute resources. If I remember correctly, each of the core Gitlab services consumes its own thread and the Unicorn service was timing out because I did not have enough compute power for it to start before its timeout period resolved. Restarting the Gitlab services always resulted in the Unicorn service failing. Upgrading my VM from 2 cores to 4 cores and rebooting resolved this issue for me.
Hi,
thanks for the all the details. I wanted to make sure that you are using the right repositories as well as that the upgrade path is correct. Based on this information, we can try to work on possible solutions.
I did not do that myself, so bear with me when I am linking some docs I have found and am reading.
My guess now is that you had disabled the PostgreSQL server update beforehand, and your 12.10 setup is not running with PostgreSQL 11.7 but 10.x.
On your current installation, please try to run gitlab-ctl pg-upgrade to ensure that 12.10 really runs the latest expected database version.
Once that succeeds, the package upgrade to 13.x can happen.
Cheers,
Michael
Hi,
Thanks for the references shared and also the tips. I have checked that I do not have the file to indicate that I want to disable the PostgreSQL upgrade, so I’m not so sure why it wasn’t upgraded during the previous version upgrade.
I will try to run the gitlab-ctl pg-upgrade command to migrate the database to see if that works, but would probably wait when everyone else is not using the system.
Thanks again.
1 Like
