Had an issue with trying to upgrade from GitLab CE omnibus 15.11.13 to 16.3.7 and want to get the postgresql piece right before trying again. I tried to upgrade GitLab CE to 16.3.7 and postgresql service showed stopped and would not start at all. I also wasn’t able to create any backups either from GitLab CE (gitlab-rake gitlab:backup:create)
This is on RHEL8 on an ec2 in an air-gapped environment that has access to an internal dnf repo. Looks like there is a way to install postgresql on RHEL8 (Does Red Hat provide Postgresql 14 - Red Hat Customer Portal). Thinking this has to be done 1st before attempting to upgrade, correct?
Will there be issue with me installing these v14 postgresql packages at the OS level? If that is case, how does Gitlab CE know that postgresql v14 is installed and to use it?
Once this is done, then from GitLab CE run the upgrade command (gitlab-ctl pg-upgrade) before upgrading GitLab CE? As I did it this way before and postgresql would not start.
But let’s back up a bit first. Is your current PostgreSQL the bundled one that comes with the gitlab-ce package? If so, you will have to export/dump the database first. Then once that is done, you can think about installing PostgreSQL 14 and restore the dumped database. Create a user within PostgreSQL for Gitlab to use and ensuring it is dbOwner of the restored database, and then you need to look at reconfiguring Gitlab to use the external PostgreSQL database instead: Configure GitLab using an external PostgreSQL service | GitLab
The Gitlab Docs link explains what you need to do in more detail for setting up the external Postgres. You will also need to ensure that your external PostgreSQL is supported as per this link: Installation system requirements | GitLab since any upgrades in the future might cause you big problems.
The bundled PostgreSQL in gitlab-ce package gets upgraded itself at various points, which means less work. So you need to be aware of the extra work of running your own external Postgres.
TBH, upgrade from 15.11.13 to 16.3.7 should work, so not really sure what the problem was that you feel you need to fix PostgreSQL. As per the docs: PostgreSQL versions shipped with the Linux package | GitLab the bundled PostgreSQL for Gitlab was at version 13.x until 14.x became available in 16.4.3.
I would be tempted to stay with the bundled one unless you really need to use an external PostgreSQL. If you do this:
gitlab-psql -V
what version does it report back? It should either return 12.12 or 13.11. If you see it report back 12.12, then I would be tempted to do:
gitlab-ctl pg-upgrade
to ensure you are on 13.11 before upgrading to 16.3.7. Then once you have upgraded to 16.3.7 you can go to 16.7.8 which is the next upgrade on the upgrade path, and after upgrade to 16.7.8 do:
gitlab-ctl pg-upgrade
so that you then end up on 14.10. You can then upgrade to 16.11.6 and then 17.1.2.
Here is the output for gitlab-rake gitlab:env:info as info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 3.0.6p216
Gem Version: 3.2.33
Bundler Version: 2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.11
Sidekiq Version: 6.5.7
Go Version: unknown
GitLab information
Version 15.11.13
Revision: cc3748fcd2d
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 13.11
URL: <-removed->
HTTP Clone URL: <-removed->
SSH Clone URL: <-removed->
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.18.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
I’m working with GitLab CE 15.11.13 in an air-gapped environment on a RHEL8 ec2, so I don’t have the capability to upgrade Postgresql (gitlab-ctl pg-upgrade) on the fly.
But if what your saying is correct, seems like I should be able to upgrade to 16.3.7 with no issue to Postgres.
I’ve created like a pre-upgrade, install and post-upgrade checklist to get to 15.11.13. I want to use this checklist to get to 16.3.7, let me know if there are any holes or something I can do better:
Pre-upgrade:
create snapshot on ebs volume of GitLab CE ec2
Run the following gitlab commands as a sanity check:
gitlab-ctl restart
gitlab-ctl status
gitlab-rake gitlab:check
gitlab-rake gitlab:doctor:secrets
gitlab-rake gitlab:env:info
Create backup tar from GitLab CE (gitlab-rake gitlab:backup:create) and backup to s3 bucket. Also backup /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.jsonto s3 bucket
Log into postgresql and log into GitLab CE Postgresql database and view tables
Install
Use dnf to remove GitLab CE 15.11.13
Use dnf to install GitLab CD 16.3.7
Run the following command: gitlab-ctl reconfigure
Post-Upgrade
Run the following commands as sanity check
gitlab-ctl restart
gitlab-ctl status
gitlab-rake gitlab:check
gitlab-rake gitlab:doctor:secrets
Log into postgresql and log into GitLab CE Postgresql database and view tables
Capture postgresql version from CLI with the following command: gitlab-rake gitlab:env:info
Log into GitLab UI, confirm version and test functionality
Use git, attempt to pull and push
EDIT: Looking thru the Release notes from Upgrade GitLab. Just want to confirm this. Have to upgrade to 16.3.0 and then 16.3.7. Is this correct?