PostgreSQL cannot start due to 'wal_keep_segments' parameter

Recently tried to upgrade my GitLab instance. I’m not actually sure what version I was previously running but it was probably from 2020. I’m using the Apt repository and the gitlab-ce packages on Debian Bookworm.

PostgreSQL is failing to start:

2023-07-15_10:49:44.65509 2023-07-15 10:49:44.655 GMT [368094] LOG:  unrecognized configuration parameter "wal_keep_segments" in file "/var/opt/gitlab/postgresql/data/runtime.conf" line 30
2023-07-15_10:49:44.65511 2023-07-15 10:49:44.655 GMT [368094] FATAL:  configuration file "/var/opt/gitlab/postgresql/data/runtime.conf" contains errors

In addition, reconfigure fails:

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run[2023-07-15T11:50:17+01:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.

    * bash_hide_env[migrate gitlab-rails database] action run
      [execute] rake aborted!
                ActiveRecord::ConnectionNotEstablished: could not connect to server: No such file or directory
                	Is the server running locally and accepting
                	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

I tried deleting runtime.conf and having reconfigure recreate it, but it continues to create a file with the bad ‘wal_keep_segments’ parameter. PostgreSQL version seems to be 13:

amoe@loopkever $ /opt/gitlab/embedded/postgresql/13/bin/psql --version                                                                                                                             
psql (PostgreSQL) 13.11

How could I fix this?

Same issue migrating from v15.11.7 to v16.0.0 and already solved

First, you can check your gitlab version under admin area.

Solved upgrading posgresql to 13.11 using the following command (it depends of your version):

# not using docker
gitlab-ctl pg-upgrade -V 13

# using docker at your compose location
docker compose exec -it <service-name> gitlab-ctl pg-upgrade -V 13

Maybe you need to move or remove a folder called data-13 but command response will guide you.

From 2020 you are maybe using gitlab v13 and you may need to upgrade to postgresql 12, upgrade gitlab until postgresql 13 compatible version and then upgrade to postgresql 13.

Hope this helps!

Thanks for the idea @hardenerdev. When I try this unfortunately I get the error message:

$ sudo gitlab-ctl pg-upgrade -V 13
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:50:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/my.gitlab.server.net.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)

I can’t run reconfigure due to the error mentioned above…

@amoe I’ve been having the same problem. Here’s what I did to resolve: reinstall to the latest version of Gitlab that supports the Postgresql version (in my case, it was the latest version of Gitlab 15), then run gitlab-ctl reconfigure, then run the pg-upgrade command that @hardenerdev provided.

Thank you @hardenerdev for the solution!

Thank you everybody. The steps for me were as follows:

• I had to switch my apt repository distribution name from bookworm back to bullseye to get the older versions to show up.
• apt-cache policy gitlab-ce would show the older versions.
• I had to downgrade to a v14 version, then upgrade to a v15 version, then finally upgrade to a v16 version.

As I only use the installation occasionally, I didn’t realise this… many thanks.