After attempt of update GitLab is broken

I’ve tried to update GitLab from 12 to 13, but now the GitLab instance doesn’t work at all. If I tried to reconfigure GitLab a post in logs appears:

2021-06-03_11:39:30.96425 FATAL: database files are incompatible with server 
2021-06-03_11:39:30.96427 DETAIL: The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 12.6

If I try to revert the version of PostgreSQL I get:

    Traceback (most recent call last):
    	7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
    	6: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
    	5: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
    	4: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
    	3: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
    	2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:38:in `block in load_file'
    	1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:62:in `get_node_attributes'
    /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:47:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/CoreD.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)

If I try to update GitLab I get the error:

Running handlers:
There was an error running gitlab-ctl reconfigure:

    database_objects[postgresql] (postgresql::standalone line 28) had an error: RuntimeError: postgresql_user[gitlab] (/opt/gitlab/embedded/cookbooks/cache/coo
    kbooks/gitlab/resources/database_objects.rb line 19) had an error: RuntimeError: execute[create gitlab postgresql user] (/opt/gitlab/embedded/cookbooks/cac
    he/cookbooks/postgresql/resources/user.rb line 11) had an error: RuntimeError: PostgreSQL did not respond before service checks were exhausted

I also tried to change permission to GitLab files, reinstall PostgreSQL, install the old version of PostgreSQL, install the new version of PostgreSQL, change shared_buffes size in .rb config, change shared_buffes size in OS

I also tried to deploy GitLab on another machine but when I copy repositories I get the error:

     * Failed to move repo: 14:failed to connect to all addresses. debug_error_string:{"created":"@1622751694.185383828","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3951,"referenced_errors":[{"created":"@1622751694.185376145","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
     * Failed trying to create .........

How can I fix that?

Hi,

Did you see the steps of the database upgrade?
Before upgrading your instance, from version 12, you have to upgrade your database first, and then, upgrade to GitLab 13.

As I can see, the database of your instance still in 9.6. With GitLab 12, you must upgrade to PostgreSQL 10 and PostgreSQL 11.
Because as it is mentioned for version 13 of GitLab “PostgreSQL 9.6 and 10 have been removed from the package.”

All information that you need is here: GitLab 13 specific changes | GitLab

Hoping this information will help you

I’ve understood my mistake. But what have I to do? Now GitLab returns 13 version and nothing works. P.S.: I’ve installed docker with the same previous version (12) and import bare repositories and finally it works. But I still wanna know how can I fix the problem if this one will repeat after my wrong actions.

Chances are a lot of your problems occured without following the upgrade guide: Upgrading GitLab | GitLab

You should follow the upgrade path: Upgrading GitLab | GitLab

If you follow the upgrade path, then the database should be upgraded for you automatically. I don’t use docker, so cannot help you with specific instructions for this. Since normally with deb/rpm omnibus packages, you can also use:

gitlab-ctl pg-upgrade

to do a db upgrade prior to upgrading to a newer version. But as I mentioned, providing you follow the upgrade path, then this db upgrade should be done automatically for you anyway. It always has for me, I think maybe once when I was on 12.9.2 or 12.10.x I did run db upgrade manually, but can’t remember exactly.

The upgrade docs have procedures how to do the docker upgrades and can be followed: GitLab Docker images | GitLab

I get:

 Traceback (most recent call last):
    	7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
    	6: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
    	5: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
    	4: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
    	3: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
    	2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:38:in `block in load_file'
    	1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:62:in `get_node_attributes'
    /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:47:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/CoreD.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)

gitlab-psql - V returns 12 version. Problem GitLab instance is used without docker container. It is using the native Linux system.

Error says has reconfigure been run yet so:

gitlab-ctl reconfigure

Then gitlab-ctl restart to make sure all is working properly.

If you are using Linux packages so deb or rpm then you are using omnibus edition so follow the upgrade docs. All the info is there.

Then I get the error:

Running handlers:
There was an error running gitlab-ctl reconfigure:

    database_objects[postgresql] (postgresql::standalone line 28) had an error: RuntimeError: postgresql_user[gitlab] (/opt/gitlab/embedded/cookbooks/cache/coo
    kbooks/gitlab/resources/database_objects.rb line 19) had an error: RuntimeError: execute[create gitlab postgresql user] (/opt/gitlab/embedded/cookbooks/cac
    he/cookbooks/postgresql/resources/user.rb line 11) had an error: RuntimeError: PostgreSQL did not respond before service checks were exhausted

And after each reconfigure command a post is pushed to the log file:

2021-06-03_11:39:30.96425 FATAL: database files are incompatible with server 
2021-06-03_11:39:30.96427 DETAIL: The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 12.6

Hi @wusikijeronii
If you can, revert to a backup with GitLab version 12 and do the upgrade again, following the upgrade path.
If you can’t rollback, you can try to upgrade the PG datafiles manually like I have explained here

1 Like

Thank you. Yep, now reconfigure was successful;