A broken configuration is preventing an upgrade

How to Use GitLab Self-managed
1

When I try to update the GitLab instance using yum, I get the following error:

Transaction test succeeded
Running transaction
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/hostgitlab.domain.com.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.

However, running gitlab-ctl reconfigure doesn’t fix the problem and I haven’t been able to figure out how to fix it, so I am stuck. I will try to attach the full logs of both of these commands and hope that someone here can help me figure out what is wrong.

2

I can’t figure out how to attach the log files (ZIP file). And I forgot to mention that the JSON file it complains about is syntactically correct according to jq.

3

Well, I decided to figure out if I can help you or not, so I used my test server to check/verify what I’m about to suggest. That way if it breaks, it’s not important for me. However my system didn’t break and still works. My Gitlab version is 15.3.2 on my test server, and there is an upgrade available to 15.3.3, so I will then do the upgrade afterwards.

Since you mentioned it says the file is in some way corrupted, I decided to do this:

mv /opt/gitlab/embedded/nodes/gitlab.mydomain.com.json /root/

now /opt/gitlab/embedded/nodes is empty, so we now reconfigure gitlab:

gitlab-ctl reconfigure

after reconfigure, I find that /opt/gitlab/embedded/nodes/gitlab.mydomain.com.json has been recreated. So the reconfigure did what I was hoping it would do. At this point I stopped and started Gitlab, and then went to my web browser to make sure everything was working. Yes it was working fine.

So, now I update Gitlab on my install. In my instance I’m using Debian, so apt instead of yum, but the end result should be the same.

I then checked/verified again from the web browser, to make sure I had 15.3.3, and all seems to be working perfectly fine.

So for you, you move the hostgitlab.domain.com.json somewhere else so that /opt/gitlab/embedded/nodes is empty, reconfigure Gitlab, and then restart/verify all is OK, and then upgrade Gitlab, and hopefully you will be sorted out.

I’m not sure what your current Gitlab version is, but if you don’t upgrade regularly, I suggest you find out what Gitlab version is installed, and then check the Gitlab documentation to make sure you follow the upgrade path. Doing a yum update or dnf update might end up with it installing the latest Gitlab over your install, not following the upgrade path and potentially breaking your Gitlab install. You can find more on that here: Upgrading GitLab | GitLab

4

Thanks for your quick response, but I should have mentioned that that file is most likely a red herring since it gets created automatically every time. I’m at version 14.10.0 and am trying to upgrade to 15.0.0 as a stepping stone to the latest version. I’ve always upgraded successfully with yum update gitlab-ee in the past. I wish I could attach a log file (let me know if there is a way to do that and I’m just missing it), but here is the relevant portion of the failure:

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run

      ================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Command execution failed. STDOUT/STDERR suppressed for sensitive resource

      Resource Declaration:
      ---------------------
      suppressed sensitive resource output

      Compiled Resource:
      ------------------
      suppressed sensitive resource output

      System Info:
      ------------
      chef_version=15.17.4
      platform=redhat
      platform_version=7.7
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client


    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================


    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55:
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60:
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end
5

FYI both, I edited the posts and removed the FQDN hostname potentially leaking sensitive company details.

I’m at version 14.10.0 and am trying to upgrade to 15.0.0 as a stepping stone to the latest version. I’ve always upgraded successfully with yum update gitlab-ee in the past. I wish I could attach a log file (let me know if there is a way to do that and I’m just missing it),

Recommend opening a support ticket with attaching the log files, in case you have an active enterprise subscription. Do not recommend attaching log files to public forums as they could contain sensitive data.

I’m at version 14.10.0 and am trying to upgrade to 15.0.0 as a stepping stone to the latest version.

Upgrading GitLab | GitLab mentions an upgrade path to 15.0 with using 15.0.2 in 14.10.5 -> 15.0.2 which leads me to two suggestions:

  • Upgrade to the latest 14.10.x release (14.10.5)
  • From there, upgrade to the latest 15.0.x release (15.0.5)

Maybe you are triggering an upgrade bug in 14.10.0 or 15.0.0. Latest releases are in Releases | GitLab

1 Like
6

Thanks for editing the post - I should have thought of that before posting.

Unfortunately, I cannot open a case with support (I tried that, and they sent me here), as we are no longer paying for our instance after being a customer for many years (this was a management decision I did not agree with). Our GitLab instance is only being used by one legacy project, and pretty much only for reading.

The only reason I’m trying to upgrade it is to eliminate any potential vulnerabilities. And upgrading to the latest 14.x release doesn’t work either, because I can’t get gitlab-ctl reconfigure to complete successfully on this installation. Is there a way to skip the rails-migration or to find out and fix whatever it is complaining about? I have no idea what the rail migration is trying to accomplish and whether it is even necessary.