Critical error during upgrade Gitlab to version 15

General
1

Hi,
During my upgrade Gitlab CE to version 15 I have follow error:
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:189:in `missing_key’: Missing setting ‘provider’ in ‘connection’ section in ‘terraform_state’ section in /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml (Settingslogic::MissingSetting)

I check this configuration section in file: /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml and I have below entry:

  terraform_state:
    enabled: true
    # The location where terraform state files are stored (default: shared/terraform_state).
    storage_path: /var/opt/gitlab/gitlab-rails/shared/terraform_state
    object_store:
      enabled: true
      remote_directory: "gitlabpipsrv-terraform"
      connection: {"provider":"AWS","region":

I used this command to check my error:

gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending.count'

I tried disable storage for terraform state:

gitlab_rails['terraform_state_enabled'] = false

but this change doesn’t work.
Please, help me because I didn’t use Gitlab from yesterday.

1 Like
2

Have you configured all the provider details, for the options you enabled?

We recently had a customer with this error, and it was because they had

gitlab_rails['artifacts_object_store_enabled'] = true

but no object storage configured (which would contain a provider key). The part about Terraform was not relevant in their case but likely a fall-through after trying to find the missing artifact object storage provider.

3

On reflection, I’m wondering if this mis-configuration was silently allowed to go through on older versions, but now it causes this Terraform error?

4

HI,
I restored my Gitlab instance from backup. I checked my terraform state configuration and it works. I have Terraform States on my AWS bucket. Fo testing, I change my terraform_state configuration in “gitlab.rb” file to this:

gitlab_rails['terraform_state_enabled'] = true
gitlab_rails['terraform_state_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/terraform_state"
gitlab_rails['terraform_state_object_store_enabled'] = true

Gitlab reconfiguration works perfect on 14.10 - it creates file gitlab.yml with follow entry:

  ## Terraform state
  terraform_state:
    enabled: true
    # The location where terraform state files are stored (default: shared/terraform_state).
    storage_path: /var/opt/gitlab/gitlab-rails/shared/terraform_state
    object_store:
      enabled: true
      remote_directory: "terraform"
      connection: {}

But during upgrade to version 15 I have the same error as in my first post :(. This error is during migration.

5

I have the same problem. Trying to upgrade to gitlab-ce 15.0.0 from the Ubuntu focal LTS release I have some strange console messages about

Error executing action “run” on resource ‘bash[migrate gitlab-rails database]’

and in the gitlab-rails migration-log gitlab-rails-db-migrate-2022-05-28-13-52 I find

Settingslogic::MissingSetting: Missing setting ‘provider’ in ‘connection’ section in ‘terraform_state’ section in /
opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml

However in /etc/gitlab/gitlab.rb there is nothing and to my memory never was anything about this “terraform” (whatever this is).

How can this be fixed? Is it possible to inject some settings into /etc/gitlab/gitlab.rb in order to make the migration step succeed?

7

Have you configured all the provider details, for the options you enabled?

We recently had a customer with this error, and it was because they had

gitlab_rails['artifacts_object_store_enabled'] = true

but no object storage configured (which would contain a provider key). The part about Terraform was not relevant in their case but likely a fall-through after trying to find the missing artifact object storage provider.

Oops. This indeed was also the case for me: it showed that accidentally I had enabled “object_store” in the lfs-settings. Disabling this erroneous setting solved my migration problems. Thanks for the hint!

8

Yes I think that the new parsing for gitlab.rb in 15 has this problem for the edge-case where you declare gitlab_rails['xxxxx_object_store_enabled'] = true, but then don’t specify the object storage. So that is, if you have any such declaration but don’t specify an object storage for it, the error will be about a missing one for Terraform.

In @rotdrop’s case it was the LFS which had object storage enabled, but no storage provider supplied, but still the error was about Terraform provider missing.

Since you report that you actually do have AWS buckets to store your terraform, and the provider supplied in you configuration (except for during your test), this error is doubly misleading.

IMO this is a bug and should be resolved either by

  • ignoring missing providers and failing back to file storage (as it did in 14.10 and earlier), or
  • by reporting which provider is really missing, not always saying that it’s Terraform.
9

Thank you @rotdrop for the tips. In my situation I had this entry :slight_smile:

gitlab_rails['uploads_object_store_enabled']

set as enabled. Of course, after disabling this setting solved my migration.
@mlockhart, I analyzed the new source code and I think that the error was made during changes for using global object store configuration.

1 Like
10

Thank you pawel,

I have raised an issue for this: Critical error on upgrade to GitLab 15 when missing arctifact, lfs or uploads object storage provider (#367625) · Issues · GitLab.org / GitLab · GitLab

Note that this occurs when you enable artifact, LFS, or uploads object stores without specifying a provider, but if you enable Terraform State storage without a provider, then the reconfigure actually succeeds. The remaining storages are unaffected.

1 Like