Hello.
My company decided to buy Gitlab support.
While we finalize the process, I tried to update Gitlab to the last version.
I have some troubles upgrading my Gitlab version from 11.11.3 to the last one.
I ran thought different errors.
I have followed the pages telling to update incrementally, and I tried to make sure all common problems doesn’t occur.
I have replicated our production server with a backup, following the instructions given here :
https://docs.gitlab.com/11.11/ee/raketasks/backup_restore.html
https://docs.gitlab.com/11.11/omnibus/settings/backups.html
After I ensured that the server was running, I restored like this :
On the production server
gitlab-rake gitlab:backup:create SKIP=builds,artifacts,registry
tar cfz /mnt/nfs/backups/gitlab/$(date “+etc-gitlab-%Y%m%d-%H%M.tgz”) -C / etc/gitlab
On the test server :
gitlab-ctl reconfigure
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-ctl status
gitlab-rake gitlab:backup:restore RAILS_ENV=production BACKUP=timestamp_of_backup force=yes
tar zxvf /mnt/nfs/backups/gitlab/(backup) -C /etc/gitlab
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true
The 11.11.3 restore was done sucessfully, and all seems to be working on the test server. The issue is the upgrade.
I had some issues with postgresql update and db:migrate.
Postgresql update :
Creating temporary data directory:Traceback (most recent call last):
11: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
9: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
8: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
7: from /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:140:in `block in load_file'
5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:194:in `general_upgrade'
4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:152:in `common_pre_upgrade'
3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:226:in `create_temp_data_dir'
2: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:383:in `progress_message'
1: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:228:in `block in create_temp_data_dir'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:39:in `run_pg_command': undefined method `[]' for nil:NilClass (NoMethodError)
When I tried to upgrade to 12.0.0, I saw the given log during db.migrate :
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200622-7400-v07k7r" ----
STDOUT: rake aborted!
Exception: Your database is missing the 'parent_id' column from the 'epics' table that is present for GitLab EE.
Even though it looks like you're running a CE installation, it appears
you may have installed GitLab EE at some point. To migrate to GitLab 12.0:
1. Install GitLab 11.11.3 EE
2. Install GitLab 12.0.x CE
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190402150158_backport_enterprise_schema.rb:917:in `raise_ee_migration_error!'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190402150158_backport_enterprise_schema.rb:897:in `check_ee_columns!'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190402150158_backport_enterprise_schema.rb:889:in `block in check_schema!'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190402150158_backport_enterprise_schema.rb:888:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190402150158_backport_enterprise_schema.rb:888:in `check_schema!'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20190402150158_backport_enterprise_schema.rb:121:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20180824202952 AddOutboundRequestsWhitelistToApplicationSettings: migrating
-- add_column(:application_settings, :outbound_local_requests_whitelist, :string, {:array=>true, :limit=>255})
-> 0.0009s
== 20180824202952 AddOutboundRequestsWhitelistToApplicationSettings: migrated (0.0009s)
== 20190219201635 AddAssetProxySettings: migrating ============================
-- add_column(:application_settings, :asset_proxy_enabled, :boolean, {:default=>false, :null=>false})
-> 0.0467s
-- add_column(:application_settings, :asset_proxy_url, :string, {})
-> 0.0004s
-- add_column(:application_settings, :asset_proxy_whitelist, :text, {})
-> 0.0003s
-- add_column(:application_settings, :encrypted_asset_proxy_secret_key, :text, {})
-> 0.0003s
-- add_column(:application_settings, :encrypted_asset_proxy_secret_key_iv, :string, {})
-> 0.0003s
== 20190219201635 AddAssetProxySettings: migrated (0.0480s) ===================
== 20190402150158 BackportEnterpriseSchema: migrating =========================
-- table_exists?(:epics)
-> 0.0005s
-- column_exists?(:epics, :parent_id)
-> 0.0018s
STDERR:
---- End output of "bash" "/tmp/chef-script20200622-7400-v07k7r" ----
Ran "bash" "/tmp/chef-script20200622-7400-v07k7r" returned 1
I don’t know what was done with the server before I was in charge of it but I assume that the log is right and I have somehow EE tables in my CE installation.
I tried to change the version then from CE 11.11.3 to EE 11.11.3 and I have each time some errors during db:migrate due to already existing tables.
I’m confused. I don’t see why there would be a problem if the tables and columns supposed to be created during the migration are already present.
I tried to remove it manually in the postgresql cli, but after 22 attempts I wonder if there is a way to just skip duplicateTables (I don’t know how modify the migration file to do something like CREATE TABLE IF NOT EXISTS)…
Attempts :
relation “prometheus_alerts” already exists
column “saml_discovery_token” of relation “namespaces” already exists
column “last_update_started_at” of relation “clusters_applications_prometheus” already exists
column “pseudonymizer_enabled” of relation “application_settings” already exists
column “user_id” of relation “lists” already exists
column “file_template_project_id” of relation “application_settings” already exists
relation “software_licenses” already exists
relation “software_license_policies” already exists
column “trial_ends_on” of relation “namespaces” already exists
relation “operations_feature_flags” already exists
relation “operations_feature_flags_clients” already exists
column “custom_project_templates_group_id” of relation “application_settings” already exists
relation “protected_environments” already exists
relation “protected_environment_deploy_access_levels” already exists
relation “packages_packages” already exists
relation “packages_maven_metadata” already exists
column “new_epic” of relation “notification_settings” already exists
column “roadmap_layout” of relation “users” already exists
column “snowplow_enabled” of relation “application_settings” already exists
column “snowplow_collector_uri” of relation “application_settings” already exists
column “snowplow_site_id” of relation “application_settings” already exists
column “snowplow_cookie_domain” of relation “application_settings” already exists
…
Could you please help me?
Once this issue is fixed, I could continue and update patch by patch with eventually disabled SQL update and disabled auto reconfigure to do the migrations manually.