Upgrade Gitlab CE from 11.9.1 to 12.5

Hello,

I want to upgrade my gitlab version from 11.9.1 to 12.5. The gitlab are installed on centos7 server .

So if I understand, I have to update from 11.9.1 -> 11.11.8 -> 12.0.9 -> 12.5, right?

I’ve tried that but when I’m trying to upgrade from 11.11.8 to 12.0.9 (yum install gitlab-ce-12.0.9-ce.0.el7.x86_64) but my upgrade is stuck here :
Updating : gitlab-ce-12.0.9-ce.0.el7.x86_64 1/2
Cleanup : gitlab-ce-11.11.8-ce.0.el7.x86_64 2/2

My production log doesn’t show any error:
tailf /var/log/gitlab/gitlab-rails/production.log Migrating to SetResolvedStateOnVulnerabilities (20191114173624) Migrating to CreateVulnerabilityIssueLinks (20191115091425) Migrating to AddEncryptedFieldsToApplicationSettings (20191120084627) Migrating to EncryptPlaintextAttributesOnApplicationSettings (20191120115530) Migrating to RemovePlaintextColumnsFromApplicationSettings (20191122135327) Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects. Creating scope :join_project. Overwriting existing method MergeRequest.join_project. Creating scope :references_project. Overwriting existing method MergeRequest.references_project. Creating scope :system. Overwriting existing method Note.system.

What I’m missing ?

You can actually skip 12.0.9.

11.9.1 -> 11.11.8 -> 12.5

If a major version like 11.11 is skipped and you go from 11.9.x to 12.x, and it can’t proceed further, it will actually say that in the log.

Example

Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
gitlab preinstall: It seems you are upgrading from major version 11 to major version 12.
gitlab preinstall: It is required to upgrade to the latest 11.11 version first before jumping to the next major version.
gitlab preinstall: Please follow the upgrade documentation at https://docs.gitlab.com/ee/policy/maintenance.html#upgrade-recommendations
error: %pre(gitlab-ce-12.4.2-ce.0.el7.x86_64) scriptlet failed, exit status 1

Try using wget to get the .rpm first. Then do a

yum localinstall <$rpm pkg name>

Otherwise may be something else that’s failing.

2 Likes

Thank you. I will try that now, but it seems I have the same result:

Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2019-12-13 09:31:38 +0000 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2019-12-13 09:37:37 +0000 -- done
2019-12-13 09:37:37 +0000 -- Dumping repositories ...
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping uploads ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping builds ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping artifacts ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping pages ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping lfs objects ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping container registry images ... 
2019-12-13 09:37:37 +0000 -- [DISABLED]
Creating backup archive: 1576229857_2019_12_13_11.11.8_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... done. (0 removed)
  Updating   : gitlab-ce-12.5.4-ce.0.el7.x86_64                                                                                                                                  1/2 
  Cleanup    : gitlab-ce-11.11.8-ce.0.el7.x86_64                                                                                                                                 2/2 

It is normal this step takes a very long time?

For the upgrade I believe it takes a default SQL backup, so the time depends on the the size of your postgresDB. The default one skips other things like wikis and container…etc.

My instance isn’t too big, for PROD it’s something like 10 minutes. 500 users / 1000 projects / we try to limit binary getting checked in. Your miles may vary.

As far as log excerpts. After backup is done it should move onto something like this:

Updating : gitlab-ce-12.4.3-ce.0.el7.x86_64 1/2
Cleanup : gitlab-ce-11.11.8-ce.0.el7.x86_64
* file[/opt/gitlab/sv/gitlab-exporter/down] action delete (up to date)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/gitlab-exporter] action create (up to date)
* ruby_block[wait for gitlab-exporter service socket] action run (skipped due to not_if)
(up to date)
Recipe: monitoring::redis-exporter

  • directory[/var/log/gitlab/redis-exporter] action create (up to date)
  • directory[/opt/gitlab/etc/redis-exporter/env] action create (up to date)
  • env_dir[/opt/gitlab/etc/redis-exporter/env] action create
    • directory[/opt/gitlab/etc/redis-exporter/env] action create (up to date)
    • file[/opt/gitlab/etc/redis-exporter/env/SSL_CERT_DIR] action create (up to date)
      (up to date)

Hopefully the upgrade kept going. Did it finish on any errors?

Other logs should be under /var/log/gitlab and can also tail /var/log/messages.

1 Like

Hello,

I’ve about 600 users / 2000 projects. With the yum localinstall gitlab-ce-12.5.4-ce.0.el7.x86_64.rpm the installation stops in the same place:
Cleanup : gitlab-ce-11.11.8-ce.0.el7.x86_64 2/2.

Now I’m trying to install with the rpm -ivh gitlab-ce-12.5.4-ce.0.el7.x86_64.rpm and it seems the installation is running, on my output installation I’ve this output:

Running handlers:
Running handlers complete
Chef Client finished, 123/927 resources updated in 07 minutes 05 seconds
Running reconfigure: OK
Waiting for Database to be running.
Database upgrade is complete, running analyze_new_cluster.sh

On my /var/log/gitlab/postgresql/current

2019-12-16_09:45:21.86052 LOG:  database system is ready to accept connections
2019-12-16_09:45:25.52066 received TERM from runit, sending INT instead to force quit connections
2019-12-16_09:45:25.52322 LOG:  received fast shutdown request
2019-12-16_09:45:25.52740 LOG:  aborting any active transactions
2019-12-16_09:45:25.52876 LOG:  worker process: logical replication launcher (PID 13354) exited with exit code 1
2019-12-16_09:45:25.52931 LOG:  shutting down
2019-12-16_09:45:25.60799 LOG:  database system is shut down
2019-12-16_09:45:25.71113 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2019-12-16_09:45:25.83508 LOG:  database system was shut down at 2019-12-16 09:45:25 GMT
2019-12-16_09:45:25.84647 LOG:  database system is ready to accept connections

On my /var/log/gitlab/gitlab-rails/production.log

Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :system. Overwriting existing method         Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Started GET "/" for  at 2019-12-16 09:52:57 +0000
Processing by RootController#index as HTML
Redirected to http://example.org/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 81ms (ActiveRecord: 25.5ms | Elasticsearch: 0.0ms)

After 2 hours the outupt on my log are the same…

The hanging seems to be near the analyze_new_cluster.sh part. Do you have a high availability setup or GEO secondary? If yes, I think there’s steps on that with

or


Digging around the upgrade step…

https://github.com/gitlabhq/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/pg-upgrade.rb

unless [:pg_secondary, :geo_secondary].include?(@instance_type)
log ‘Database upgrade is complete, running analyze_new_cluster.sh’
analyze_cluster
end

def analyze_cluster
user_home = @attributes.dig(:gitlab, :postgresql, :home) || @attributes.dig(:postgresql, :home)
analyze_script = File.join(File.realpath(user_home), ‘analyze_new_cluster.sh’)
begin
@db_worker.run_pg_command("/bin/sh #{analyze_script}")
rescue GitlabCtl::Errors::ExecutionError => e
log ‘Error running analyze_new_cluster.sh’
log “STDOUT: #{e.stdout}”
log “STDERR: #{e.stderr}”
log ‘Please check the output, and rerun the command if needed:’
log “/bin/sh #{analyze_script}”
log 'If the error persists, please open an issue at: ’
log ‘Issues · GitLab.org / omnibus-gitlab · GitLab
end
end

Ideally if something errors out, STDOUT or STDERR should give more info.

unless [:pg_secondary, :geo_secondary].include?(@instance_type)
log ‘Database upgrade is complete, running analyze_new_cluster.sh’
analyze_cluster
end

maintenance_mode(‘disable’) if disable_maintenance
goodbye_message

def goodbye_message
log ‘==== Upgrade has completed ====’

when :pg_secondary
log “As part of PostgreSQL upgrade, this secondary node was removed from”
log “the HA cluster. Once the primary node is upgraded to new version of”
log “PostgreSQL, you will have to configure this secondary node to follow”
log “the primary node again.”
log “Check Database settings | GitLab for details.”

when :geo_primary, :geo_secondary
log ‘As part of the PostgreSQL upgrade, replication between primary and secondary has’
log ‘been shut down. After the secondary has been upgraded, it needs to be re-initialized’
log ‘Please see the instructions at Database settings | GitLab

1 Like

No, I don’t have a high availability setup or GEO secondary.

I found this issue: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2067#note_261123434

Two years ago someone have the same problem, and he changed the analyze_new_cluster.sh
to looking like this "/opt/gitlab/embedded/bin/vacuumdb" -h /var/opt/gitlab/postgresql --all --analyze-in-stages . I’ve tried that, but when the script is executed the value inside changed to the old one :frowning:

I was trying to run the upgrade with the manually upgrade for PostgreSQL.

What I’ve done:

  • wget --content-disposition https://packages.gitlab.com/gitlab/gitlab-ce/packages/el/7/gitlab-ce-12.5.5-ce.0.el7.x86_64.rpm/download.rpm

  • sudo touch /etc/gitlab/disable-postgresql-upgrade

  • rpm -Uvh gitlab-ce-12.5.5-ce.0.el7.x86_64.rpm

  • gitlab-ctl reconfigure

After that my Gitlab version is now in 12.5.5

image

Now when was trying to run the PostgreSQL upgrade manually with gitlab-ctl pg-upgrade command, the process always get stuck on the same step:

Database upgrade is complete, running analyze_new_cluster.sh

The script analyze_new_cluster says to run the "/opt/gitlab/embedded/bin/vacuumdb" --all --analyze-only' manually if necessary, so I was trying that but now I have this error:

-sh-4.2$ "/opt/gitlab/embedded/postgresql/10/bin/vacuumdb" -h /var/opt/gitlab/postgresql --all --analyze-in-stages
vacuumdb: processing database "gitlabhq_production": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "gitlabhq_production": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "gitlabhq_production": Generating default (full) optimizer statistics
vacuumdb: vacuuming of database "gitlabhq_production" failed: ERROR:  canceling statement due to statement timeout