Upgrade Gitlab CE from 11.9.1 to 12.5

winnertako · December 12, 2019, 4:00pm

Hello,

I want to upgrade my gitlab version from 11.9.1 to 12.5. The gitlab are installed on centos7 server .

So if I understand, I have to update from 11.9.1 -> 11.11.8 -> 12.0.9 -> 12.5, right?

I’ve tried that but when I’m trying to upgrade from 11.11.8 to 12.0.9 (yum install gitlab-ce-12.0.9-ce.0.el7.x86_64) but my upgrade is stuck here :
Updating : gitlab-ce-12.0.9-ce.0.el7.x86_64 1/2
Cleanup : gitlab-ce-11.11.8-ce.0.el7.x86_64 2/2

My production log doesn’t show any error:
tailf /var/log/gitlab/gitlab-rails/production.log Migrating to SetResolvedStateOnVulnerabilities (20191114173624) Migrating to CreateVulnerabilityIssueLinks (20191115091425) Migrating to AddEncryptedFieldsToApplicationSettings (20191120084627) Migrating to EncryptPlaintextAttributesOnApplicationSettings (20191120115530) Migrating to RemovePlaintextColumnsFromApplicationSettings (20191122135327) Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects. Creating scope :join_project. Overwriting existing method MergeRequest.join_project. Creating scope :references_project. Overwriting existing method MergeRequest.references_project. Creating scope :system. Overwriting existing method Note.system.

What I’m missing ?

sirmuttonchop · December 12, 2019, 7:06pm

You can actually skip 12.0.9.

11.9.1 → 11.11.8 → 12.5

If a major version like 11.11 is skipped and you go from 11.9.x to 12.x, and it can’t proceed further, it will actually say that in the log.

Example

Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
gitlab preinstall: It seems you are upgrading from major version 11 to major version 12.
gitlab preinstall: It is required to upgrade to the latest 11.11 version first before jumping to the next major version.
gitlab preinstall: Please follow the upgrade documentation at GitLab release and maintenance policy | GitLab
error: %pre(gitlab-ce-12.4.2-ce.0.el7.x86_64) scriptlet failed, exit status 1

Try using wget to get the .rpm first. Then do a

yum localinstall <$rpm pkg name>

Otherwise may be something else that’s failing.

winnertako · December 13, 2019, 9:12am

Thank you. I will try that now, but it seems I have the same result:

Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2019-12-13 09:31:38 +0000 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2019-12-13 09:37:37 +0000 -- done
2019-12-13 09:37:37 +0000 -- Dumping repositories ...
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping uploads ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping builds ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping artifacts ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping pages ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping lfs objects ... 
2019-12-13 09:37:37 +0000 -- [SKIPPED]
2019-12-13 09:37:37 +0000 -- Dumping container registry images ... 
2019-12-13 09:37:37 +0000 -- [DISABLED]
Creating backup archive: 1576229857_2019_12_13_11.11.8_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
Deleting old backups ... done. (0 removed)
  Updating   : gitlab-ce-12.5.4-ce.0.el7.x86_64                                                                                                                                  1/2 
  Cleanup    : gitlab-ce-11.11.8-ce.0.el7.x86_64                                                                                                                                 2/2

It is normal this step takes a very long time?

sirmuttonchop · December 13, 2019, 3:00pm

For the upgrade I believe it takes a default SQL backup, so the time depends on the the size of your postgresDB. The default one skips other things like wikis and container…etc.

My instance isn’t too big, for PROD it’s something like 10 minutes. 500 users / 1000 projects / we try to limit binary getting checked in. Your miles may vary.

As far as log excerpts. After backup is done it should move onto something like this:

Updating : gitlab-ce-12.4.3-ce.0.el7.x86_64 1/2
Cleanup : gitlab-ce-11.11.8-ce.0.el7.x86_64
* file[/opt/gitlab/sv/gitlab-exporter/down] action delete (up to date)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/gitlab-exporter] action create (up to date)
* ruby_block[wait for gitlab-exporter service socket] action run (skipped due to not_if)
(up to date)
Recipe: monitoring::redis-exporter

directory[/var/log/gitlab/redis-exporter] action create (up to date)

directory[/opt/gitlab/etc/redis-exporter/env] action create (up to date)

env_dir[/opt/gitlab/etc/redis-exporter/env] action create

directory[/opt/gitlab/etc/redis-exporter/env] action create (up to date)

file[/opt/gitlab/etc/redis-exporter/env/SSL_CERT_DIR] action create (up to date)
(up to date)

Hopefully the upgrade kept going. Did it finish on any errors?

Other logs should be under /var/log/gitlab and can also tail /var/log/messages.

winnertako · December 16, 2019, 10:25am

Hello,

I’ve about 600 users / 2000 projects. With the yum localinstall gitlab-ce-12.5.4-ce.0.el7.x86_64.rpm the installation stops in the same place:
Cleanup : gitlab-ce-11.11.8-ce.0.el7.x86_64 2/2.

Now I’m trying to install with the rpm -ivh gitlab-ce-12.5.4-ce.0.el7.x86_64.rpm and it seems the installation is running, on my output installation I’ve this output:

Running handlers:
Running handlers complete
Chef Client finished, 123/927 resources updated in 07 minutes 05 seconds
Running reconfigure: OK
Waiting for Database to be running.
Database upgrade is complete, running analyze_new_cluster.sh

On my /var/log/gitlab/postgresql/current

2019-12-16_09:45:21.86052 LOG:  database system is ready to accept connections
2019-12-16_09:45:25.52066 received TERM from runit, sending INT instead to force quit connections
2019-12-16_09:45:25.52322 LOG:  received fast shutdown request
2019-12-16_09:45:25.52740 LOG:  aborting any active transactions
2019-12-16_09:45:25.52876 LOG:  worker process: logical replication launcher (PID 13354) exited with exit code 1
2019-12-16_09:45:25.52931 LOG:  shutting down
2019-12-16_09:45:25.60799 LOG:  database system is shut down
2019-12-16_09:45:25.71113 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2019-12-16_09:45:25.83508 LOG:  database system was shut down at 2019-12-16 09:45:25 GMT
2019-12-16_09:45:25.84647 LOG:  database system is ready to accept connections

On my /var/log/gitlab/gitlab-rails/production.log

Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :system. Overwriting existing method         Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Started GET "/" for  at 2019-12-16 09:52:57 +0000
Processing by RootController#index as HTML
Redirected to http://example.org/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 81ms (ActiveRecord: 25.5ms | Elasticsearch: 0.0ms)

After 2 hours the outupt on my log are the same…

sirmuttonchop · December 16, 2019, 7:46pm

The hanging seems to be near the analyze_new_cluster.sh part. Do you have a high availability setup or GEO secondary? If yes, I think there’s steps on that with

or

…
Digging around the upgrade step…

https://github.com/gitlabhq/omnibus-gitlab/blob/master/files/gitlab-ctl-commands/pg-upgrade.rb

unless [:pg_secondary, :geo_secondary].include?(@instance_type)
log ‘Database upgrade is complete, running analyze_new_cluster.sh’
analyze_cluster
end

…

def analyze_cluster
user_home = @attributes.dig(:gitlab, :postgresql, :home) || @attributes.dig(:postgresql, :home)
analyze_script = File.join(File.realpath(user_home), ‘analyze_new_cluster.sh’)
begin
@db_worker.run_pg_command(“/bin/sh #{analyze_script}”)
rescue GitlabCtl::Errors::ExecutionError => e
log ‘Error running analyze_new_cluster.sh’
log “STDOUT: #{e.stdout}”
log “STDERR: #{e.stderr}”
log ‘Please check the output, and rerun the command if needed:’
log “/bin/sh #{analyze_script}”
log 'If the error persists, please open an issue at: ’
log ‘Issues · GitLab.org / omnibus-gitlab · GitLab’
end
end

Ideally if something errors out, STDOUT or STDERR should give more info.

unless [:pg_secondary, :geo_secondary].include?(@instance_type)
log ‘Database upgrade is complete, running analyze_new_cluster.sh’
analyze_cluster
end

maintenance_mode(‘disable’) if disable_maintenance
goodbye_message
…
def goodbye_message
log ‘==== Upgrade has completed ====’
…
when :pg_secondary
log “As part of PostgreSQL upgrade, this secondary node was removed from”
log “the HA cluster. Once the primary node is upgraded to new version of”
log “PostgreSQL, you will have to configure this secondary node to follow”
log “the primary node again.”
log “Check Database settings | GitLab for details.”
…
when :geo_primary, :geo_secondary
log ‘As part of the PostgreSQL upgrade, replication between primary and secondary has’
log ‘been shut down. After the secondary has been upgraded, it needs to be re-initialized’
log ‘Please see the instructions at Database settings | GitLab’

winnertako · December 17, 2019, 12:45pm

No, I don’t have a high availability setup or GEO secondary.

I found this issue: gitlab-ctl pg-upgrade stuck (#2067) · Issues · GitLab.org / omnibus-gitlab · GitLab

Two years ago someone have the same problem, and he changed the analyze_new_cluster.sh
to looking like this "/opt/gitlab/embedded/bin/vacuumdb" -h /var/opt/gitlab/postgresql --all --analyze-in-stages . I’ve tried that, but when the script is executed the value inside changed to the old one

winnertako · December 18, 2019, 9:08pm

I was trying to run the upgrade with the manually upgrade for PostgreSQL.

What I’ve done:

wget --content-disposition https://packages.gitlab.com/gitlab/gitlab-ce/packages/el/7/gitlab-ce-12.5.5-ce.0.el7.x86_64.rpm/download.rpm
sudo touch /etc/gitlab/disable-postgresql-upgrade
rpm -Uvh gitlab-ce-12.5.5-ce.0.el7.x86_64.rpm
gitlab-ctl reconfigure

After that my Gitlab version is now in 12.5.5

Now when was trying to run the PostgreSQL upgrade manually with gitlab-ctl pg-upgrade command, the process always get stuck on the same step:

Database upgrade is complete, running analyze_new_cluster.sh

The script analyze_new_cluster says to run the "/opt/gitlab/embedded/bin/vacuumdb" --all --analyze-only' manually if necessary, so I was trying that but now I have this error:

-sh-4.2$ "/opt/gitlab/embedded/postgresql/10/bin/vacuumdb" -h /var/opt/gitlab/postgresql --all --analyze-in-stages
vacuumdb: processing database "gitlabhq_production": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "gitlabhq_production": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "gitlabhq_production": Generating default (full) optimizer statistics
vacuumdb: vacuuming of database "gitlabhq_production" failed: ERROR:  canceling statement due to statement timeout

rajdeepbhattacharya · June 21, 2023, 3:02am

But is this step a blocker ?

running analyze_new_cluster.sh

? Does gitlab function as usual even after this step fails ?

Topic		Replies	Views
Can't upgrade from CE v11.10.4 (62c464651d2) to v12, and can't find latest v11 version How to Use GitLab upgrade	3	2453	July 19, 2019
Migrate 8.17.6 to 11.5.0 Migrations upgrade	4	1852	December 7, 2018
Git Upgrade from 11.9.4 to 12+ How to Use GitLab upgrade	2	1412	August 23, 2019
Upgrade Gitlab CE from 11.8 to 11.9? Upgrade	1	817	October 31, 2019
Upgrade system while keeping same version of gitlab-CE How to Use GitLab upgrade	2	461	April 10, 2019

Upgrade Gitlab CE from 11.9.1 to 12.5

Related topics