Hello GitLab Experts,

I hope you can help me. I am upgrading GitLab ultimately from 14.0.12 → 14.1.8. I am running GitLab on rhel7 Virutal Machine it is a small gitlab server. /var/gitlab is 37G. The VM has 8 CPUs and 24 GB RAM.

The upgrade fails. The GUI says "500 Whoops, something went wrong on our end. It has been left to finish any background jobs for 2 hours now. In presious upgrades this part, going from 502 to 500 to GUI being available takes about 2 minutes.

I have done the following so far.

1. gitlab-ctl restart
2. gitlab-ctl reconfigure

Above does not change anything.

The main log files appear to be OK.

[root@rhosgitlab2 opt]# gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/state <==

==> /var/log/gitlab/postgresql/current <==
2022-09-15_13:32:30.63914 LOG: listening on Unix socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”
2022-09-15_13:32:30.74374 FATAL: the database system is starting up
2022-09-15_13:32:30.74375 FATAL: the database system is starting up
2022-09-15_13:32:30.74490 LOG: database system was shut down at 2022-09-15 13:32:30 GMT
2022-09-15_13:32:30.74492 FATAL: the database system is starting up
2022-09-15_13:32:30.74589 FATAL: the database system is starting up
2022-09-15_13:32:30.74591 FATAL: the database system is starting up
2022-09-15_13:32:30.74781 FATAL: the database system is starting up
2022-09-15_13:32:30.75295 FATAL: the database system is starting up
2022-09-15_13:32:30.75427 LOG: database system is ready to accept connections

[root@rhosgitlab2 opt]# gitlab-ctl tail nginx
==> /var/log/gitlab/nginx/state <==

==> /var/log/gitlab/nginx/error.log <==

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.2.14.170 - - [15/Sep/2022:14:32:33 +0100] “GET /admin HTTP/2.0” 502 2940 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:33:18 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:33:20 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:35:06 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:36:03 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:42:58 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:43:27 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:44:41 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:49:45 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -
10.2.14.170 - - [15/Sep/2022:14:51:47 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -

==> /var/log/gitlab/nginx/gitlab_error.log <==

==> /var/log/gitlab/nginx/access.log <==

==> /var/log/gitlab/nginx/current <==

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.2.14.170 - - [15/Sep/2022:14:52:46 +0100] “GET /admin HTTP/2.0” 500 2926 “” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0” -

[root@rhosgitlab2 opt]# gitlab-ctl tail puma
==> /var/log/gitlab/puma/state <==

==> /var/log/gitlab/puma/puma_stderr.log <==
unknown OID 194: failed to recognize type of ‘relpartbound’. It will be treated as String.
unknown OID 28: failed to recognize type of ‘relfrozenxid’. It will be treated as String.
unknown OID 1034: failed to recognize type of ‘relacl’. It will be treated as String.
unknown OID 194: failed to recognize type of ‘relpartbound’. It will be treated as String.
unknown OID 28: failed to recognize type of ‘relfrozenxid’. It will be treated as String.
unknown OID 1034: failed to recognize type of ‘relacl’. It will be treated as String.
unknown OID 194: failed to recognize type of ‘relpartbound’. It will be treated as String.
unknown OID 28: failed to recognize type of ‘relfrozenxid’. It will be treated as String.
unknown OID 1034: failed to recognize type of ‘relacl’. It will be treated as String.
unknown OID 194: failed to recognize type of ‘relpartbound’. It will be treated as String.

==> /var/log/gitlab/puma/puma_stdout.log <==
{“timestamp”:“2022-09-15T13:50:17.857Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8239.1171875 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:50:37.860Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8239.5 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:50:57.862Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8239.65234375 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:51:17.865Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8240.0703125 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:51:37.867Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8240.4375 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:51:57.868Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8253.62890625 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:52:17.871Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8253.734375 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:52:37.873Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8253.61328125 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:52:57.875Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8266.765625 mb with master and 8 workers.”}
{“timestamp”:“2022-09-15T13:53:17.877Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8267.62890625 mb with master and 8 workers.”}

==> /var/log/gitlab/puma/current <==
2022-09-15_13:32:32.59870 {“timestamp”:“2022-09-15T13:32:32.598Z”,“pid”:23060,“message”:“* Environment: production”}
2022-09-15_13:32:32.59871 {“timestamp”:“2022-09-15T13:32:32.598Z”,“pid”:23060,“message”:“* Master PID: 23060”}
2022-09-15_13:32:32.59871 {“timestamp”:“2022-09-15T13:32:32.598Z”,“pid”:23060,“message”:“* Workers: 8”}
2022-09-15_13:32:32.59872 {“timestamp”:“2022-09-15T13:32:32.598Z”,“pid”:23060,“message”:“* Restarts: () hot () phased”}
2022-09-15_13:32:32.59872 {“timestamp”:“2022-09-15T13:32:32.598Z”,“pid”:23060,“message”:“* Preloading application”}
2022-09-15_13:33:17.44170 {“timestamp”:“2022-09-15T13:33:17.439Z”,“pid”:23060,“message”:“* Listening on unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket”}
2022-09-15_13:33:17.44171 {“timestamp”:“2022-09-15T13:33:17.440Z”,“pid”:23060,“message”:“* Listening on http://127.0.0.1:8080”}
2022-09-15_13:33:17.44172 {“timestamp”:“2022-09-15T13:33:17.440Z”,“pid”:23060,“message”:“! WARNING: Detected 1 Thread(s) started in app boot:”}
2022-09-15_13:33:17.44172 {“timestamp”:“2022-09-15T13:33:17.440Z”,“pid”:23060,“message”:“! #\u003cThread:0x00007f7177e842d0 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:91:in `sleep’”}
2022-09-15_13:33:17.44172 {“timestamp”:“2022-09-15T13:33:17.440Z”,“pid”:23060,“message”:“Use Ctrl-C to stop”}

==> /var/log/gitlab/puma/puma_stdout.log <==
{“timestamp”:“2022-09-15T13:53:37.880Z”,“pid”:23060,“message”:“PumaWorkerKiller: Consuming 8267.75390625 mb with master and 8 workers.”}

root@rhosgitlab2 opt]# gitlab-ctl tail redis
==> /var/log/gitlab/redis/state <==

==> /var/log/gitlab/redis/current <==
2022-09-15_13:47:35.05730 23066:M 15 Sep 2022 14:47:35.057 * 10 changes in 300 seconds. Saving…
2022-09-15_13:47:35.05974 23066:M 15 Sep 2022 14:47:35.059 * Background saving started by pid 25595
2022-09-15_13:47:35.17469 25595:C 15 Sep 2022 14:47:35.174 * DB saved on disk
2022-09-15_13:47:35.17557 25595:C 15 Sep 2022 14:47:35.175 * RDB: 7 MB of memory used by copy-on-write
2022-09-15_13:47:35.25979 23066:M 15 Sep 2022 14:47:35.259 * Background saving terminated with success
2022-09-15_13:52:36.08872 23066:M 15 Sep 2022 14:52:36.088 * 10 changes in 300 seconds. Saving…
2022-09-15_13:52:36.09055 23066:M 15 Sep 2022 14:52:36.090 * Background saving started by pid 26333
2022-09-15_13:52:36.24830 26333:C 15 Sep 2022 14:52:36.248 * DB saved on disk
2022-09-15_13:52:36.25044 26333:C 15 Sep 2022 14:52:36.250 * RDB: 6 MB of memory used by copy-on-write
2022-09-15_13:52:36.29178 23066:M 15 Sep 2022 14:52:36.291 * Background saving terminated with success

Any help much appreciated. Please let me know if I need to provid any more logs or information.

Best Regards,

Kevin.

Some more information on my set-up as it stands with broken installation. It is strange as this check passes as good but cannot get into GUI.

[root@rhosgitlab2 opt]# gitlab-rake gitlab:check
Checking GitLab subtasks …

Checking GitLab Shell …

GitLab Shell: … GitLab Shell version >= 13.19.1 ? … OK (13.19.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell … Finished

Checking Gitaly …

Gitaly: … default … OK

Checking Gitaly … Finished

Checking Sidekiq …

Sidekiq: … Running? … yes
Number of Sidekiq processes (cluster/worker) … 1/1

Checking Sidekiq … Finished

Checking Incoming Email …

Incoming Email: … Reply by email is disabled in config/gitlab.yml

Checking Incoming Email … Finished

Checking LDAP …

LDAP: … Server: ldapmain
LDAP authentication… Success
LDAP users with access to your GitLab server (only showing the first 100 results)

LDAP checks cut for privacy reasons but all worked.

Checking LDAP … Finished

Checking GitLab App …

Git configured correctly? … yes
Database config exists? … yes
All migrations up? … yes
Database contains orphaned GroupMembers? … no
GitLab config exists? … yes
GitLab config up to date? … yes
Log directory writable? … yes
Tmp directory writable? … yes
Uploads directory exists? … yes
Uploads directory has correct permissions? … yes
Uploads directory tmp has correct permissions? … yes
Init script exists? … skipped (omnibus-gitlab has no init script)
Init script up-to-date? … skipped (omnibus-gitlab has no init script)
Projects have namespace: …

All the projects / repos are listed as good. I cut them for privacy reasons.

Redis version >= 5.0.0? … yes
Ruby version >= 2.7.2 ? … yes (2.7.2)
Git version >= 2.31.0 ? … yes (2.32.0)
Git user has default SSH configuration? … no
Try fixing it:
mkdir ~/gitlab-check-backup-1663255151
sudo mv /var/opt/gitlab/.ssh/id_rsa_kumar.pub ~/gitlab-check-backup-1663255151
sudo mv /var/opt/gitlab/.ssh/id_rsa_kumar ~/gitlab-check-backup-1663255151
For more information see:
doc/ssh/index.md in section “Overriding SSH settings on the GitLab server”
Please fix the error above and rerun the checks.
Active users: … 372
Is authorized keys file accessible? … yes
GitLab configured to store new projects in hashed storage? … yes
All projects are in hashed storage? … yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? … skipped (elasticsearch is disabled)

Checking GitLab App … Finished

Checking GitLab subtasks … Finished

I know sometimes restarting gitlab using gitlab-ctl doesn’t always work. You can also try:

 systemctl restart gitlab-runsvdir

that will completely ensure all gitlab processes are shut down. Either that or restart the server which would do the same thing anyway. Try that, and see if it’s any better. If not, check to see what else might appear in the log files once you’ve done that.

Using tail on it’s own won’t be enough, so you’ll need to go through a lot of the log files under /var/log/gitlab and look for anything that might show up errors, etc.

Hi iwalker,

Thanks for the suggestion. It did not work unfortunately.

Do any of the entries below mean anything to you?

[root@rhosgitlab2 opt]# gitlab-psql
psql (12.7)
Type “help” for help.

gitlabhq_production=# select * from background_migration_jobs;
id | created_at | updated_at | status | class_name | arguments
----±------------------------------±------------------------------±-------±-------------------------------------------------------------------------±-----------------------------------------------------------------------
1 | 2020-12-13 21:19:49.442495+00 | 2020-12-13 21:19:53.65028+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [1, 2574, “audit_events”, “audit_events_part_5fc467ac26”, “id”]
2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 0 | BackfillJiraTrackerDeploymentType2 | [1, 678]
3 | 2021-09-04 20:44:18.585858+00 | 2021-09-04 20:44:26.297106+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [18355, 35331, “web_hook_logs”, “web_hook_logs_part_0c5294f417”, “id”]
5 | 2021-09-04 20:44:34.899439+00 | 2021-09-04 20:48:35.411041+00 | 1 | BackfillNamespaceTraversalIdsRoots | [1, 362, 100]
6 | 2021-09-04 20:44:34.991474+00 | 2021-09-04 20:50:36.30535+00 | 1 | BackfillNamespaceTraversalIdsChildren | [38, 363, 100]
7 | 2022-09-15 13:28:05.092917+00 | 2022-09-15 13:28:05.092917+00 | 0 | BackfillDraftStatusOnMergeRequests | [330, 13471]
8 | 2022-09-15 13:28:06.734538+00 | 2022-09-15 13:31:13.930026+00 | 1 | MigrateMergeRequestDiffCommitUsers | [1, 40001]
9 | 2022-09-15 13:28:06.740887+00 | 2022-09-15 13:32:14.985722+00 | 1 | MigrateMergeRequestDiffCommitUsers | [40001, 80001]
(8 rows)

BackfillDraftStatusOnMergeRequests from today has 0 status.

Best Regards,

Kevin.

I found another thing which might help the debugging…

[root@rhosgitlab2 opt]# gitlab-psql
psql (12.7)
Type “help” for help.

gitlabhq_production=# select * from batched_background_migrations where job_class_name like ‘Copy%’;
id | created_at | updated_at | min_value | max_value | batch_size | sub_batch_size | interval | status | job_class_name | batch_class_name | table_name
| column_name | job_arguments | total_tuple_count | pause_ms
----±------------------------------±------------------------------±----------±----------±-----------±---------------±---------±-------±--------------------------------------±---------------------------±------------------------
--------±------------±-----------------------------------------------------------------------------------±------------------±---------
5 | 2021-09-04 21:37:09.852951+00 | 2021-09-04 21:37:09.852951+00 | 1 | 1 | 20000 | 2000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_job_artifacts
| id | [[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_convert_to_bigint”]] | | 100
6 | 2021-09-04 21:37:09.889908+00 | 2021-09-04 21:37:09.889908+00 | 1 | 1 | 20000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_sources_pipelines
| id | [[“source_job_id”], [“source_job_id_convert_to_bigint”]] | | 100
7 | 2021-09-04 21:37:09.947644+00 | 2021-09-04 21:37:09.947644+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_needs
| id | [[“build_id”], [“build_id_convert_to_bigint”]] | | 100
9 | 2021-09-04 21:37:10.054482+00 | 2021-09-04 21:37:10.054482+00 | 1 | 1 | 20000 | 5000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_runner_session
| id | [[“build_id”], [“build_id_convert_to_bigint”]] | | 100
10 | 2021-09-04 21:37:10.087256+00 | 2021-09-04 21:37:10.087256+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_trace_chunks
| id | [[“build_id”], [“build_id_convert_to_bigint”]] | | 100
12 | 2021-09-04 21:37:10.545477+00 | 2021-09-04 21:37:10.545477+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | deployments
| id | [[“deployable_id”], [“deployable_id_convert_to_bigint”]] | | 100
13 | 2021-09-04 21:37:10.632121+00 | 2021-09-04 21:37:10.632121+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | geo_job_artifact_deleted
_events | id | [[“job_artifact_id”], [“job_artifact_id_convert_to_bigint”]] | | 100
2 | 2021-09-04 21:37:09.468794+00 | 2021-09-04 22:30:07.36095+00 | 1 | 5271 | 15000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_metadata
| id | [[“build_id”], [“build_id_convert_to_bigint”]] | 4395 | 100
3 | 2021-09-04 21:37:09.687101+00 | 2021-09-04 22:40:07.449488+00 | 1 | 70537 | 20000 | 500 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | events
| id | [[“id”], [“id_convert_to_bigint”]] | 69133 | 100
4 | 2021-09-04 21:37:09.793892+00 | 2021-09-04 22:49:35.914658+00 | 1 | 70537 | 20000 | 2500 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | push_event_payloads
| event_id | [[“event_id”], [“event_id_convert_to_bigint”]] | 46075 | 100
8 | 2021-09-04 21:37:10.009046+00 | 2021-09-04 22:53:10.012781+00 | 1 | 17899 | 20000 | 250 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds
| id | [[“id”, “stage_id”], [“id_convert_to_bigint”, “stage_id_convert_to_bigint”]] | 17017 | 100
11 | 2021-09-04 21:37:10.179302+00 | 2021-09-04 22:56:35.9511+00 | 1 | 3 | 15000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | taggings
| id | [[“id”, “taggable_id”], [“id_convert_to_bigint”, “taggable_id_convert_to_bigint”]] | 3 | 100
14 | 2021-09-04 21:37:13.015371+00 | 2021-09-04 22:59:04.912447+00 | 1 | 10365 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_stages
| id | [[“id”], [“id_convert_to_bigint”]] | 9855 | 100
15 | 2022-09-14 12:09:46.457297+00 | 2022-09-14 13:12:29.097867+00 | 1 | 9672 | 15000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_metadata
| id | [[“id”], [“id_convert_to_bigint”]] | 8182 | 100
(14 rows)

gitlabhq_production=#

Use this command to properly identify migration status:

gitlab-rake db:migrate:status

if you see any that haven’t ran, or were problematic, you can then do:

gitlab-rake db:migrate

Thanks for very quick responses.

So when I run this I get the following…

database: gitlabhq_production

Status Migration ID Migration Name

up 20140313092127 ********** NO FILE **********
up 20140319135450 ********** NO FILE **********
up 20140407135544 ********** NO FILE **********

Load of stuff cut

up 20181220163029 ********** NO FILE **********
up 20181220165848 ********** NO FILE **********
up 20181221135205 ********** NO FILE **********
up 20181228140935 ********** NO FILE **********
up 20181228175414 Init schema
up 20190102152410 Delete inconsistent internal id records2
up 20190103140724 Make legacy false default
up 20190104182041 Cleanup legacy artifact migration
up 20190107151020 Add services type index
up 20190107151029 ********** NO FILE **********
up 20190108192941 Remove partial index from ci builds artifacts file
up 20190109153125 Add merge request external diffs
up 20190110200434 ********** NO FILE **********
up 20190111183834 ********** NO FILE **********
up 20190111231855 ********** NO FILE **********
up 20190114040404 ********** NO FILE **********
up 20190114040405 ********** NO FILE **********
up 20190114172110 Add domain to cluster
up 20190115054215 Migrate delete container repository worker
up 20190115054216 Add error notification sent to remote mirrors
up 20190115092821 Add columns project error tracking settings
up 20190116234221 Add sorting fields to user preference
up 20190121140418 ********** NO FILE **********
up 20190121140658 ********** NO FILE **********
up 20190122101816 ********** NO FILE **********
up 20190123211816 ********** NO FILE **********
up 20190124200344 Migrate storage migrator sidekiq queue
up 20190128104236 ********** NO FILE **********
up 20190128172533 ********** NO FILE **********
up 20190129013538 ********** NO FILE **********
up 20190130091630 Add local cached markdown version
up 20190130164903 ********** NO FILE **********
up 20190131122559 Fix null type labels
up 20190204115450 Migrate auto dev ops domain to cluster domain
up 20190206193120 Add index to tags
up 20190211131150 Add state id to issuables
up 20190214112022 Schedule sync issuables state
up 20190215154930 Add merge pipelines enabled to ci cd settings
up 20190218031401 ********** NO FILE **********
up 20190218134158 Add masked to ci variables
up 20190218134209 Add masked to ci group variables
up 20190218144405 ********** NO FILE **********
up 20190219134239 ********** NO FILE **********
up 20190219201635 Add asset proxy settings
up 20190219210244 ********** NO FILE **********
up 20190220112238 ********** NO FILE **********
up 20190220142344 Add email header and footer enabled flag to appearances table
up 20190220150130 Add extra shas to ci pipelines
up 20190222051615 Add indexes for merge request diffs query
up 20190222105948 ********** NO FILE **********
up 20190222110418 ********** NO FILE **********
up 20190225152525 Add auto dev ops enabled to namespaces
up 20190225160300 ********** NO FILE **********
up 20190225160301 Add runner tokens indexes
up 20190225173106 ********** NO FILE **********
up 20190226154144 ********** NO FILE **********
up 20190228134845 ********** NO FILE **********
up 20190228192410 Add multi line attributes to suggestion
up 20190301081611 Migrate project migrate sidekiq queue
up 20190301095211 ********** NO FILE **********
up 20190301182031 ********** NO FILE **********
up 20190301182457 Add external hostname to ingress and knative
up 20190302144241 ********** NO FILE **********
up 20190304020812 ********** NO FILE **********
up 20190304223216 ********** NO FILE **********
up 20190304223220 ********** NO FILE **********
up 20190305162221 ********** NO FILE **********
up 20190311132500 ********** NO FILE **********
up 20190311132527 ********** NO FILE **********
up 20190312071108 Add detected repository languages to projects
up 20190312113229 Add remove at to pages domains
up 20190312113634 Add remove at index to pages domains
up 20190313092516 Clean up noteable id for notes on commits
up 20190315191339 Create merge request assignees table
up 20190318020549 ********** NO FILE **********
up 20190318021429 ********** NO FILE **********
up 20190318120957 ********** NO FILE **********
up 20190320174702 Add lets encrypt notification email to application settings
up 20190321103531 ********** NO FILE **********
up 20190322132835 Schedule populate merge request assignees table
up 20190322145954 ********** NO FILE **********
up 20190322164830 Add auto ssl enabled to pages domain
up 20190325080727 Truncate user fullname
up 20190325105715 Add fields to user preferences
up 20190325111602 Rename v2 root namespaces
up 20190325165127 Add managed to cluster
up 20190326164045 Import common metrics knative
up 20190327085945 ********** NO FILE **********
up 20190327163904 Add notification email to notification settings
up 20190328210840 ********** NO FILE **********
up 20190329085614 Add lets encrypt terms of service accepted to application settings
up 20190401150745 ********** NO FILE **********
up 20190401150746 ********** NO FILE **********
up 20190402111436 ********** NO FILE **********
up 20190402112450 ********** NO FILE **********
up 20190402150158 Backport enterprise schema
up 20190402224749 ********** NO FILE **********
up 20190403161806 Update designs index
up 20190404143330 Add unique constraint to approvals user id and merge request
up 20190404231137 Remove alternate url from geo nodes
up 20190408163745 Prometheus knative05 fix
up 20190409224933 Add name to geo nodes
up 20190410173409 Add name index to geo nodes
up 20190412155659 Add merge request blocks
up 20190412183653 Remove url index from geo nodes
up 20190414185432 Add comment to vulnerability feedback
up 20190415030217 Add variable type to ci variables
up 20190415095825 Add packages size to project statistics
up 20190415172035 Update insights foreign keys

Load of stuff cut.

Does the up status mean they are good? Is the NO FILE entry a concern?

Best Regards,

Kevin.

Hi,

I am doing some more checking. I hope this can give you a full picture of my set-up / problem. I look at some other entries from you to find the commands. Hopefully this helps.

#<Gitlab::Database::BackgroundMigrationJob:0x00007f6f86aafc10
id: 2,
created_at: Sat, 04 Sep 2021 20:26:53.000973000 UTC +00:00,
updated_at: Sat, 04 Sep 2021 20:26:53.000973000 UTC +00:00,
status: “pending”,
class_name: “BackfillJiraTrackerDeploymentType2”,
arguments: [1, 678]>
[root@rhosgitlab2 opt]#

What does above mean? It is Ok?

Best Regards,

Kevin.

Yes, up means that it has finished. Any other status would mean that there is some kind of problem. However, your migrations seem to only show 2019, so there should be migrations for 2020, 2021, 2022. Or did you just not paste them here? As for the no file, I don’t think this is a problem, I also have this from at least 20121220xxx to 20210301xxx. No idea why, assume that the information doesn’t exist in the database related to these migrations. But what caused it, only Gitlab would be able to explain I think.

As for the last post, that seems to show some kind of migration as pending. See my post here of when I had a pending issue, and then the commands I used to fix it: Pending migration for over 24 hours

You’ll use gitlab-psql in this instance, to get to the postgres command line to use those commands. I would however make sure that the background migration shows as up before you do this though. There should be a better way than what my post suggests, but sadly I didn’t get any assistance for a better way to fix it.

Hi iwalker,

I cut a lot of the output from the migrations. So I think they are OK. I will check and post more shortly.

Just to follow up on your Pending Migration for over 24 post. I have followed the command there. I get the results below.

[root@rhosgitlab2 ~]# gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect’
#<Gitlab::Database::BackgroundMigrationJob:0x00007f571fb46310
id: 2,
created_at: Sat, 04 Sep 2021 20:26:53.000973000 UTC +00:00,
updated_at: Sat, 04 Sep 2021 20:26:53.000973000 UTC +00:00,
status: “pending”,
class_name: “BackfillJiraTrackerDeploymentType2”,
arguments: [1, 678]>
[root@rhosgitlab2 ~]# gitlab-rails db:migrate
[root@rhosgitlab2 ~]# gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect’
#<Gitlab::Database::BackgroundMigrationJob:0x00007f1b82dc42a0
id: 2,
created_at: Sat, 04 Sep 2021 20:26:53.000973000 UTC +00:00,
updated_at: Sat, 04 Sep 2021 20:26:53.000973000 UTC +00:00,
status: “pending”,
class_name: “BackfillJiraTrackerDeploymentType2”,
arguments: [1, 678]>
[root@rhosgitlab2 ~]# gitlab-rails c

Ruby: ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
GitLab: 14.1.8-ee (ef8ecf9b858) EE
GitLab Shell: 13.19.1
PostgreSQL: 12.7

Loading production environment (Rails 6.1.3.2)
irb(main):001:0>
irb(main):002:0> quit

gitlab-psql
gitlabhq_production=# select * from background_migration_jobs;
id | created_at | updated_at | status | class_name | arguments
----±------------------------------±------------------------------±-------±-------------------------------------------------------------------------±-----------------------------------------------------------------------
1 | 2020-12-13 21:19:49.442495+00 | 2020-12-13 21:19:53.65028+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [1, 2574, “audit_events”, “audit_events_part_5fc467ac26”, “id”]
2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 0 | BackfillJiraTrackerDeploymentType2 | [1, 678]
3 | 2021-09-04 20:44:18.585858+00 | 2021-09-04 20:44:26.297106+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [18355, 35331, “web_hook_logs”, “web_hook_logs_part_0c5294f417”, “id”]
5 | 2021-09-04 20:44:34.899439+00 | 2021-09-04 20:48:35.411041+00 | 1 | BackfillNamespaceTraversalIdsRoots | [1, 362, 100]
6 | 2021-09-04 20:44:34.991474+00 | 2021-09-04 20:50:36.30535+00 | 1 | BackfillNamespaceTraversalIdsChildren | [38, 363, 100]
7 | 2022-09-15 13:28:05.092917+00 | 2022-09-15 13:28:05.092917+00 | 0 | BackfillDraftStatusOnMergeRequests | [330, 13471]
8 | 2022-09-15 13:28:06.734538+00 | 2022-09-15 13:31:13.930026+00 | 1 | MigrateMergeRequestDiffCommitUsers | [1, 40001]
9 | 2022-09-15 13:28:06.740887+00 | 2022-09-15 13:32:14.985722+00 | 1 | MigrateMergeRequestDiffCommitUsers | [40001, 80001]
(8 rows)

gitlabhq_production=#

So following the solution should id 1, 3, 5, 6, 8 and 9 have status changed from 1 to 0?

1, 3, 5, 6 are old and I don’t see how they could be creating a problem as gitlab worked before this recent upgrade. Mayby just change 8 and 9? which are more recent?

Apologies I am a bit of a beginner when it comes to the inner workings of GitLab.

Best Regards,

Kevin.

Hi iwalker,

I re-ran gitlab-rake db:migrate:status

There is a total of 3,769 entries there are multiple entries for 2020, 2021, 2022. They all have status up. I will not paste them all here as to many entries but below are examples.

up 20140313092127 ********** NO FILE **********
up 20140319135450 ********** NO FILE **********
up 20140407135544 ********** NO FILE **********
up 20140414093351 ********** NO FILE **********
up 20140414131055 ********** NO FILE **********
up 20140415124820 ********** NO FILE **********

Loads of 2014 cut

up 20150108073740 ********** NO FILE **********
up 20150116234544 ********** NO FILE **********
up 20150116234545 ********** NO FILE **********
up 20150125163100 ********** NO FILE **********
up 20150125163158 ********** NO FILE **********
up 20150205211843 ********** NO FILE **********
up 20150206181414 ********** NO FILE **********
up 20150206222854 ********** NO FILE **********

Loads of 2015 cut

up 20151231202530 ********** NO FILE **********
up 20160106162223 ********** NO FILE **********
up 20160106164438 ********** NO FILE **********
up 20160109054846 ********** NO FILE **********
up 20160112174440 ********** NO FILE **********
up 20160113111034 ********** NO FILE **********

Loads of 2016 cut

up 20170104150317 ********** NO FILE **********
up 20170106142508 ********** NO FILE **********
up 20170106172224 ********** NO FILE **********
up 20170106172234 ********** NO FILE **********
up 20170106172235 ********** NO FILE **********
up 20170106172236 ********** NO FILE **********
up 20170106172237 ********** NO FILE **********
up 20170118194941 ********** NO FILE **********

Loads of 2017 cut

up 20180101160629 ********** NO FILE **********
up 20180101160630 ********** NO FILE **********
up 20180102220145 ********** NO FILE **********
up 20180103123548 ********** NO FILE **********
up 20180103234731 ********** NO FILE **********
up 20180104001824 ********** NO FILE **********
up 20180104131052 ********** NO FILE **********
up 20180105212544 ********** NO FILE **********
up 20180105233807 ********** NO FILE **********

Load of 2018 cut

up 20181228175414 Init schema
up 20190102152410 Delete inconsistent internal id records2
up 20190103140724 Make legacy false default
up 20190104182041 Cleanup legacy artifact migration
up 20190107151020 Add services type index
up 20190107151029 ********** NO FILE **********
up 20190108192941 Remove partial index from ci builds artifacts file
up 20190109153125 Add merge request external diffs
up 20190110200434 ********** NO FILE **********
up 20190111183834 ********** NO FILE **********
up 20190111231855 ********** NO FILE **********
up 20190114040404 ********** NO FILE **********
up 20190114040405 ********** NO FILE **********
up 20190114172110 Add domain to cluster
up 20190115054215 Migrate delete container repository worker

Loads of 2019 cut

up 20200102140148 Add expanded environment name to ci build metadata
up 20200102170221 Add storage version index to projects
up 20200103190741 Add column for instance administrators group
up 20200103192859 Add fk for instance administrators group
up 20200103192914 Add index for instance administrators group
up 20200103195205 Add autoclose referenced issues to projects
up 20200104113850 Add forking access level to project feature

Loads of 2020 cut

up 20201230180202 Create onboarding progress
up 20201231133921 Schedule set default iteration cadences
up 20210101110640 Set limit for rate limiting response text
up 20210102164121 Drop temporary index on ci builds
up 20210104163218 Add epic board position index
up 20210105030125 Cleanup projects with bad has external wiki data
up 20210105052034 Rename asset proxy whitelist on application settings
up 20210105052229 Clean up asset proxy whitelist rename on application settings
up 20210105103649 Delete column group id on compliance framework
up 20210105153342 Add entity columns to vulnerability occurrences
up 20210105154321 Add text limit to vulnerability occurrences entity columns

When I look at the full list they all show status up

So I think the the years are OK? What do you think?

Thanks again for your help in this.

Best Regards,

Kevin.

Hi iwalker,

Next questions is how do I map the entries from “gitlab-psql → gitlabhq_production=# select * from background_migration_jobs;” to the list in gitlab-rake db:migrate:status ?

So what do these two entries

8 | 2022-09-15 13:28:06.734538+00 | 2022-09-15 13:31:13.930026+00 | 1 | MigrateMergeRequestDiffCommitUsers | [1, 40001]
9 | 2022-09-15 13:28:06.740887+00 | 2022-09-15 13:32:14.985722+00 | 1 | MigrateMergeRequestDiffCommitUsers

match to in gitlab-rake db:migrate:status

Best Regards,

Kevin.

Apologies I have got the required status in-correct. If following this solution →

I need to change things from status 0 to 1.

gitlab-psql
gitlabhq_production=# select * from background_migration_jobs;
id | created_at | updated_at | status | class_name | arguments
----±------------------------------±------------------------------±-------±-------------------------------------------------------------------------±-----------------------------------------------------------------------
1 | 2020-12-13 21:19:49.442495+00 | 2020-12-13 21:19:53.65028+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [1, 2574, “audit_events”, “audit_events_part_5fc467ac26”, “id”]
2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 0 | BackfillJiraTrackerDeploymentType2 | [1, 678]
3 | 2021-09-04 20:44:18.585858+00 | 2021-09-04 20:44:26.297106+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [18355, 35331, “web_hook_logs”, “web_hook_logs_part_0c5294f417”, “id”]
5 | 2021-09-04 20:44:34.899439+00 | 2021-09-04 20:48:35.411041+00 | 1 | BackfillNamespaceTraversalIdsRoots | [1, 362, 100]
6 | 2021-09-04 20:44:34.991474+00 | 2021-09-04 20:50:36.30535+00 | 1 | BackfillNamespaceTraversalIdsChildren | [38, 363, 100]
7 | 2022-09-15 13:28:05.092917+00 | 2022-09-15 13:28:05.092917+00 | 0 | BackfillDraftStatusOnMergeRequests | [330, 13471]
8 | 2022-09-15 13:28:06.734538+00 | 2022-09-15 13:31:13.930026+00 | 1 | MigrateMergeRequestDiffCommitUsers | [1, 40001]
9 | 2022-09-15 13:28:06.740887+00 | 2022-09-15 13:32:14.985722+00 | 1 | MigrateMergeRequestDiffCommitUsers | [40001, 80001]
(8 rows)

gitlabhq_production=#

So this means

7 | 2022-09-15 13:28:05.092917+00 | 2022-09-15 13:28:05.092917+00 | 0 | BackfillDraftStatusOnMergeRequests | [330, 13471]

and potentially

2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 0 | BackfillJiraTrackerDeploymentType2 | [1, 678]

Need to change from status 0 to status 1.

How do I map these to the big list in gitlab-rake db:migrate:status ? Or does everything map back to there?

Best Regards,

Kevin.

Got the status change wrong. See new entry below…

Status needs to change from 0 to 1…

Hi iwalker,

I think have figured out the mapping…

up 20210526222715 Backfill draft status on merge requests = BackfillDraftStatusOnMergeRequests

up 20201028182809 Backfill jira tracker deployment type2 = BackfillJiraTrackerDeploymentType2

So it looks like they did migrate and just have status problem.

I will try to change it now.

I think this whole problem is because during the cause of this upgrade postgresql went from 12.6 to 12.7.

Best Regards,

Kevin.

So I got up the courage to change the status of back ground migration jobs from 0 to 1 after I could see they had status “up” unfortunately I still have status 500 when I try and login.

Any error logs I can check? Any other ideas?

[root@rhosgitlab2 ~]# gitlab-psql
psql (12.7)
Type “help” for help.

gitlabhq_production=# select * from background_migration_jobs;
id | created_at | updated_at | status | class_name | arguments
----±------------------------------±------------------------------±-------±-------------------------------------------------------------------------±-----------------------------------------------------------------------
1 | 2020-12-13 21:19:49.442495+00 | 2020-12-13 21:19:53.65028+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [1, 2574, “audit_events”, “audit_events_part_5fc467ac26”, “id”]
3 | 2021-09-04 20:44:18.585858+00 | 2021-09-04 20:44:26.297106+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [18355, 35331, “web_hook_logs”, “web_hook_logs_part_0c5294f417”, “id”]
5 | 2021-09-04 20:44:34.899439+00 | 2021-09-04 20:48:35.411041+00 | 1 | BackfillNamespaceTraversalIdsRoots | [1, 362, 100]
6 | 2021-09-04 20:44:34.991474+00 | 2021-09-04 20:50:36.30535+00 | 1 | BackfillNamespaceTraversalIdsChildren | [38, 363, 100]
8 | 2022-09-15 13:28:06.734538+00 | 2022-09-15 13:31:13.930026+00 | 1 | MigrateMergeRequestDiffCommitUsers | [1, 40001]
9 | 2022-09-15 13:28:06.740887+00 | 2022-09-15 13:32:14.985722+00 | 1 | MigrateMergeRequestDiffCommitUsers | [40001, 80001]
7 | 2022-09-15 13:28:05.092917+00 | 2022-09-15 13:28:05.092917+00 | 1 | BackfillDraftStatusOnMergeRequests | [330, 13471]
2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 1 | BackfillJiraTrackerDeploymentType2 | [1, 678]
(8 rows)

gitlabhq_production=#

Hi iwalker,

I think I have found the source of the problem but not the solution yet.

I took at look in /var/log/gitlab/postgresql/current I see the following.

[root@rhosgitlab2 postgresql]# more current
2022-09-20_08:59:41.15626 ERROR: syntax error at or near “gitlabhq_production” at character 1
2022-09-20_08:59:41.15633 STATEMENT: gitlabhq_production=# select * from background_migration_jobs;
2022-09-20_10:18:57.91541 ERROR: syntax error at or near “select” at character 58
2022-09-20_10:18:57.91545 STATEMENT: update background_migration_jobs set status=1 where id=7
2022-09-20_10:18:57.91546 select * from background_migration_jobs;
2022-09-20_10:19:53.65007 WARNING: there is no transaction in progress
2022-09-20_10:20:41.19762 received TERM from runit, sending INT instead to force quit connections
2022-09-20_10:20:41.20191 LOG: received fast shutdown request
2022-09-20_10:20:41.20770 LOG: aborting any active transactions
2022-09-20_10:20:41.20773 FATAL: terminating connection due to administrator command
2022-09-20_10:20:41.20774 FATAL: terminating connection due to administrator command
2022-09-20_10:20:41.20775 FATAL: terminating connection due to administrator command
2022-09-20_10:20:41.20776 FATAL: terminating connection due to administrator command
2022-09-20_10:20:41.20776 FATAL: terminating connection due to administrator command
2022-09-20_10:20:41.20777 FATAL: terminating connection due to administrator command
2022-09-20_10:20:41.21101 LOG: background worker “logical replication launcher” (PID 8876) exited with exit code 1
2022-09-20_10:20:41.21352 LOG: shutting down
2022-09-20_10:20:41.37334 LOG: database system is shut down
2022-09-20_10:20:41.42878 LOG: starting PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2022-09-20_10:20:41.43072 LOG: listening on Unix socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”
2022-09-20_10:20:41.56793 LOG: database system was shut down at 2022-09-20 10:20:41 GMT
2022-09-20_10:20:41.58689 LOG: database system is ready to accept connections
2022-09-20_10:46:17.91170 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2022-09-20_10:46:17.91171 FATAL: Peer authentication failed for user “gitlab”
2022-09-20_10:46:17.91172 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2022-09-20_10:46:18.03482 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2022-09-20_10:46:18.03485 FATAL: Peer authentication failed for user “gitlab”
2022-09-20_10:46:18.03486 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2022-09-20_10:46:18.13149 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2022-09-20_10:46:18.13152 FATAL: Peer authentication failed for user “gitlab”
2022-09-20_10:46:18.13153 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
[root@rhosgitlab2 postgresql]# pwd
/var/log/gitlab/postgresql
[root@rhosgitlab2 postgresql]#

[root@rhosgitlab2 postgresql]# more /var/opt/gitlab/postgresql/data/pg_hba.conf

This file is managed by gitlab-ctl. Manual changes will be
erased! To change the contents below, edit /etc/gitlab/gitlab.rb
and run sudo gitlab-ctl reconfigure.
PostgreSQL Client Authentication Configuration File

Refer to the “Client Authentication” section in the
PostgreSQL documentation for a complete description
of this file. A short synopsis follows.
This file controls: which hosts are allowed to connect, how clients
are authenticated, which PostgreSQL user names they can use, which
databases they can access. Records take one of these forms:
local DATABASE USER METHOD [OPTION]
host DATABASE USER CIDR-ADDRESS METHOD [OPTION]
hostssl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
hostnossl DATABASE USER CIDR-ADDRESS METHOD [OPTION]
(The uppercase items must be replaced by actual values.)
The first field is the connection type: “local” is a Unix-domain socket,
“host” is either a plain or SSL-encrypted TCP/IP socket, “hostssl” is an
SSL-encrypted TCP/IP socket, and “hostnossl” is a plain TCP/IP socket.
DATABASE can be “all”, “sameuser”, “samerole”, a database name, or
a comma-separated list thereof.
USER can be “all”, a user name, a group name prefixed with “+”, or
a comma-separated list thereof. In both the DATABASE and USER fields
you can also write a file name prefixed with “@” to include names from
a separate file.
CIDR-ADDRESS specifies the set of hosts the record matches.
It is made up of an IP address and a CIDR mask that is an integer
(between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that specifies
the number of significant bits in the mask. Alternatively, you can write
an IP address and netmask in separate columns to specify the set of hosts.
METHOD can be “trust”, “reject”, “md5”, “crypt”, “password”, “gss”, “sspi”,
“krb5”, “ident”, “pam” or “ldap”. Note that “password” sends passwords
in clear text; “md5” is preferred since it sends encrypted passwords.
OPTION is the ident map or the name of the PAM service, depending on METHOD.
Database and user names containing spaces, commas, quotes and other special
characters must be quoted. Quoting one of the keywords “all”, “sameuser” or
“samerole” makes the name lose its special character, and just match a
database or username with that name.
This file is read on server startup and when the postmaster receives
a SIGHUP signal. If you edit the file on a running system, you have
to SIGHUP the postmaster for the changes to take effect. You can use
“pg_ctl reload” to do that.
Put your actual configuration here

If you want to allow non-local connections, you need to add more
“host” records. In that case you will also need to make PostgreSQL listen
on a non-local interface via the listen_addresses configuration parameter,
or via the -i or -h command line switches.
TYPE DATABASE USER CIDR-ADDRESS METHOD
“local” is for Unix domain socket connections only
local all all peer map=gitlab

When I look in /etc/passwd I see the users gitlab-www, git, gitlab-redis, gitlab-psql and gitlab-prometheus.

I don’t see gitlab. Does postgresql and gitlab expect the user gitlab to be in /etc/passwd? Or does gitlab user just exist in gitlab / postgresql?

When I look at database section of /etc/gitlab/gitlab.rb I can see it is all commented out.

Enable or disable automatic database migrations
gitlab_rails[‘auto_migrate’] = true
This is advanced feature used by large gitlab deployments where loading
whole RAILS env takes a lot of time.
gitlab_rails[‘rake_cache_clear’] = true
GitLab database settings
###! Docs: Database settings | GitLab
###! Only needed if you use an external database.

gitlab_rails[‘db_adapter’] = “postgresql”
gitlab_rails[‘db_encoding’] = “unicode”
gitlab_rails[‘db_collation’] = nil
gitlab_rails[‘db_database’] = “gitlabhq_production”
gitlab_rails[‘db_pool’] = 10
gitlab_rails[‘db_username’] = “gitlab”
gitlab_rails[‘db_password’] = nil
gitlab_rails[‘db_host’] = nil
gitlab_rails[‘db_port’] = 5432
gitlab_rails[‘db_socket’] = nil
gitlab_rails[‘db_sslmode’] = nil
gitlab_rails[‘db_sslcompression’] = 0
gitlab_rails[‘db_sslrootcert’] = nil
gitlab_rails[‘db_prepared_statements’] = false
gitlab_rails[‘db_statements_limit’] = 1000

When I look in postgresql I can see the username gitlab in the database.

gitlab-psql
psql (12.7)
Type “help” for help.

gitlabhq_production=# help
You are using psql, the command-line interface to PostgreSQL.
Type: \copyright for distribution terms
\h for help with SQL commands
? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
gitlabhq_production=# \du
List of roles
Role name | Attributes | Member of
-------------------±-----------------------------------------------------------±----------
gitlab | | {}
gitlab-psql | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
gitlab_replicator | Replication | {}

gitlabhq_production=#

I wonder is this causing problems?

Any help / advice much appreciated.

Best Regards,

Kevin.

The commented out config is not the problem, or is pg_hba.conf because it’s the same as mine. The problems are your migrations are not marked as finished. If your database shows them as pending so similar in what I posted before, then your upgrade is not going to work.

In my post I said changing status from 0 to 1. Once you change that, that is all you need to do. You do not need to map back to anything else. Updating that status should be enough.

Obviously make sure you have a backup just in case you need to revert it. Whilst it worked for me, and I have been able to upgrade many versions since, that’s not necessarily meaning that you will not have other problems. You might or might not. It’s best only to change one thing and see what happens. If you start changing too much data in the database tables, then we will never know what is wrong with your system.

For now, only change status in background_migration_jobs from 0 to 1.

HI iwalker,

I did change the status from 0 to 1. See below.

[root@rhosgitlab2 postgresql]# gitlab-psql
psql (12.7)
Type “help” for help.

gitlabhq_production=# select * from background_migration_jobs;
id | created_at | updated_at | status | class_name | arguments
----±------------------------------±------------------------------±-------±-------------------------------------------------------------------------±-----------------------------------------------------------------------
1 | 2020-12-13 21:19:49.442495+00 | 2020-12-13 21:19:53.65028+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [1, 2574, “audit_events”, “audit_events_part_5fc467ac26”, “id”]
3 | 2021-09-04 20:44:18.585858+00 | 2021-09-04 20:44:26.297106+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::BackfillPartitionedTable | [18355, 35331, “web_hook_logs”, “web_hook_logs_part_0c5294f417”, “id”]
5 | 2021-09-04 20:44:34.899439+00 | 2021-09-04 20:48:35.411041+00 | 1 | BackfillNamespaceTraversalIdsRoots | [1, 362, 100]
6 | 2021-09-04 20:44:34.991474+00 | 2021-09-04 20:50:36.30535+00 | 1 | BackfillNamespaceTraversalIdsChildren | [38, 363, 100]
8 | 2022-09-15 13:28:06.734538+00 | 2022-09-15 13:31:13.930026+00 | 1 | MigrateMergeRequestDiffCommitUsers | [1, 40001]
9 | 2022-09-15 13:28:06.740887+00 | 2022-09-15 13:32:14.985722+00 | 1 | MigrateMergeRequestDiffCommitUsers | [40001, 80001]
7 | 2022-09-15 13:28:05.092917+00 | 2022-09-15 13:28:05.092917+00 | 1 | BackfillDraftStatusOnMergeRequests | [330, 13471]
2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 1 | BackfillJiraTrackerDeploymentType2 | [1, 678]
(8 rows)

gitlabhq_production=#

All status are 1 but unfortunately I still the the same problems.

I tried a git clone from CLI and it worked!!! This is good news and points to the problem being just with the GUI?

Any help/suggestions much appreciated.

Are there any log files I can provide to help debugging?

Best Regards,

Kevin.

Hi iwalker,

I have reverted to working GitLab 14.0.12. I did this using VMware Snaps. All is working fine.

I had tried to upgrade from 14.0.12 to 14.1.8 but it did not work. This is discussed in detail above. I will try upgrading to 14.1.7 next. I need to get to 15.x eventually but intend to take baby steps.

I am doing some pre-upgrade checks…

[root@rhosgitlab2 kevin]# gitlab-psql
psql (12.7)
Type “help” for help.

gitlabhq_production=# select * from background_migration_jobs;
id | created_at | updated_at | status | class_name
| arguments
----±------------------------------±------------------------------±-------±--------------------------------------------------------
-----------------±-----------------------------------------------------------------------
1 | 2020-12-13 21:19:49.442495+00 | 2020-12-13 21:19:53.65028+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::Backfill
PartitionedTable | [1, 2574, “audit_events”, “audit_events_part_5fc467ac26”, “id”]
2 | 2021-09-04 20:26:53.000973+00 | 2021-09-04 20:26:53.000973+00 | 0 | BackfillJiraTrackerDeploymentType2
| [1, 678]
3 | 2021-09-04 20:44:18.585858+00 | 2021-09-04 20:44:26.297106+00 | 1 | Gitlab::Database::PartitioningMigrationHelpers::Backfill
PartitionedTable | [18355, 35331, “web_hook_logs”, “web_hook_logs_part_0c5294f417”, “id”]
5 | 2021-09-04 20:44:34.899439+00 | 2021-09-04 20:48:35.411041+00 | 1 | BackfillNamespaceTraversalIdsRoots
| [1, 362, 100]
6 | 2021-09-04 20:44:34.991474+00 | 2021-09-04 20:50:36.30535+00 | 1 | BackfillNamespaceTraversalIdsChildren
| [38, 363, 100]
(5 rows)

I have these versions now working.

GitLab 14.0.12-ee
GitLab Shell 13.19.1
GitLab Workhorse v14.0.12
GitLab API v4
Ruby 2.7.2p137
Rails 6.1.3.2
PostgreSQL 12.7
Redis 6.0.14

I will update this conversation with results of 14.1.7 upgrade shortly.

Best Regards,

Kevin.