Can't upgrade to version higher than 14.02 due to migration

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,’[[“event_id”], [“event_id_convert_to_bigint”]]’]
rake aborted!
Don’t know how to build task ‘gitlab:background_migrations:finalize’ (See the list of available tasks with rake --tasks)
Did you mean? gitlab:packages:migrate
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
(See full trace by running task with --trace)

for understanding

another link to the community solution topic - url forum

Check the status using:

gitlab-rake db:migrate:status

You can then force running them if they haven’t completed - eg: status is not up by doing this:

gitlab-rake db:migrate

also, please do not post screenshots of console output, copy and paste the text instead. That black screenshot you posted with red text is impossible to read because it’s way too small - and nobody likes reading screenshots. Especially since, when you post it as text, we can copy fragments of it and google search. Doing it the way you did makes it impossible for us to help, even more so when we cannot even read the screenshot in the first place.

It also helps to use the formatting tools available when you are creating your posts, just like I did in this one with the commands I posted.

1 Like

iwalker, I already did it, read on the official website about the update. It does not help. Above there are links to solutions to problems, but I don’t understand how to do it on my server. And also above, I sent how I try to forcefully terminate one of the processes, but this does not help and I get an error. As for the screenshot, at the moment there is no way to send a log. I did it at my workplace, now I’m at home. The screenshot remains in the phone’s memory. Sorry for the screenshot, I delete it.

OK, tomorrow run the migrate status command that I posted, and then copy and paste the output here, so we can see what the status is. That way we can decide what needs to be done next.

up 20210525100603 Backfill geo job artifact deleted events for bigint co nversion
up 20210525184900 Add latest pipeline id into vulnerability statistics t able
up 20210526135911 Create ci minutes additional packs
up 20210526155257 Rename sync security report approval rules sidekiq que ue
up 20210526160133 Remove segment selections table
up 20210526181820 Add index to vulnerability statistics on latest pipeli ne
up 20210526181821 Add foreign key for latest pipeline id to ci pipelines
up 20210526190259 Add ci daily pipeline schedule triggers to plan limits
up 20210526190553 Insert ci daily pipeline schedule triggers plan limits
up 20210526222715 Backfill draft status on merge requests
up 20210527065005 Add index for cadence iterations automation
up 20210527130524 Rename experiment subjects group id to namespace
up 20210527131039 Clean up rename experiment subjects group id to namesp ace
up 20210527133919 Add diff max lines to application settings
up 20210527134019 Add diff max files to application settings
up 20210527185542 Add prevent sharing groups outside hierarchy to namesp ace settings
up 20210527194558 Create ci job token project scope links
up 20210529164247 Change iterations title uniqueness index
up 20210531053916 Rename instance statistics measurements
up 20210531054108 Finalize rename instance statistics measurements
up 20210531070452 Default enforce ssh key expiration
up 20210531071107 Enable enforce ssh key expiration
up 20210601073400 Fix total stage in vsa
up 20210601080039 Group protected environments add index and constraint
up 20210601123341 Add running builds table
up 20210601125410 Add runners created at index
up 20210601131742 Update web hook calls limit
up 20210601133459 Replace runners contacted at index
up 20210602122213 Add upcoming reconciliations
up 20210602122233 Add runners description index
up 20210603140302 Add pronouns to user details
up 20210603222333 Remove builds email service from services
up 20210604070207 Retry backfill traversal ids
up 20210604082145 Create external status checks table
up 20210604085600 Rename status check responses approval rule
up 20210606143426 Add index for container registry access level
up 20210607050531 Add text limit to user details pronouns
up 20210607080044 Remove temporary index on security findings scan
up 20210607154719 Add job token scope enabled to ci cd settings
up 20210608072312 Initialize conversion of ci stages to bigint
up 20210608072346 Backfill ci stages for bigint conversion
up 20210608103230 Add issue id to test report
up 20210608103235 Add issue index to test report
up 20210608110752 Change column null test report requirement
up 20210608110760 Add requirement test reports foreign key
up 20210609013512 Add unique index for batched background migrations
up 20210609090856 Add expiry id ssh key notification index
up 20210609125005 Drop non partitioned web hook logs
up 20210609192728 Add status check foreign key to external status check
up 20210609193101 Add status check foreign key to protected branch
up 20210610102410 Add protected attribute to pending builds
up 20210610102413 Migrate protected attribute to pending builds
up 20210610113229 Add index to protected pending builds
up 20210610141711 Disable expiration policies linked to no container ima ges
up 20210611080951 Fix missing traversal ids
up 20210611100359 Rebuild index for cadence iterations automation
up 20210614131002 Add detection method to vulnerabilities finding
up 20210614143954 Add unique index for helm packages

root@gitlab:/home/administrator# gitlab-rake db:migrate
root@gitlab:/home/administrator#

All logs do not fit here. But they all get status up

Please run this command:

gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect'

you will need to scroll to the right to get the entire command.

administrator@gitlab:~$ sudo gitlab-rails runner -e production ‘puts Gitlab::Database::BackgroundMigrationJob.pending[0].pretty_inspect’
nil

Seems like you have issues like this post: Fail upgrade from 14.0.12 to 14.8.2 - #6 by stevets

Try this:

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']

once you’ve done that, do:

gitlab-ctl reconfigure
gitlab-ctl restart


I have done it already. It didn’t help and found the solution above, even in the answer to solve this problem, but don’t know how to implement it. As I understand it, the address is incorrect. It connects not to the local server, but to gitlab.com.

You need to fix the problems with the CopyColumnUsingBackgroundMigrationJob. The output from your console command shows that it didn’t even run the commands properly. Which means those fixes have not been applied.

And what should be done in this case?

UPD


I’ve reached my message limit. Only 10 messages a day are given to newcomers. Which is very difficult in my case.

One more thing, let’s check the postgres database:

gitlab-psql
select * from background_migration_jobs;

you will get some output, what we are more interested in, are if any of these gitlab background migrations, have a 0 in the status column.

An example, you can find here for my post: Pending migration for over 24 hours - #2 by iwalker

this will show how my query results looked when I had a background migration that hadn’t finished. You will also see, that I posted a post below that one, showing how to change the status from 0 to 1 - but only do this if you have those kind of results.

Hi,

I know you can’t reply, so perhaps wait until tomorrow, but check this command:

gitlab-psql
select * from batched_background_migrations where job_class_name like 'Copy%';

I would like to compare this with my results table. Here is mine:

gitlabhq_production=# select * from batched_background_migrations where job_class_name like 'Copy%';
 id |          created_at           |          updated_at           | min_value | max_value | batch_size | sub_batch_size | interval | status |            job_class_name             |      batch_class_name      |           table_name     
       | column_name |                                   job_arguments                                    | total_tuple_count | pause_ms | max_batch_size 
----+-------------------------------+-------------------------------+-----------+-----------+------------+----------------+----------+--------+---------------------------------------+----------------------------+--------------------------
-------+-------------+------------------------------------------------------------------------------------+-------------------+----------+----------------
  5 | 2021-06-22 14:24:51.849768+00 | 2021-06-22 14:24:51.849768+00 |         1 |         1 |      20000 |           2000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_job_artifacts         
       | id          | [["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]           |                   |      100 |               
  6 | 2021-06-22 14:24:51.881563+00 | 2021-06-22 14:24:51.881563+00 |         1 |         1 |      20000 |            100 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_sources_pipelines     
       | id          | [["source_job_id"], ["source_job_id_convert_to_bigint"]]                           |                   |      100 |               
  7 | 2021-06-22 14:24:51.90664+00  | 2021-06-22 14:24:51.90664+00  |         1 |         1 |      20000 |           1000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_needs           
       | id          | [["build_id"], ["build_id_convert_to_bigint"]]                                     |                   |      100 |               
  9 | 2021-06-22 14:24:51.985026+00 | 2021-06-22 14:24:51.985026+00 |         1 |         1 |      20000 |           5000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_runner_session 
       | id          | [["build_id"], ["build_id_convert_to_bigint"]]                                     |                   |      100 |               
 10 | 2021-06-22 14:24:52.006145+00 | 2021-06-22 14:24:52.006145+00 |         1 |         1 |      20000 |           1000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_trace_chunks    
       | id          | [["build_id"], ["build_id_convert_to_bigint"]]                                     |                   |      100 |               
 11 | 2021-06-22 14:24:52.259621+00 | 2021-06-22 14:24:52.259621+00 |         1 |         1 |      15000 |            100 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | taggings                 
       | id          | [["id", "taggable_id"], ["id_convert_to_bigint", "taggable_id_convert_to_bigint"]] |                   |      100 |               
 12 | 2021-06-22 14:24:52.481622+00 | 2021-06-22 14:24:52.481622+00 |         1 |         1 |      20000 |           1000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | deployments              
       | id          | [["deployable_id"], ["deployable_id_convert_to_bigint"]]                           |                   |      100 |               
 13 | 2021-06-22 14:24:52.630802+00 | 2021-06-22 14:24:52.630802+00 |         1 |         1 |      20000 |           1000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | geo_job_artifact_deleted_
events | id          | [["job_artifact_id"], ["job_artifact_id_convert_to_bigint"]]                       |                   |      100 |               
  2 | 2021-06-22 14:24:51.703593+00 | 2021-07-07 10:19:18.890575+00 |         1 |        25 |      15000 |            100 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_metadata       
       | id          | [["build_id"], ["build_id_convert_to_bigint"]]                                     |                25 |      100 |               
  3 | 2021-06-22 14:24:51.784371+00 | 2021-07-07 10:22:20.978587+00 |         1 |      9958 |      20000 |            500 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | events                   
       | id          | [["id"], ["id_convert_to_bigint"]]                                                 |              7698 |      100 |               
  4 | 2021-06-22 14:24:51.816388+00 | 2021-07-07 10:26:18.388222+00 |         1 |      9958 |      20000 |           2500 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | push_event_payloads      
       | event_id    | [["event_id"], ["event_id_convert_to_bigint"]]                                     |              4873 |      100 |               
  8 | 2021-06-22 14:24:51.95206+00  | 2021-07-07 10:30:18.897485+00 |         1 |        25 |      20000 |            250 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds                
       | id          | [["id", "stage_id"], ["id_convert_to_bigint", "stage_id_convert_to_bigint"]]       |                25 |      100 |               
 14 | 2021-06-22 14:24:55.782762+00 | 2021-07-07 10:33:08.329385+00 |         1 |         6 |      20000 |           1000 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_stages                
       | id          | [["id"], ["id_convert_to_bigint"]]                                                 |                 6 |      100 |               
 15 | 2021-07-09 07:27:39.16178+00  | 2021-07-09 07:31:12.546168+00 |         1 |        25 |      15000 |            100 |      120 |      3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_metadata       
       | id          | [["id"], ["id_convert_to_bigint"]]                                                 |                25 |      100 |               
(14 rows)


gitlabhq_production=# select * from batched_background_migrations;
id | created_at | updated_at | min_value | max_value | batch_size | sub_batch_size | interval | status | job_class_name | batch_class_name | table_name | column_name | job_arguments
| total_tuple_count | pause_ms
----±------------------------------±------------------------------±----------±----------±-----------±---------------±---------±-------±--------------------------------------±---------------------------±--------------------------------±------------±--------------------------------------------------------
---------------------------±------------------±---------
1 | 2021-07-05 12:26:16.396198+00 | 2021-07-05 12:26:16.396198+00 | 1 | 1 | 15000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_trace_sections | build_id | [[“build_id”], [“build_id_convert_to_bigint”]]
| | 100
2 | 2021-07-05 12:26:16.662849+00 | 2021-07-05 12:26:16.668397+00 | 1 | 130 | 15000 | 100 | 120 | 1 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_metadata | id | [[“build_id”], [“build_id_convert_to_bigint”]]
| 84 | 100
3 | 2021-07-05 12:26:16.680868+00 | 2021-07-05 12:26:16.686324+00 | 1 | 2820 | 20000 | 500 | 120 | 1 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | events | id | [[“id”], [“id_convert_to_bigint”]]
| 2762 | 100
4 | 2021-07-05 12:26:16.691681+00 | 2021-07-05 12:26:16.696718+00 | 1 | 2820 | 20000 | 2500 | 120 | 1 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | push_event_payloads | event_id | [[“event_id”], [“event_id_convert_to_bigint”]]
| 1257 | 100
5 | 2021-07-05 12:26:16.704958+00 | 2021-07-05 12:26:16.704958+00 | 1 | 1 | 20000 | 2000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_job_artifacts | id | [[“id”, “job_id”], [“id_convert_to_bigint”, “job_id_conv
ert_to_bigint”]] | | 100
6 | 2021-07-05 12:26:16.715022+00 | 2021-07-05 12:26:16.715022+00 | 1 | 1 | 20000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_sources_pipelines | id | [[“source_job_id”], [“source_job_id_convert_to_bigint”]]
| | 100
7 | 2021-07-05 12:26:16.725961+00 | 2021-07-05 12:26:16.725961+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_needs | id | [[“build_id”], [“build_id_convert_to_bigint”]]
| | 100
8 | 2021-07-05 12:26:16.746143+00 | 2021-07-05 12:26:16.751726+00 | 1 | 249 | 20000 | 250 | 120 | 1 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds | id | [[“id”, “stage_id”], [“id_convert_to_bigint”, “stage_id_
convert_to_bigint”]] | 216 | 100
9 | 2021-07-05 12:26:16.759154+00 | 2021-07-05 12:26:16.759154+00 | 1 | 1 | 20000 | 5000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_builds_runner_session | id | [[“build_id”], [“build_id_convert_to_bigint”]]
| | 100
10 | 2021-07-05 12:26:16.770675+00 | 2021-07-05 12:26:16.770675+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_build_trace_chunks | id | [[“build_id”], [“build_id_convert_to_bigint”]]
| | 100
11 | 2021-07-05 12:26:16.804698+00 | 2021-07-05 12:26:16.804698+00 | 1 | 1 | 15000 | 100 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | taggings | id | [[“id”, “taggable_id”], [“id_convert_to_bigint”, “taggab
le_id_convert_to_bigint”]] | | 100
12 | 2021-07-05 12:26:16.845565+00 | 2021-07-05 12:26:16.845565+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | deployments | id | [[“deployable_id”], [“deployable_id_convert_to_bigint”]]
| | 100
13 | 2021-07-05 12:26:16.868583+00 | 2021-07-05 12:26:16.868583+00 | 1 | 1 | 20000 | 1000 | 120 | 3 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | geo_job_artifact_deleted_events | id | [[“job_artifact_id”], [“job_artifact_id_convert_to_bigin
t”]] | | 100
14 | 2021-07-05 12:26:17.731005+00 | 2021-07-05 12:26:17.737291+00 | 1 | 124 | 20000 | 1000 | 120 | 1 | CopyColumnUsingBackgroundMigrationJob | PrimaryKeyBatchingStrategy | ci_stages | id | [[“id”], [“id_convert_to_bigint”]]
| 109 | 100
(14 rows)

It is unlikely that the problem in my actions was updated six months ago from off source. I have attached the solution above. But I don’t understand how to implement the current version inside.

Right, so you can see that certain columns from your output have a status of 1, so what you are going to have to do now is verify the table, to see if they have been converted to bigint, I’ve summarised below:

ci_builds_metadata - build_id (bigint)
events - id (bigint)
push_event_payloads - event_id (bigint)
ci_builds - stage_id (bigint)
ci_stages - id (bigint)

you can see that from your results. So that will mean, you need to do something like this:

gitlab-psql
\d ci_builds_metadata;

and check that build_id is bigint. If it is, then check the next one, so events and check that id is bigint, then push_event_payloads to check event_id is bigint, and then ci_builds that stage_id is bigint, and ci_stages that id is bigint.

If you check, and all of these entries have been converted to bigint, then all that would need to be done is update the status for those entries from 1 to 3 in the table for batched_background_migrations. They would then disappear from the pending list.

Of course, you only do this, if the table entries have been converted to bigint. If not, then you need to run the rails commands that I posted before that should have done it for you, although not sure why they wouldn’t run as other people did it, and it worked fine.

So check that, and then we can decide whether to change the status from 1 to 3 because they are already converted to bigint, or if not, then we need to figure out now to get those tables updated.


What am I doing wrong?

So as you can see, your build_id is still integer from when you ran the \d command, look how mine looks:

gitlabhq_production=# \d ci_builds_metadata;
                                             Table "public.ci_builds_metadata"
          Column           |          Type          | Collation | Nullable |                    Default                     
---------------------------+------------------------+-----------+----------+------------------------------------------------
 project_id                | integer                |           | not null | 
 timeout                   | integer                |           |          | 
 timeout_source            | integer                |           | not null | 1
 config_options            | jsonb                  |           |          | 
 config_variables          | jsonb                  |           |          | 
 interruptible             | boolean                |           |          | 
 has_exposed_artifacts     | boolean                |           |          | 
 environment_auto_stop_in  | character varying(255) |           |          | 
 expanded_environment_name | character varying(255) |           |          | 
 secrets                   | jsonb                  |           | not null | '{}'::jsonb
 build_id                  | bigint                 |           | not null |
 id                        | bigint                 |           | not null | nextval('ci_builds_metadata_id_seq'::regclass)
 runner_features           | jsonb                  |           | not null | '{}'::jsonb
 runtime_runner_features   | jsonb                  |           | not null | '{}'::jsonb

you can see mine shows bigint. So your jobs have not finished, and as per the post before, you need to run all those gitlab-rake commands that I posted earlier on in this thread. Unfortunately, I don’t have anyway of helping you with that, but you will have to somehow figure out how to make sure all those commands get run to update your tables. The link I posted also in that post should help as plenty of people had the issue and managed to get it to work.

I don’t have a test server to help you figure it out, so unless someone else replies to this post and is able to help you out, you will have to find a way to make those rake commands work.

1 Like

Thanks a lot. I’ll post on completion.