I had to manually modify the migration to finish upgrade

Hi there,

I had to upgrade an old 13.6-ee installed via Debian 9 packages to the latest 14.4.1 as a docker version on a Debian 11. Like many people, I faced the hard problem of the 14.2.x upgrade failing because of the migrations. I tried the official solution which states that the problem has been solved and that it is sufficient to follow a very precise upgrade path, checking each time that the migrations are completed. I’m sorry to tell it: but like many people on the Internet, it never worked for me.

Failing being absolutely not an option, I tried harder and found this topic on the forum:

It was less or more my problem : the bigInt migrations were failing. So, like Dave Robertson I decided to manually modify the migration to check if it works. Finally, I deleted all migrations concerning bigInt. First I launched my docker, then I logged in immediately, and then I executed:

cd /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/ && rm *bigint*

It worked. Now, I finally have Gitlab working again… less or more.

Some things are not working (like the project activity not being updated, adding a new user to a project leading to a 500 (adding group works)). Maybe it’s related to the big Int migration problems, maybe not. But I’d like to finish correctly this upgrade, and maybe, help Gitlab team to release a new release that really work for everybody, even for people upgrading from 13.x in a docker installation.

So right now, when I go to monitoring health checks, I see:

20210701141346_finalize_ci_builds_stage_id_bigint_conversion.rb 20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb 20210707210916_finalize_ci_stages_bigint_conversion.rb 20210708011426_finalize_ci_builds_metadata_bigint_conversion.rb 20210713042153_finalize_ci_sources_pipelines_bigint_conversion.rb 20210714015537_finalize_ci_build_trace_chunks_bigint_conversion.rb 20210802131812_finalize_convert_deployments_bigint.rb 20210804151444_prepare_indexes_for_ci_job_artifact_bigint_conversion.rb 20210804153307_prepare_indexes_for_tagging_bigint_conversion.rb 20210804154407_prepare_indexes_for_ci_stage_bigint_conversion.rb 20210805131510_finalize_ci_builds_runner_session_bigint_conversion.rb 20210806131706_finalize_taggins_bigint_conversion.rb 20210809143931_finalize_job_id_conversion_to_bigint_for_ci_job_artifacts.rb 20210817024335_prepare_indexes_for_events_bigint_conversion.rb 20210901044202_push_event_payloads_bigint_conversion_remove_triggers.rb 20210901044237_events_bigint_conversion_remove_triggers.rb 20210907013944_cleanup_bigint_conversion_for_ci_builds_metadata.rb 20210907021940_cleanup_bigint_conversion_for_ci_stages.rb 20210907033745_cleanup_bigint_conversion_for_deployments.rb 20210907041000_cleanup_bigint_conversion_for_geo_job_artifact_deleted_events.rb 20210907211557_finalize_ci_builds_bigint_conversion.rb 20210915022415_cleanup_bigint_conversion_for_ci_builds.rb

what would you suggest?
I think I’m going to check those migration one by one, and verify if the columns they apply to really exist.
What do you think?

so, first tries.

$vi /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210701141346_finalize_ci_builds_stage_id_bigint_conversion.rb

give me:

class FinalizeCiBuildsStageIdBigintConversion < ActiveRecord::Migration[6.1]
  include Gitlab::Database::MigrationHelpers

  disable_ddl_transaction!

  TABLE_NAME = 'ci_builds'

  def up
    ensure_batched_background_migration_is_finished(
      job_class_name: 'CopyColumnUsingBackgroundMigrationJob',
      table_name: TABLE_NAME,
      column_name: 'id',
      job_arguments: [%w[id stage_id], %w[id_convert_to_bigint stage_id_convert_to_bigint]]
    )


    swap_columns
  end

so if I understand well, it’s trying to swap columns id <-> id_convert_to_bigint and stage_id <-> stage_id_convert_to_bigint

If I connect to the database:

$ su gitlab-psql
$ /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d  gitlabhq_production

and then checks the ci_builds table:

gitlabhq_production=# SELECT 
   table_name, 
   column_name, 
   data_type 
FROM 
   information_schema.columns
WHERE 
   table_name = 'ci_builds';

it gives me:

 table_name |        column_name         |          data_type          
------------+----------------------------+-----------------------------
 ci_builds  | stage_id_convert_to_bigint | bigint
 ci_builds  | queued_at                  | timestamp without time zone
 ci_builds  | lock_version               | integer
 ci_builds  | auto_canceled_by_id        | integer
 ci_builds  | retried                    | boolean
 ci_builds  | stage_id                   | integer
 ci_builds  | protected                  | boolean
 ci_builds  | failure_reason             | integer
 ci_builds  | scheduled_at               | timestamp with time zone
 ci_builds  | upstream_pipeline_id       | integer
 ci_builds  | resource_group_id          | bigint
 ci_builds  | waiting_for_resource_at    | timestamp with time zone
 ci_builds  | processed                  | boolean
 ci_builds  | scheduling_type            | smallint
 ci_builds  | id_convert_to_bigint       | bigint
 ci_builds  | id                         | integer

and many others cols.
But basically the needed cols are here.

Now if I do:

gitlabhq_production=# SELECT id, stage_id, id_convert_to_bigint, stage_id_convert_to_bigint  
                                   FROM ci_builds                                     
                                   ORDER BY id DESC   
                                   LIMIT 10;

It gives me:

  id  | stage_id | id_convert_to_bigint | stage_id_convert_to_bigint 
------+----------+----------------------+----------------------------
 5014 |     2804 |                 5014 |                       2804
 5013 |     2804 |                 5013 |                       2804
 5012 |     2804 |                 5012 |                       2804
 5011 |     2804 |                 5011 |                       2804
 5010 |     2804 |                 5010 |                       2804
 5009 |     2804 |                 5009 |                       2804
 5008 |     2804 |                 5008 |                       2804
 5007 |     2804 |                 5007 |                       2804
 5006 |     2803 |                 5006 |                       2803
 5005 |     2802 |                 5005 |                       2802
(10 rows)

So needed data are here too.
So I bet this migration is ok and could be apply.

I’ll continue my checks.

Since everything is fine and should work for this migration, I followed the documentation :

so I did:

$ gitlab-rake --trace gitlab:background_migrations:finalize[FinalizeCiBuildsStageIdBigintConversion,ci_builds,id,'[%w[id stage_id], %w[id_convert_to_bigint stage_id_convert_to_bigint]]']

and it fails with this result:

** Invoke gitlab:background_migrations:finalize (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:background_migrations:finalize
rake aborted!
JSON::ParserError: unexpected character ([0]) at line 1, column 2 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:87:in `rescue in adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:82:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Oj::ParseError: unexpected character ([0]) at line 1, column 2 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize

I’ll continue checking the other migrations one by one tomorrow. Maybe when I’ll find a wrong one, with a missing column or something, I’ll fix it and then just do a

sudo gitlab-rake db:migrate

@GalacticLouis have you by chance figured out the issue? I’m running into the same issue.

Thank you.

EDIT:: I was able to run each one it was dumping to the screen but I had to copy it from the terminal window or it was giving me the same error you had. I finally got thru the migration.

well finally I started the process from the beginning again, until the very last release before yesterday (14.4.2). To control the migration backups, I used the web interface (before I was doing it from the command line). Everything went well, and there are no more problems.