Stuck on Onmibus 7.14.3, db:migrate upgrades don't work

Has anyone had success with omnibus upgrades from version 7.14.3 ? I am running Ubuntu 16.04 LTS with 7.14.3 installed. I have tried upgrading to 8.0.0, 8.2.0, and 8.5.8. All upgrades fail during db:migrate. It has problems with tables that already exist, or don’t exist. Columns that exist, or don’t exist. Here’s an example:

Running handlers:
[2016-06-20T16:10:22-07:00] ERROR: Running exception handlers
Running handlers complete
[2016-06-20T16:10:22-07:00] ERROR: Exception handlers complete
Chef Client failed. 26 resources updated in 32.204422255 seconds
[2016-06-20T16:10:22-07:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2016-06-20T16:10:22-07:00] ERROR: bash[migrate gitlab-rails database] (gitlab::database_migrations line 19) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20160620-28038-1r5r0h7” ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedColumn: ERROR: column “github_access_token” of relation “users” does not exist
: ALTER TABLE “users” DROP “github_access_token”/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in async_exec' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in block in execute’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract_adapter.rb:378:in block in log' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/notifications/instrumenter.rb:20:in instrument’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract_adapter.rb:372:in log' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:127:in execute’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/schema_statements.rb:397:in remove_column' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:656:in block in method_missing’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:628:in block in say_with_time' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:628:in say_with_time’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:648:in method_missing' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150814065925_remove_oauth_tokens_from_users.rb:3:in change’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:602:in exec_migration' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:586:in block (2 levels) in migrate’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:585:in block in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in with_connection’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:584:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:759:in migrate’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:1044:in block in ddl_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in block in transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in within_new_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/transactions.rb:208:in transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:1044:in ddl_transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:997:in execute_migration_in_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:959:in block in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:955:in each’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:955:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:814:in up’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:792:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/railties/databases.rake:34:in block (2 levels) in <top (required)>’
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column “github_access_token” of relation “users” does not exist
: ALTER TABLE “users” DROP “github_access_token”
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in async_exec' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in block in execute’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract_adapter.rb:378:in block in log' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/notifications/instrumenter.rb:20:in instrument’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract_adapter.rb:372:in log' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:127:in execute’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/schema_statements.rb:397:in remove_column' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:656:in block in method_missing’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:628:in block in say_with_time' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:628:in say_with_time’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:648:in method_missing' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150814065925_remove_oauth_tokens_from_users.rb:3:in change’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:602:in exec_migration' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:586:in block (2 levels) in migrate’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:585:in block in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in with_connection’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:584:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:759:in migrate’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:1044:in block in ddl_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in block in transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in within_new_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/transactions.rb:208:in transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:1044:in ddl_transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:997:in execute_migration_in_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:959:in block in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:955:in each’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:955:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:814:in up’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:792:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/railties/databases.rake:34:in block (2 levels) in <top (required)>’
PG::UndefinedColumn: ERROR: column “github_access_token” of relation “users” does not exist
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in async_exec' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:128:in block in execute’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract_adapter.rb:378:in block in log' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/notifications/instrumenter.rb:20:in instrument’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract_adapter.rb:372:in log' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/postgresql/database_statements.rb:127:in execute’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/schema_statements.rb:397:in remove_column' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:656:in block in method_missing’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:628:in block in say_with_time' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:628:in say_with_time’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:648:in method_missing' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150814065925_remove_oauth_tokens_from_users.rb:3:in change’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:602:in exec_migration' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:586:in block (2 levels) in migrate’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:585:in block in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/connection_pool.rb:294:in with_connection’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:584:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:759:in migrate’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:998:in block in execute_migration_in_transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:1044:in block in ddl_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in block in transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in within_new_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/transactions.rb:208:in transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:1044:in ddl_transaction' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:997:in execute_migration_in_transaction’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:959:in block in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:955:in each’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:955:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:814:in up’
/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/migration.rb:792:in migrate' /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.1.12/lib/active_record/railties/databases.rake:34:in block (2 levels) in <top (required)>’
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20150814065925 RemoveOauthTokensFromUsers: migrating =======================
– remove_column(:users, :github_access_token, :string)
STDERR:
---- End output of “bash” “/tmp/chef-script20160620-28038-1r5r0h7” ----
Ran “bash” “/tmp/chef-script20160620-28038-1r5r0h7” returned 1
[2016-06-20T16:10:24-07:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
`

Hi

Direct upgrade from 7.1 to 8.0 will fail in db migrate steps. Even I faced the similar problem.Instead please take two steps to upgrade from 7.1 to 8.0.

Gitlab 7.1 to Gitlab 7.4
Gitlab 7.4 to Gitlab 7.6
Gitlab 7.6 to Gitlab 8.0

I have documented my experiences doing a similar upgrade you can have a look. Please post questions if any.

Thank you for your detailed instructions! I found them before I posted my issue, but unfortunately going from 7.14.3 to 8.0.0 does not work.

What I ended up doing is looking at each failed DB migration source file and applying the migration by hand using psql. If the migration required altering a table that already has data, I ran alter table mytable alter column mycolumn type character varying (that was the most comment alteration required. Then I would insert the migration timestamp into the schema_migrations table.

If the table had no data, I would just do a drop table mytable, then grep the migrations files for all references to that table and remove those migration timestamps from the schema_migrations table, then re-run db:migrate

It was a very painstaking and painful process, but I finally got an upgrade to Omnibus 8.8.5. The 8.9.0 release came out yesterday and it automatically upgraded without any issues.

1 Like