Unable to login after performing upgrade to v15

chewie · April 25, 2023, 9:12pm

I have been attempting to upgrade our instance of Gitlab from v13.12.15-ee to v15 via this path…

14.012 > 14.3.6 > 14.9.5 > 14.10.5 > 15.0.5 > 15.4.6 > 15.5.0. I have a clone of our production server and have changed the URL just to ensure the upgrade goes well. Everything was fine until i got to 15.5.0. Now I am not able to login via LDAP and I cannot reset the root(Administrator) password.

The error I’m getting when logging in via LDAP is:
Could not authenticate you from Ldapmain because “Undefined method `website url changed?’ for #<user id:175 @some.user> did you mean? user type changed?”.

I’ve ran a trace to reset root’s password (no one knows what it is) and here’s the output. Any help is appreciated… I do have snapshots but would like to get this working as I really want to get our prod env up to date.


root@vmhq-p-gitlab:/var/log/gitlab/gitlab-rails# gitlab-rake "gitlab:password:reset[root]" --trace
** Invoke gitlab:password:reset (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:password:reset
Enter password:
Confirm password:
rake aborted!
**NoMethodError: undefined method `website_url_changed?' for #<User id:1 @root>**
**Did you mean?  user_type_changed?**
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.6.1/lib/active_model/attribute_methods.rb:469:in `method_missing'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:427:in `block in make_lambda'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:178:in `block (2 levels) in halting_and_conditional'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:178:in `all?'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:178:in `block in halting_and_conditional'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:512:in `block in invoke_before'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:512:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:512:in `invoke_before'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:105:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:824:in `_run_validate_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.6.1/lib/active_model/validations.rb:406:in `run_validations!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.6.1/lib/active_model/validations/callbacks.rb:117:in `block in run_validations!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-activemodel-0.8.0/lib/state_machines/integrations/active_model.rb:400:in `block in around_validation'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-activemodel-0.8.0/lib/state_machines/integrations/active_model.rb:400:in `around_validation'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:137:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:824:in `_run_validation_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.6.1/lib/active_model/validations/callbacks.rb:117:in `run_validations!'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activemodel-6.1.6.1/lib/active_model/validations.rb:337:in `valid?'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/validations.rb:68:in `valid?'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/validations.rb:84:in `perform_validations'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/validations.rb:47:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:298:in `block in save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:193:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:298:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/suppressor.rb:44:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:511:in `block in save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:530:in `block in around_save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:530:in `around_save'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:511:in `save'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/password.rake:19:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:483:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:31:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/cli.rb:25:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:48:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/exe/bundle:36:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:password:reset

timsimo · April 26, 2023, 3:28am

Experiencing the same error preforming a backup and restore on version 15.10.

Attempted the backup and restore yesterday and did not encounter this error. This was a new error when we tried to conduct the backup and restore again today.

chewie · April 26, 2023, 2:36pm

I managed to get this resolved by performing an apt-get install gitlab-ee. I am now at the latest revision and have no LDAP errors.

wallemand · October 11, 2023, 11:53pm

Hi there,
I have same error on GitLab CE on docker: i’ve upgraded from 14.10.5 to 15.0.0 it was ok
then i upgraded to 15.11.3 and i have this error:

Could not authenticate you from Ldapmain because “Undefined method `website url changed?’ for #<user id:294 @wallemand> did you mean? user type changed?”

Anyone have an idea on this ?
I performed a rollback to 14.10.5 but I’m stuck same with the error. It’s a mess !

Do i need to restore the datas ?

wallemand · October 16, 2023, 11:23am

Solution found: in fact we managed to upgrade from 14.10.5 to 16.2.8 (we are in docker mode) with this path: 14.10.5 => 15.0.0 => 15.2.2 => 15.11.5 => 16.2.8

Details:

First rollback to 14.10.5 with restore of data.

Then we upgraded again from 14.10.5 to 15.0.0 => it was OK

Then we upgraded again to 15.11.5 and we had an error with migration of POSTGRE (BDD of the git) :

web_1 | Caused by:
web_1 | PG::CheckViolation: ERROR: check constraint “check_70f294ef54” is violated by some row
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:119:in block in validate_check_constraint'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/timeout_helpers.rb:31:in disable_statement_timeout’
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:116:in validate_check_constraint'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/constraints_helpers.rb:232:in validate_not_null_constraint’
web_1 | /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20230223014251_validate_not_null_constraint_on_oauth_access_tokens_expires_in.rb:7:in up'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in block in exec_migration’
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in within'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in exec_migration’
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:19:in exec_migration'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in ddl_transaction’
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in block in with_advisory_lock_connection'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in with_advisory_lock_connection’
web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:117:in configure_database'* *web_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in block (3 levels) in <top (required)>’
web_1 | /opt/gitlab/embedded/bin/bundle:23:in load'* *web_1 | /opt/gitlab/embedded/bin/bundle:23:in ’
web_1 | Tasks: TOP => db:migrate
web_1 | (See full trace by running task with --trace)
web_1 | Running db:migrate rake task
web_1 | main: == [advisory_lock_connection] object_id: 223560, pg_backend_pid: 386
web_1 | main: == 20230223014251 ValidateNotNullConstraintOnOauthAccessTokensExpiresIn: migrating
web_1 | main: – execute(“SET statement_timeout TO 0”)
web_1 | main: → 0.0004s
web_1 | main: – execute(“ALTER TABLE oauth_access_tokens VALIDATE CONSTRAINT check_70f294ef54;”)
web_1 | main: – execute(“RESET statement_timeout”)
web_1 | main: → 0.0004s
web_1 | STDERR:
web_1 | ---- End output of “bash” ----
web_1 | Ran “bash” returned 1

In GitLab forum we found some informations on db migration errors: rails_migration[gitlab-rails] Error executing action run from ee-15.1.3 to ee-15.2.0 (#368631) · Issues · GitLab.org / GitLab · GitLab
Which are fixed in release 15.2.2: Fix RescheduleBackfillImportedIssueSearchData migration (!93311) · Merge requests · GitLab.org / GitLab · GitLab

Then we start step back to 15.0.0 (with restore of the snapshot) and upgrade with following steps: 15.0.0 => 15.2.2 => 15.11.5 =>16.2.8.

And it worked fine