Error 500 when accessing Users in Admin Area section

Hello,

I’m not exactly sure when this started happening, but I think even since I’ve upgraded to gitlab 16.5.0-ee, which is also my current version.

When I trying accessing the “Users” section in the “Admin Area”, I get a 500 error:

We're sorry. Something went wrong on our end.

Looking at the rails log, I see:

ActionView::Template::Error (PG::UndefinedColumn: ERROR:  column "manage_project_access_tokens" does not exist
LINE 1: ...dependency = true OR read_vulnerability = true OR manage_pro...
                                                             ^
):
    65:         = label_tag s_('AdminUsers|Sort by')
    66:         = gl_redirect_listbox_tag admin_users_sort_options(filter: params[:filter], search_query: params[:search_query]), @sort, data: { placement: 'right' }
    67:
    68: #js-admin-users-app{ data: admin_users_data_attributes(@users) }
    69:   = render Pajamas::SpinnerComponent.new(size: :lg, class: 'gl-my-7')
    70:
    71: = paginate_collection @users

lib/gitlab/database/load_balancing/connection_proxy.rb:107:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:107:in `block in read_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:224:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:106:in `read_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'
app/serializers/base_serializer.rb:15:in `represent'
app/helpers/users_helper.rb:6:in `admin_users_data_attributes'
app/views/admin/users/_users.html.haml:68
app/views/admin/users/index.html.haml:7
app/controllers/application_controller.rb:162:in `render'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'
app/controllers/application_controller.rb:498:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:489:in `set_session_storage'
lib/gitlab/i18n.rb:114:in `with_locale'
lib/gitlab/i18n.rb:120:in `with_user_locale'
app/controllers/application_controller.rb:480:in `set_locale'
app/controllers/application_controller.rb:473:in `set_current_context'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/path_traversal_check.rb:25:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

Given that manage_project_access_tokens (Custom roles | GitLab) seems to have been introduced in 16.5, I’m guessing this is a bug introduced along with this version.

Does anyone know if this bug has been addressed?

I’ve also run:

gitlab-rake gitlab:check

which references two following two errors (which intuitively I think is not strictly related to my current problem, but I might be, of course, wrong):

All migrations up? ... no
  Try fixing it:
  sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
  Please fix the error above and rerun the checks.

and:

Git user has default SSH configuration? ... no
  Try fixing it:
  mkdir ~/gitlab-check-backup-1700841257
  sudo mv /var/opt/gitlab/.ssh/id_rsa ~/gitlab-check-backup-1700841257
  sudo mv /var/opt/gitlab/.ssh/id_rsa.pub ~/gitlab-check-backup-1700841257
  For more information see:
  doc/user/ssh.md#overriding-ssh-settings-on-the-gitlab-server
  Please fix the error above and rerun the checks.

I’m running this in Centos 7 directly on the host.
Let me know if you think I should provide more information/logs. As far as I could see, there were no other relevant logs.

I’ll try to run the git command suggested above in a few hours.

Trying to run sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production doesn’t work, because it cannot find the bundle executable, so the instructions here are wrong.
Trying to run the bundle executable found under /opt/gitlab/embedded/bin as root or as git errors out:

-sh-4.2$ ./bundle exec rake db:migrate RAILS_ENV=production
/usr/bin/env: ruby: No such file or directory

I’ve also tried running it from the gitlab-rails console, but the syntax is not correct, it seems:

-sh-4.2$ gitlab-rails console
--------------------------------------------------------------------------------
 Ruby:         ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]
 GitLab:       16.5.0-ee (fc87c9d4cca) EE
 GitLab Shell: 14.29.0
 PostgreSQL:   13.11
------------------------------------------------------------[ booted in 69.52s ]
Loading production environment (Rails 7.0.8)
irb(main):001:0> bundle exec rake db:migrate RAILS_ENV=production
/opt/gitlab/embedded/lib/ruby/3.0.0/irb/workspace.rb:116:in `eval': (irb):1: syntax error, unexpected constant, expecting `do' or '{' or '(' (SyntaxError)
... exec rake db:migrate RAILS_ENV=production
...                      ^~~~~~~~~

Any help would be much appreciated.

[Later edit:]
With chatGPT’s help I was able to get the right syntax:

-sh-4.2$ gitlab-rake db:migrate RAILS_ENV=production

This resulted in the following:

-sh-4.2$ gitlab-rake db:migrate RAILS_ENV=production
main: == [advisory_lock_connection] object_id: 122100, pg_backend_pid: 3861
main: == 20231004091113 SwapColumnsForCiSourcesPipelinesPipelineIdBigint: migrating =
main: -- transaction_open?(nil)
main:    -> 0.0000s
main: -- execute("LOCK TABLE ci_pipelines, ci_sources_pipelines IN ACCESS EXCLUSIVE MODE")
main:    -> 0.0012s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME COLUMN pipeline_id TO temp_pipeline_id")
main:    -> 0.0110s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME COLUMN pipeline_id_convert_to_bigint TO pipeline_id")
main:    -> 0.0006s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME COLUMN temp_pipeline_id TO pipeline_id_convert_to_bigint")
main:    -> 0.0006s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME COLUMN source_pipeline_id TO temp_source_pipeline_id")
main:    -> 0.0006s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME COLUMN source_pipeline_id_convert_to_bigint TO source_pipeline_id")
main:    -> 0.0006s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME COLUMN temp_source_pipeline_id TO source_pipeline_id_convert_to_bigint")
main:    -> 0.0006s
main: -- quote_column_name(:trigger_68d7b6653c7d)
main:    -> 0.0001s
main: -- execute("ALTER FUNCTION \"trigger_68d7b6653c7d\" RESET ALL")
main:    -> 0.0080s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME CONSTRAINT fk_e1bad85861 TO temp_fk_e1bad85861")
main:    -> 0.0059s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME CONSTRAINT fk_c1b5dc6b6f TO fk_e1bad85861")
main:    -> 0.0009s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME CONSTRAINT temp_fk_e1bad85861 TO fk_c1b5dc6b6f")
main:    -> 0.0024s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME CONSTRAINT fk_d4e29af7d7 TO temp_fk_d4e29af7d7")
main:    -> 0.0006s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME CONSTRAINT fk_1df371767f TO fk_d4e29af7d7")
main:    -> 0.0006s
main: -- execute("ALTER TABLE ci_sources_pipelines RENAME CONSTRAINT temp_fk_d4e29af7d7 TO fk_1df371767f")
main:    -> 0.0008s
main: -- execute("ALTER INDEX index_ci_sources_pipelines_on_pipeline_id RENAME TO temp_index_ci_sources_pipelines_on_pipeline_id")
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "index_ci_sources_pipelines_on_pipeline_id" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:71:in `block (2 levels) in swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `each_with_index'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `block in swap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:39:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:29:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR:  relation "index_ci_sources_pipelines_on_pipeline_id" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:71:in `block (2 levels) in swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `each_with_index'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `block in swap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:39:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:29:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
PG::UndefinedTable: ERROR:  relation "index_ci_sources_pipelines_on_pipeline_id" does not exist
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:71:in `block (2 levels) in swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `each_with_index'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:67:in `block in swap'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retries_helpers.rb:52:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/v2.rb:99:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:39:in `swap'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20231004091113_swap_columns_for_ci_sources_pipelines_pipeline_id_bigint.rb:29:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:37:in `within'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

This looks as ugly as hell :slight_smile:

The issue has been reported here.

I’ve solved it using the solution in tn the referenced link:

#confirm table status:
gitlab-psql
\d public.ci_sources_pipelines

#If missing, add
CREATE INDEX index_ci_sources_pipelines_on_source_pipeline_id ON ci_sources_pipelines USING btree (source_pipeline_id);
CREATE INDEX index_ci_sources_pipelines_on_pipeline_id ON ci_sources_pipelines USING btree (pipeline_id);

#run db-migrate
gitlab-rake db:migrate
gitlab-rake db:migrate:status