500 Error access /admin/runners, not a migration

Problem to solve

The basic problem is I get a 500 error when I go to /admin/runner. This seems to be a common issue.

This is not a migration, I suspect it is either file corruption or db corruption but I can’t tell. I have a valid ‘gitlab-secrets.json’ file and it has a ci_jwt_signing_key.

In the log file I get:

{"method":"GET","path":"/admin/runners","format":"html","controller":"Admin::RunnersController","action":"index","status":500,"time":"2024-03-06T21:45:51.375Z","params":[],"correlation_id":"01HRAVNPDBR82XPZ0F6QRD7WY7","meta.caller_id":"Admin::RunnersController#index","meta.remote_ip":"10.*.*.*","meta.feature_category":"runner","meta.user":"*****","meta.user_id":1,"meta.client_id":"user/1","remote_ip":"10.*.*.*","user_id":1,"username":"****","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36","queue_duration_s":0.054201,"request_urgency":"low","target_duration_s":5,"redis_calls":22,"redis_allowed_cross_slot_calls":1,"redis_duration_s":0.007308,"redis_read_bytes":3690,"redis_write_bytes":1717,"redis_cache_calls":17,"redis_cache_duration_s":0.005667,"redis_cache_read_bytes":3297,"redis_cache_write_bytes":1022,"redis_sessions_calls":1,"redis_sessions_duration_s":0.000126,"redis_sessions_read_bytes":390,"redis_sessions_write_bytes":85,"redis_shared_state_calls":4,"redis_shared_state_allowed_cross_slot_calls":1,"redis_shared_state_duration_s":0.001515,"redis_shared_state_read_bytes":3,"redis_shared_state_write_bytes":610,"db_count":5,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":5,"db_main_count":5,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.033,"db_main_duration_s":0.033,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.157737,"mem_objects":87300,"mem_bytes":10527416,"mem_mallocs":42012,"mem_total_bytes":14019416,"pid":521,"worker_id":"puma_4","rate_limiting_gates":[],"exception.class":"ActionView::Template::Error","exception.message":"","exception.backtrace":["lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'","app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'","app/models/concerns/token_authenticatable_strategies/base.rb:44:in `ensure_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:38:in `ensure_token'","app/models/concerns/token_authenticatable.rb:49:in `block in add_authentication_token_field'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'","app/models/concerns/token_authenticatable_strategies/base.rb:57:in `reset_token!'","app/models/concerns/token_authenticatable_strategies/base.rb:50:in `ensure_token!'","app/models/concerns/token_authenticatable.rb:54:in `block in add_authentication_token_field'","app/models/application_setting_implementation.rb:499:in `runners_registration_token'","lib/gitlab/current_settings.rb:31:in `method_missing'","app/helpers/ci/runners_helper.rb:62:in `admin_runners_data_attributes'","app/views/admin/runners/index.html.haml:4","app/controllers/application_controller.rb:132:in `render'","app/controllers/application_controller.rb:468:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:459:in `set_session_storage'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_controller.rb:450:in `set_locale'","app/controllers/application_controller.rb:443:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/path_traversal_check.rb:35:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"exception.cause_class":"OpenSSL::Cipher::CipherError","db_duration_s":0.35108,"view_duration_s":0.0,"duration_s":0.54022}

Steps to reproduce

What I have tried:

  1. Database queries:
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
-- Clear key used for JWT authentication
-- This may break the $CI_JWT_TOKEN job variable:
-- https://gitlab.com/gitlab-org/gitlab/-/issues/325965
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;

UPDATE ci_builds SET token_encrypted = null;

-- truncate web_hooks table
TRUNCATE integrations, chat_names, issue_tracker_data, jira_tracker_data, slack_integrations, web_hooks, zentao_tracker_data, web_hook_logs CASCADE;
  1. In gitlab-rails console, gleamed from here I have run:
settings = ApplicationSetting.last
settings.update_column(:runners_registration_token_encrypted, nil)
settings.update_column(:encrypted_ci_jwt_signing_key, nil)
  1. I have run gitlab-rake gitlab:doctor:secrets VERBOSE=1 and get this:
    links: Gitlab Docs and issue report
...
I, [2024-03-06T21:57:12.478678 #1069]  INFO -- : - ApplicationSetting failures: 1
D, [2024-03-06T21:57:12.478834 #1069] DEBUG -- :   - ApplicationSetting[1]: runners_registration_token, error_tracking_access_token
...
I, [2024-03-06T21:57:12.711438 #1069]  INFO -- : - Operations::FeatureFlagsClient failures: 1
D, [2024-03-06T21:57:12.711529 #1069] DEBUG -- :   - Operations::FeatureFlagsClient[1]: token
...

Then I have run:
gitlab-rake gitlab:doctor:reset_encrypted_tokens VERBOSE=true MODEL_NAMES='ApplicationSetting' TOKEN_NAMES='runners_registration_token' DRY_RUN=false

and it errors with:

I, [2024-03-06T22:00:32.005295 #1116]  INFO -- : Resetting runners_registration_token on ApplicationSetting if they can not be read
D, [2024-03-06T22:00:35.486382 #1116] DEBUG -- : > Fix ApplicationSetting[1].runners_registration_token
rake aborted!
OpenSSL::Cipher::CipherError: 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:44:in `ensure_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:38:in `ensure_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:49:in `block in add_authentication_token_field'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:57:in `reset_token!'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:59:in `block in add_authentication_token_field'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:51:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:51:in `rescue in fix_attribute'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:47:in `fix_attribute'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:39:in `block (2 levels) in fix_model'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:38:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:38:in `block in fix_model'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:37:in `with_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:37:in `fix_model'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:23:in `block in run!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:22:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:22:in `run!'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/doctor/secrets.rake:27:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
OpenSSL::Cipher::CipherError: 
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in `get_encrypted_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:113:in `token_set?'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:44:in `ensure_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:38:in `ensure_token'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:49:in `block in add_authentication_token_field'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:57:in `reset_token!'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:50:in `ensure_token!'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:54:in `block in add_authentication_token_field'
/opt/gitlab/embedded/service/gitlab-rails/app/models/application_setting_implementation.rb:499:in `runners_registration_token'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:48:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:48:in `fix_attribute'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:39:in `block (2 levels) in fix_model'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:38:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:38:in `block in fix_model'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:37:in `with_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:37:in `fix_model'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:23:in `block in run!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:22:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/doctor/reset_tokens.rb:22:in `run!'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/doctor/secrets.rake:27:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:doctor:reset_encrypted_tokens
(See full trace by running task with --trace

Configuration

This is running in a Microk8s v1.28.7 cluster in Ubuntu 22.04.4 LTS. This is not using helm charts. I have removed and stopped all runners.

Versions

Please select whether options apply, and add the version information.

Versions

  • GitLab Community Edition v16.9.2

Had the same problem on 16.9.3 CE

You have two “broken” tokens for ApplicationSetting as shown in logs from gitlab:doctor:secrets:

ApplicationSetting[1]: runners_registration_token, error_tracking_access_token

Fixes for runners_registration_token will not work if there are any other broken tokens.

Connect to database and set second broken token to null too

UPDATE application_settings SET error_tracking_access_token_encrypted = null;