Internal Server Error 500 while accessing $GITLAB/admin/runners

I have restored Gitlab from a backup, now every time i try to access the runners webpage, I get a Internal Server error.

I have tried uninstalling all configured gitlab-runners and accessing it, the problem persists. Here is a trace from the gitlab-rails/production.log:

Started GET "/admin/runners" for 127.0.0.1 at 2019-01-16 07:17:12 -0500
Processing by Admin::RunnersController#index as HTML
Completed 500 Internal Server Error in 66ms (ActiveRecord: 5.7ms)

ActionView::Template::Error ():
    37: 
    38:     .col-sm-6
    39:       .bs-callout
    40:         = render partial: 'ci/runner/how_to_setup_runner',
    41:                  locals: { registration_token: Gitlab::CurrentSettings.runners_registration_token,
    42:                            type: 'shared',
    43:                            reset_token_url: reset_registration_token_admin_application_settings_path }

lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
app/models/application_setting.rb:409:in `runners_registration_token'
lib/gitlab/current_settings.rb:19:in `method_missing'
app/views/admin/runners/index.html.haml:40:in `_app_views_admin_runners_index_html_haml___977288809002382166_69944849285200'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:420:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:19:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Started GET "/-/metrics" for 127.0.0.1 at 2019-01-16 07:17:24 -0500
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms)

I have tried reinstalling using docker as well as omnibus, the issue exists only after restoring any backup that I currently have. Is there a way to reset the runners from the command line or API?

Regards,
N

@NerdSec I am having the same issue after upgrading from 11.4.3 to 11.6.5.

Did you manage to find a resolution to this?

Thanks,

Sean

You can reset runner`s token by:
sudo gitlab-rails runner -e production ‘ApplicationSetting.current.reset_runners_registration_token!’

hello! i have restore gitlab from a backup, now every time i try to access the runners webpage, I get a Internal Server error.

gitlab 12.2.9

use “sudo gitlab-rails runner -e production ‘ApplicationSetting.current.reset_runners_registration_token!’” not Solve

gitlab-rails/production.log:
Completed 500 Internal Server Error in 421ms (ActiveRecord: 29.6ms)

ActionView::Template::Error ():
16: .table-section.section-10
17: .table-mobile-header{ role: ‘rowheader’ }= _(‘Runner token’)
18: .table-mobile-content
19: = link_to runner.short_sha, admin_runner_path(runner)
20:
21: .table-section.section-20
22: .table-mobile-header{ role: ‘rowheader’ }= _(‘Description’)

lib/gitlab/crypto_helper.rb:27:in aes256_gcm_decrypt' app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in get_token’
app/models/concerns/token_authenticatable.rb:30:in block in add_authentication_token_field' app/models/ci/runner.rb:213:in short_sha’
lib/gitlab/metrics/instrumentation.rb:161:in block in short_sha' lib/gitlab/metrics/method_call.rb:36:in measure’
lib/gitlab/metrics/instrumentation.rb:161:in short_sha' app/views/admin/runners/_runner.html.haml:19:in _app_views_admin_runners__runner_html_haml__924215408576677336_23895487474980’
app/views/admin/runners/index.html.haml:138:in block in _app_views_admin_runners_index_html_haml__1648131156014963702_23895507343420' app/views/admin/runners/index.html.haml:137:in _app_views_admin_runners_index_html_haml__1648131156014963702_23895507343420’
app/controllers/application_controller.rb:120:in render' lib/gitlab/session.rb:11:in with_session’
app/controllers/application_controller.rb:452:in set_session_storage' lib/gitlab/i18n.rb:55:in with_locale’
lib/gitlab/i18n.rb:61:in with_user_locale' app/controllers/application_controller.rb:446:in set_locale’
lib/gitlab/middleware/rails_queue_duration.rb:27:in call' lib/gitlab/metrics/rack_middleware.rb:17:in block in call’
lib/gitlab/metrics/transaction.rb:57:in run' lib/gitlab/metrics/rack_middleware.rb:17:in call’
lib/gitlab/middleware/multipart.rb:103:in call' lib/gitlab/request_profiler/middleware.rb:17:in call’
lib/gitlab/middleware/go.rb:20:in call' lib/gitlab/etag_caching/middleware.rb:13:in call’
lib/gitlab/middleware/correlation_id.rb:16:in block in call' lib/gitlab/middleware/correlation_id.rb:15:in call’
lib/gitlab/middleware/read_only/controller.rb:40:in call' lib/gitlab/middleware/read_only.rb:18:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/request_context.rb:26:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:29:in call' lib/gitlab/middleware/release_env.rb:12:in call’

I did all the troubleshooting steps from Back up and restore GitLab | GitLab and I can access the runners page, but restarting the runners with a new token still doesn’t work.

May 28 08:24:54 arm-runner gitlab-runner[19317]: ERROR: Runner http://arm-gitlab:8080/vfQfqNdby3jzD7qaCg15 is not healthy, but will be checked!
May 28 08:24:55 arm-runner gitlab-runner[19317]: ERROR: Checking for jobs... forbidden               runner=vfQfqNdb
May 28 08:25:01 arm-runner gitlab-runner[19317]: ERROR: Checking for jobs... forbidden               runner=vfQfqNdb
May 28 08:25:09 arm-runner gitlab-runner[19317]: ERROR: Checking for jobs... forbidden               runner=vfQfqNdb
May 28 08:25:09 arm-runner gitlab-runner[19317]: ERROR: Runner http://arm-gitlab:8080/vfQfqNdby3jzD7qaCg15 is not healthy and will be disabled!