After updating from version Gitlab 11.8.10 breaks runners CI

ggeeorgg · September 18, 2019, 7:14am

Summary
After upgrading from Omnibus GitLab 11.8.10 to any higher version breaks Runners CI

Symptoms

Page https://gitlab.local/admin/runners returns error 500
Existing Runners cannot complete jobs

What is the expected correct behavior?

Upgrading Gitlab to the current version with existing Runners.

Relevant logs and/or screenshots

Log Runner:

WARNING: Appending trace to coordinator... failed   code=500 job=59435 job-log= job-status= runner=******** sent-log=0-501 status=500 Internal Server Error
WARNING: Appending trace to coordinator... failed   code=500 job=59435 job-log= job-status= runner=******** sent-log=0-501 status=500 Internal Server Error

Log production.log

Started PATCH "/api/v4/jobs/59435/trace" for 192.168.1.158 at 2019-09-01 15:21:22 +0300
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms)

OpenSSL::Cipher::CipherError ():
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/project.rb:1602:in `runners_token'
  /opt/gitlab/embedded/service/gitlab-rails/app/models/ci/build.rb:770:in `hide_secrets'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/trace.rb:59:in `block in append'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/trace.rb:136:in `unsafe_write!'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/trace.rb:97:in `block in write'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/exclusive_lease_helpers.rb:29:in `in_lock'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/trace.rb:167:in `in_write_lock'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/trace.rb:96:in `write'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/trace.rb:55:in `append'
  /opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:195:in `block (2 levels) in <class:Runner>'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in `block (2 levels) in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in `block in generate_api_method'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in `block in run'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in `block in build_stack'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource.rb:20:in `_call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-oauth2-1.9.3/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:103:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:17:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.4.2/lib/labkit/correlation/correlation_id.rb:18:in `use_id'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:15:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/batch-loader-1.4.0/lib/batch_loader/middleware.rb:11:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/apollo_upload_server-2.0.0.beta.3/lib/apollo_upload_server/middleware.rb:20:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:40:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:26:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/rack.rb:51:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:12:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
  /opt/gitlab/embedded/bin/unicorn:23:in `load'
  /opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
  /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
  /opt/gitlab/embedded/bin/bundle:23:in `load'
  /opt/gitlab/embedded/bin/bundle:23:in `<main>'

starcraft66 · September 24, 2019, 4:46am

I had the same issue that turned out to be related to some secrets not being properly restored when restoring a gitlab backup. Pasting the following code into the gitlab-rails console and then restarting gitlab and the runners fixed the problem for me.

table_column_combos = [
    [Namespace, "runners_token_encrypted", "runners_token"],
    [Project, "runners_token_encrypted", "runners_token"],
    [Ci::Build, "token_encrypted", "token"],
    [Ci::Runner, "token_encrypted", "token"],
    [ApplicationSetting, "runners_registration_token_encrypted", "runners_registration_token"],
    [Group, "runners_token_encrypted"],
]


table_column_combos.each do |table,column,column2|
  total = 0
  fixed = 0
  removed = 0
  bad = []
  table.find_each do |data|
    begin
      total += 1
      ::Gitlab::CryptoHelper.aes256_gcm_decrypt(data[column])
    rescue => e
      if data[column2].to_s.empty?
        puts "Bad Value column2 is empty: #{data[column]}"
        data[column] = nil
        data.save()
        removed += 1
      else
        puts "Bad Value: #{data[column]}"
        data[column] = ::Gitlab::CryptoHelper.aes256_gcm_encrypt(data[column2])
        data.save()
        fixed += 1
      end
      bad << data
    end
  end
  puts "Table: #{table.name}    Bad #{bad.length} / Good #{total}, Fixed #{fixed}, Removed #{removed}"
end

Source: https://gitlab.com/gitlab-org/gitlab-foss/issues/58524

gitlab-greg · September 25, 2019, 6:58pm

Hi @ggeeorgg and welcome to the GitLab Community forums!

What version did you upgrade to?

I suspect the error you’re seeing is the result of the GitLab runner version being out of sync with the GitLab version after the upgrade.

The GitLab Runner version should be in sync with the GitLab version. While older Runners may still work with newer GitLab versions, and vice versa, in some cases, features may be not available or work properly if there’s a version difference.

Can you try upgrading your runner version to match your GitLab version and confirm if the problem persists?

Topic		Replies	Views
All Runners are no more online while in contact after update 16.11 GitLab CI/CD runner	6	1330	April 25, 2024
Error 500 on job submission GitLab CI/CD	3	2565	August 25, 2017
17.7 Upgrade breaks Gitlab-Runner jobs with token not set error Upgrade runner	14	1325	January 8, 2025
Internal Server Error 500 while accessing /admin/runners Self-managed api , ci , runner	2	2258	November 30, 2021
"Can't verify CSRF token authenticity" on Gitlab Runner How to Use GitLab	0	1273	November 3, 2019

After updating from version Gitlab 11.8.10 breaks runners CI

Related topics