Need help to resolve problem with gitlab-runners

Hello,

I have already move gitlab-ce from a to b server according to instruction (gitlab-secrets.json has been moved from a to b server). On a new server a had added 4 runners: mac, windows, centos & ubuntu. Mac & windows runner work fine, centos & ubuntu do not work fine (see P.S.). Errors on the gitlab server occurs (see PP.S.). Does anybody have any idea to ho resolve issue?

software versions: gitlab-ce.x86_64 - 11.7.5-ce.0.el7, gitlab-runner.x86_64 - 11.7.0-1

Artem.

p.s.
● gitlab-runner.service - GitLab Runner
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-02-16 15:23:45 CET; 1h 19min ago
Main PID: 16421 (gitlab-runner)
CGroup: /system.slice/gitlab-runner.service
└─16421 /usr/lib/gitlab-runner/gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner

Feb 16 16:38:33 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:38:33 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:39:33 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:39:33 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:40:34 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:40:34 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:41:34 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:41:34 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:42:34 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
Feb 16 16:42:34 centos7-4 gitlab-runner[16421]: WARNING: Checking for jobs… failed runner=_dGxnWmg status=500 Internal Server Error
[root@centos7-4 etc]# packet_write_wait: Connection to 116.203.107.250 port 22: Broken pipe

PP.S.
Started POST “/api/v4/jobs/request” for 116.203.107.250 at 2019-02-16 17:19:37 +0100
Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON
Parameters: {“info”=>{“name”=>“gitlab-runner”, “version”=>“11.7.0”, “revision”=>“8bb608ff”, “platform”=>“linux”, “architecture”=>“amd64”, “executor”=>“shell”, “shell”=>“bash”, “features”=>{“variables”=>"[FILTERED]", “image”=>false, “services”=>false, “artifacts”=>true, “cache”=>true, “shared”=>true, “upload_multiple_artifacts”=>true, “upload_raw_artifacts”=>true, “session”=>true, “terminal”=>true}}, “token”=>"[FILTERED]", “request_forgery_protection”=>{“info”=>{“name”=>“gitlab-runner”, “version”=>“11.7.0”, “revision”=>“8bb608ff”, “platform”=>“linux”, “architecture”=>“amd64”, “executor”=>“shell”, “shell”=>“bash”, “features”=>{“variables”=>"[FILTERED]", “image”=>false, “services”=>false, “artifacts”=>true, “cache”=>true, “shared”=>true, “upload_multiple_artifacts”=>true, “upload_raw_artifacts”=>true, “session”=>true, “terminal”=>true}}, “token”=>"[FILTERED]"}}
Can’t verify CSRF token authenticity.
This CSRF token verification failure is handled internally by GitLab::RequestForgeryProtection
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that current_user will be nil for the duration of the request
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms)

OpenSSL::Cipher::CipherError ():
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in final' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in crypt’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in decrypt' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/crypto_helper.rb:27:in aes256_gcm_decrypt’
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in get_token' /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/base.rb:27:in ensure_token’
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in ensure_token' /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/token_authenticatable.rb:38:in block in add_authentication_token_field’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:382:in block in make_lambda' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:169:in block (2 levels) in halting’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:770:in block (2 levels) in deprecated_false_terminator' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:769:in catch’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:769:in block in deprecated_false_terminator' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:170:in block in halting’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in block in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:454:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:101:in run_callbacks
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in _run_save_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/callbacks.rb:298:in create_or_update’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/persistence.rb:129:in save' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/validations.rb:44:in save’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/attribute_methods/dirty.rb:22:in save' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:319:in block (2 levels) in save’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:395:in block in with_transaction_returning_status' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:211:in transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:392:in with_transaction_returning_status’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:319:in block in save' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:334:in rollback_active_record_state!’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:318:in save' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/suppressor.rb:41:in save’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in block in save' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in block in around_save’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in block in run_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in catch_exceptions’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in run_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in run_callbacks’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:224:in run_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:128:in block in run_callbacks’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:371:in before' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in block in run_callbacks’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in block in pausable' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in catch’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:292:in pausable' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:190:in run_callbacks’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:127:in run_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in run_callbacks’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in block (2 levels) in perform' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in catch’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in block in perform' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:182:in block in within_transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:171:in block in within_transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in block in within_transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:555:in block in transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in block in transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/transaction.rb:189:in within_new_transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:232:in transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/transactions.rb:211:in transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:554:in transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in within_transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in within_transaction’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in within_transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in perform’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in around_save' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in save’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in block (2 levels) in run_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:154:in block in run_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in catch_exceptions’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in run_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:60:in perform’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in perform' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in fire’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in block in add_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in block (2 levels) in define_helper’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:224:in block in add_actions' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in block (2 levels) in define_helper’
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:90:in assign_runner!' /opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:45:in block in execute’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/delegation.rb:38:in each' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7.1/lib/active_record/relation/delegation.rb:38:in each’
/opt/gitlab/embedded/service/gitlab-rails/app/services/ci/register_job_service.rb:39:in execute' /opt/gitlab/embedded/service/gitlab-rails/lib/api/runner.rb:117:in block (2 levels) in class:Runner
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:57:in block (2 levels) in generate_api_method’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in instrument' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:56:in block in generate_api_method’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:262:in block in run' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in instrument’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:243:in run' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:313:in block in build_stack’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in block in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in catch’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in block in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in catch’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in call!’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in exec’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:121:in process_route' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:74:in block in identity’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:93:in transaction' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:72:in identity’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:57:in block in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:137:in with_optimization’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:56:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:119:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:45:in call!' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:40:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:17:in block in <class:Constraints>' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:46:in serve’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:39:in block in serve' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in each’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in serve' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:727:in call’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:103:in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:16:in call’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:20:in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:13:in call’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:16:in block in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/correlation_id.rb:15:in use_id’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/correlation_id.rb:15:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/batch-loader-1.2.2/lib/batch_loader/middleware.rb:11:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in block in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in catch' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in call’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:40:in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:18:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in context' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/cookies.rb:613:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:38:in block in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:97:in __run_callbacks__' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in _run_call_callbacks’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:90:in run_callbacks' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:36:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/remote_ip.rb:79:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:49:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/basic_health_check.rb:25:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:36:in call_app' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in block in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in block in tagged' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:26:in tagged’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in tagged' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in call’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:20:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/request_id.rb:24:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/executor.rb:12:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in call' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:29:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/sentry-raven-2.7.4/lib/raven/integrations/rack.rb:51:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/engine.rb:522:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in public_send' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in method_missing’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:13:in call' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in block in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in each' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in call’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in process_client' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in process_client’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in worker_loop' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in spawn_missing_workers’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:132:in start' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/unicorn-5.1.0/bin/unicorn:126:in <top (required)>’
/opt/gitlab/embedded/bin/unicorn:23:in load' /opt/gitlab/embedded/bin/unicorn:23:in <top (required)>’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in load' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in kernel_load’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:28:in run' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:424:in exec’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in dispatch' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in dispatch’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in start' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in start’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:30:in block in <top (required)>' /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in with_friendly_errors’
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:22:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’

It appears you didn’t transfer your gitlab-secrets.json to your new server. You need that file for any encrypted variables, secrets, tokens, etc..