Hi!
After upgrading from Gitlab CE 10.7.4 -> 10.8.5 -> 11.0.4 -> 11.1.0 -> 11.1.1, we are seeing issues with unicorn processes that start using 100% CPU for long time, producing slow requests and failing SSH Git access.
We started to see this behavior after upgrading to 11.1.0 last Monday, but upgrading to latest patch didn’t help at all.
We have gitlab.rb set to control memory consumption of unicorn workers, with a timeout of 60 seconds per process and a maximum of 2 workers (it’s a 2 CPU instance).
We are sure it’s not because of the load, because it happens only a few times a day, and the request rate is mostly the same during all working hours.
We have checked and:
- There is no sidekiq job running
- There is no cron job running (neither in the system or in sidekiq)
- There is nothing strange in the logs
- There is no rake task running (like backups or so on)
We run all kind of Gitlab check tasks and everything looks fine. We also tried to update all packages in the system, reboot the instance and even reconfigure Gitlab, without any luck at all. The only way to bring it back to normal is restarting the unicorn process or wait until the timeout is reached in some request (usually takes about 25 minutes in high load to have unicorn auto-restarted because of timeouts).
We also got a backtrace using gdb, and it looks like is busy, but we have no clue about how to solve it:
from /opt/gitlab/embedded/bin/bundle:23:in `<main>'
from /opt/gitlab/embedded/bin/bundle:23:in `load'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:22:in `<top (required)>'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/exe/bundle:30:in `block in <top (required)>'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:18:in `start'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:27:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli.rb:424:in `exec'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:28:in `run'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/bundler-1.16.2/lib/bundler/cli/exec.rb:74:in `load'
from /opt/gitlab/embedded/bin/unicorn:23:in `<top (required)>'
from /opt/gitlab/embedded/bin/unicorn:23:in `load'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:283:in `join'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:519:in `maintain_worker_count'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:658:in `worker_loop'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:562:in `process_client'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:50:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:50:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/urlmap.rb:66:in `block in call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/release_env.rb:10:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `method_missing'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/railtie.rb:194:in `public_send'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/application.rb:165:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:518:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/rack.rb:51:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/sendfile.rb:113:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/lock.rb:17:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/runtime.rb:18:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/methodoverride.rb:22:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/request_id.rb:21:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_context.rb:18:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:20:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:68:in `tagged'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:26:in `tagged'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/tagged_logging.rb:68:in `block in tagged'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:20:in `block in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/railties-4.2.10/lib/rails/rack/logger.rb:38:in `call_app'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:81:in `run_callbacks'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/query_cache.rb:36:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/cookies.rb:560:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/session/abstract/id.rb:220:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/session/abstract/id.rb:225:in `context'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/flash.rb:260:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only.rb:16:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/read_only/controller.rb:38:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/head.rb:13:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/conditionalget.rb:25:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-1.6.10/lib/rack/etag.rb:24:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/batch-loader-1.2.1/lib/batch_loader/middleware.rb:11:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/etag_caching/middleware.rb:11:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/go.rb:17:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/request_profiler/middleware.rb:14:in `call'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/multipart.rb:97:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/route_set.rb:817:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:30:in `serve'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:30:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/journey/router.rb:43:in `block in serve'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/mapper.rb:49:in `serve'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/route_set.rb:43:in `serve'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal.rb:237:in `block in action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal.rb:196:in `dispatch'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionview-4.2.10/lib/action_view/rendering.rb:30:in `process'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/abstract_controller/base.rb:137:in `process'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activerecord-4.2.10/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:164:in `instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:164:in `block in instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/rescue.rb:29:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/abstract_controller/callbacks.rb:19:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:81:in `run_callbacks'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:505:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:497:in `block in around'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:312:in `block in halting'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:441:in `block in make_lambda'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:441:in `instance_exec'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sentry-raven-2.7.2/lib/raven/integrations/rails/controller_transaction.rb:7:in `block in included'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:505:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:497:in `block in around'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:298:in `block in halting_and_conditional'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:505:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:497:in `block in around'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:312:in `block in halting'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:432:in `block in make_lambda'
from /opt/gitlab/embedded/service/gitlab-rails/app/controllers/application_controller.rb:362:in `set_locale'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/i18n.rb:57:in `with_user_locale'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/i18n.rb:51:in `with_locale'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:505:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:117:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/rendering.rb:10:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/abstract_controller/base.rb:198:in `process_action'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
from /opt/gitlab/embedded/service/gitlab-rails/app/controllers/projects/merge_requests_controller.rb:34:in `show'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/actionpack-4.2.10/lib/action_controller/metal/mime_responds.rb:217:in `respond_to'
from /opt/gitlab/embedded/service/gitlab-rails/app/controllers/projects/merge_requests_controller.rb:61:in `block (2 levels) in show'
from /opt/gitlab/embedded/service/gitlab-rails/app/serializers/merge_request_serializer.rb:14:in `represent'
from /opt/gitlab/embedded/service/gitlab-rails/app/serializers/base_serializer.rb:14:in `represent'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/entity.rb:489:in `serializable_hash'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:40:in `serializable_value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `map_entity_exposures'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `each_with_object'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:119:in `block in map_entity_exposures'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:115:in `with_attr_path'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/options.rb:90:in `with_attr_path'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:116:in `block in with_attr_path'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:120:in `block (2 levels) in map_entity_exposures'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:41:in `block in serializable_value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:40:in `serializable_value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `map_entity_exposures'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `each_with_object'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:119:in `block in map_entity_exposures'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:115:in `with_attr_path'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/options.rb:90:in `with_attr_path'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:116:in `block in with_attr_path'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:120:in `block (2 levels) in map_entity_exposures'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:41:in `block in serializable_value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:66:in `serializable_value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:82:in `valid_value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/block_exposure.rb:10:in `value'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/entity.rb:494:in `exec_with_object'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/grape-entity-0.7.1/lib/grape_entity/entity.rb:494:in `instance_exec'
from /opt/gitlab/embedded/service/gitlab-rails/app/serializers/merge_request_widget_entity.rb:98:in `block (2 levels) in <class:MergeRequestWidgetEntity>'
from /opt/gitlab/embedded/service/gitlab-rails/app/presenters/merge_request_presenter.rb:132:in `mentioned_issues_links'
from /opt/gitlab/embedded/lib/ruby/2.4.0/delegate.rb:83:in `method_missing'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/merge_request.rb:791:in `issues_mentioned_but_not_closing'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/reference_extractor.rb:37:in `issues'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/reference_extractor.rb:20:in `references'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/reference_extractor.rb:16:in `references'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/reference_extractor.rb:31:in `html_documents'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:97:in `cache_collection_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:97:in `each'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:98:in `block in cache_collection_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:28:in `render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:138:in `cacheless_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/influx_db.rb:98:in `measure'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:139:in `block in cacheless_render'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/renderer.rb:108:in `render_result'
from /opt/gitlab/embedded/service/gitlab-rails/lib/banzai/pipeline/base_pipeline.rb:21:in `block (2 levels) in singleton class'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:113:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:169:in `instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:115:in `block in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:115:in `inject'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:115:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:116:in `block (2 levels) in call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:130:in `perform_filter'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:169:in `instrument'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline.rb:131:in `block in perform_filter'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline/filter.rb:127:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/html-pipeline-2.8.3/lib/html/pipeline/sanitization_filter.rb:120:in `call'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:57:in `node!'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:158:in `node!'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:245:in `traverse'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:239:in `traverse'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:159:in `block in node!'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:219:in `transform_node!'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:219:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:224:in `block in transform_node!'
from /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/sanitize-4.6.5/lib/sanitize.rb:224:in `node_name'
any ideas?
