How to Debug Gitaly Trouble

Hi

My 13.5.0-ce source install works mostly fine, however, since some 13.4.x-ce release, saving a wiki page on any project fails with 503. So I’m diving into the logs:

gitlab/logs/production.log:

Started PUT "/bitcetera/tower/-/wikis/home" for [FILTERED] at 2020-10-22 15:52:00 +0200
Processing by Projects::WikisController#update as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "wiki"=>{"last_commit_sha"=>"41f58fcbe7a286535ebded9b39674149a0852fec", "title"=>"[FILTERED]", "format"=>"markdown", "content"=>"[FILTERED]", "message"=>"Update home"}, "namespace_id"=>"bitcetera", "project_id"=>"tower", "id"=>"home"}

Gitlab::Git::CommandError (4:Deadline Exceeded.):
  lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
  lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
  lib/gitlab/git/wiki.rb:83:in `update_page'
  app/models/wiki.rb:180:in `update_page'
  app/models/wiki_page.rb:219:in `block in update'
  app/models/wiki_page.rb:354:in `save'
  app/models/wiki_page.rb:218:in `update'
  app/services/wiki_pages/update_service.rb:9:in `execute'
  app/controllers/concerns/wiki_actions.rb:102:in `update'
  app/controllers/application_controller.rb:493:in `set_current_admin'
  lib/gitlab/session.rb:11:in `with_session'
  app/controllers/application_controller.rb:484:in `set_session_storage'
  lib/gitlab/i18n.rb:73:in `with_locale'
  lib/gitlab/i18n.rb:79:in `with_user_locale'
  app/controllers/application_controller.rb:478:in `set_locale'
  lib/gitlab/error_tracking.rb:52:in `with_context'
  app/controllers/application_controller.rb:543:in `sentry_context'
  app/controllers/application_controller.rb:471:in `block in set_current_context'
  lib/gitlab/application_context.rb:54:in `block in use'
  lib/gitlab/application_context.rb:54:in `use'
  lib/gitlab/application_context.rb:21:in `with_context'
  app/controllers/application_controller.rb:463:in `set_current_context'
  lib/gitlab/request_profiler/middleware.rb:17:in `call'
  lib/gitlab/jira/middleware.rb:19:in `call'
  lib/gitlab/middleware/go.rb:20:in `call'
  lib/gitlab/etag_caching/middleware.rb:13:in `call'
  lib/gitlab/middleware/multipart.rb:218:in `call'
  lib/gitlab/middleware/handle_null_bytes.rb:19:in `call'
  lib/gitlab/middleware/read_only/controller.rb:51:in `call'
  lib/gitlab/middleware/read_only.rb:18:in `call'
  lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  lib/gitlab/middleware/request_context.rb:23:in `call'
  config/initializers/fix_local_cache_middleware.rb:9:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
  lib/gitlab/middleware/release_env.rb:12:in `call'

Completed 503 Service Unavailable in 29166ms (Views: 19.0ms | ActiveRecord: 1.9ms | Elasticsearch: 0.0ms | Allocations: 17280)

gitaly.log:

time=“2020-10-22T15:11:14+02:00” level=info msg=“finished streaming call with code Canceled” correlation_id=98804576-34ba-4b75-8720-ffb9cac3d4b6 error=“rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled” grpc.code=Canceled grpc.meta.auth_version=v2 grpc.meta.client_name=gitlab-web grpc.meta.deadline_type=regular grpc.method=WikiUpdatePage grpc.request.deadline=“2020-10-22T15:11:14+02:00” grpc.request.fullMethod=/gitaly.WikiService/WikiUpdatePage grpc.request.glProjectPath=bitcetera/tower grpc.request.glRepository=wiki-28 grpc.request.repoPath=@hashed/59/e1/59e19706d51d39f66711c2653cd7eb1291c94d9b55eb14bda74ce4dc636d015a.wiki.git grpc.request.repoStorage=default grpc.request.topLevelGroup=@hashed grpc.service=gitaly.WikiService grpc.start_time=“2020-10-22T15:10:44+02:00” grpc.time_ms=29730.195 peer.address=@ pid=21438 span.kind=server system=grpc

Unfortunately, WikiUpdatePage fails with code = Canceled is not very helpful, any idea where to go from here?

Thanks for your help!