Frequent 503 errors

How to Use GitLab
1

Replace this template with your information

Describe your question in as much detail as possible:
When I navigate the web UI of GitLab I am receiving a lot of 503 errors. It seems to be happening mostly when I navigate through projects but the admin section of the root account is fine. After receiving a 503 I can quickly reload and the page loads as normal. But the next time I load another page, I’ll receive a 503.

{"method":"GET","path":"/root/kubernetes-home/activity","format":"html","controller":"ProjectsController","action":"activity","status":503,"time":"2020-07-16T17:54:14.133Z","params":[{"key":"namespace_id","value":"root"},{"key":"id","value":"kubernetes-home"}],"remote_ip":"172.30.230.215","user_id":2,"username":"kbreit","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Firefox/78.0","db_count":20,"db_write_count":0,"db_cached_count":3,"correlation_id":"tDmsjsctK31","meta.user":"kbreit","meta.project":"root/kubernetes-home","meta.root_namespace":"root","meta.caller_id":"ProjectsController#activity","gitaly_calls":1,"gitaly_duration_s":0.007322,"redis_calls":23,"redis_duration_s":0.006402,"redis_read_bytes":998,"redis_write_bytes":3153,"redis_cache_calls":22,"redis_cache_duration_s":0.006134,"redis_cache_read_bytes":787,"redis_cache_write_bytes":2567,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000268,"redis_shared_state_read_bytes":211,"redis_shared_state_write_bytes":586,"queue_duration_s":0.053916,"cpu_s":0.36,"db_duration_s":0.05924,"view_duration_s":0.00072,"duration_s":0.37851}
Completed 503 Service Unavailable in 378ms (Views: 0.7ms | ActiveRecord: 59.2ms | Elasticsearch: 0.0ms | Allocations: 40900)
  • What version are you on (Hint: /help) ? and are you using self-managed or gitlab.com?
    Self managed on Kubernetes, 13.1.2
2

I did find this in the webserver logs.

*** /var/log/gitlab/production.log ***
Started GET "/kbreit/sample" for 172.30.230.215 at 2020-07-17 21:50:35 +0000
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"kbreit", "id"=>"sample"}

Gitlab::Git::CommandError (7:permission denied):
  lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
  lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
  lib/gitlab/git/repository.rb:1039:in `uncached_has_local_branches?'
  lib/gitlab/git/repository.rb:167:in `block in has_local_branches?'
  lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
  lib/gitlab/git/repository.rb:166:in `has_local_branches?'
  app/models/repository.rb:553:in `has_visible_content?'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in _uncached_has_visible_content?'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `_uncached_has_visible_content?'
  lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'
  lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'
  lib/gitlab/repository_cache_adapter.rb:179:in `block (2 levels) in cache_method_output_asymmetrically'
  lib/gitlab/safe_request_store.rb:12:in `fetch'
  lib/gitlab/repository_cache.rb:25:in `fetch'
  lib/gitlab/repository_cache_adapter.rb:178:in `block in cache_method_output_asymmetrically'
  lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
  lib/gitlab/repository_cache_adapter.rb:192:in `block in memoize_method_output'
  lib/gitlab/repository_cache_adapter.rb:201:in `no_repository_fallback'
  lib/gitlab/repository_cache_adapter.rb:191:in `memoize_method_output'
  lib/gitlab/repository_cache_adapter.rb:177:in `cache_method_output_asymmetrically'
  lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in has_visible_content?'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `has_visible_content?'
  app/models/repository.rb:527:in `empty?'
  lib/gitlab/metrics/instrumentation.rb:161:in `block in empty?'
  lib/gitlab/metrics/method_call.rb:36:in `measure'
  lib/gitlab/metrics/instrumentation.rb:161:in `empty?'
  app/models/concerns/has_repository.rb:75:in `empty_repo?'
  app/controllers/projects_controller.rb:429:in `repo_exists?'
  app/controllers/projects_controller.rb:21:in `block in <class:ProjectsController>'
  app/controllers/application_controller.rb:552:in `block in allow_gitaly_ref_name_caching'
  lib/gitlab/gitaly_client.rb:355:in `allow_ref_name_caching'
  app/controllers/application_controller.rb:551:in `allow_gitaly_ref_name_caching'
  app/controllers/application_controller.rb:497:in `set_current_admin'
  lib/gitlab/session.rb:11:in `with_session'
  app/controllers/application_controller.rb:488:in `set_session_storage'
  lib/gitlab/i18n.rb:55:in `with_locale'
  lib/gitlab/i18n.rb:61:in `with_user_locale'
  app/controllers/application_controller.rb:482:in `set_locale'
  lib/gitlab/error_tracking.rb:51:in `with_context'
  app/controllers/application_controller.rb:547:in `sentry_context'
  app/controllers/application_controller.rb:475:in `block in set_current_context'
  lib/gitlab/application_context.rb:52:in `block in use'
  lib/gitlab/application_context.rb:52:in `use'
  lib/gitlab/application_context.rb:20:in `with_context'
  app/controllers/application_controller.rb:468:in `set_current_context'
  lib/gitlab/metrics/elasticsearch_rack_middleware.rb:24:in `call'
  lib/gitlab/metrics/redis_rack_middleware.rb:22:in `call'
  lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
  lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
  lib/gitlab/metrics/transaction.rb:54:in `run'
  lib/gitlab/metrics/rack_middleware.rb:17:in `call'
  lib/gitlab/request_profiler/middleware.rb:17:in `call'
  lib/gitlab/middleware/go.rb:20:in `call'
  lib/gitlab/etag_caching/middleware.rb:13:in `call'
  lib/gitlab/middleware/multipart.rb:125:in `call'
  lib/gitlab/middleware/read_only/controller.rb:51:in `call'
  lib/gitlab/middleware/read_only.rb:18:in `call'
  lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  lib/gitlab/middleware/request_context.rb:23:in `call'
  config/initializers/fix_local_cache_middleware.rb:9:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
  lib/gitlab/middleware/release_env.rb:12:in `call'

Completed 503 Service Unavailable in 52ms (Views: 0.9ms | ActiveRecord: 14.5ms | Elasticsearch: 0.0ms | Allocations: 13525)

*** /var/log/gitlab/exceptions_json.log ***
{"severity":"ERROR","time":"2020-07-17T21:50:36.096Z","correlation_id":"5S6Ij11XcLa","tags.correlation_id":"5S6Ij11XcLa","tags.locale":"en","exception.class":"Gitlab::Git::CommandError","exception.message":"7:permission denied","exception.backtrace":["lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'","lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:1039:in `uncached_has_local_branches?'","lib/gitlab/git/repository.rb:167:in `block in has_local_branches?'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","lib/gitlab/git/repository.rb:166:in `has_local_branches?'","app/models/repository.rb:553:in `has_visible_content?'","lib/gitlab/repository_cache_adapter.rb:84:in `block (2 levels) in cache_method_asymmetrically'","lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'","lib/gitlab/repository_cache_adapter.rb:179:in `block (2 levels) in cache_method_output_asymmetrically'","lib/gitlab/safe_request_store.rb:12:in `fetch'","lib/gitlab/repository_cache.rb:25:in `fetch'","lib/gitlab/repository_cache_adapter.rb:178:in `block in cache_method_output_asymmetrically'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","lib/gitlab/repository_cache_adapter.rb:192:in `block in memoize_method_output'","lib/gitlab/repository_cache_adapter.rb:201:in `no_repository_fallback'","lib/gitlab/repository_cache_adapter.rb:191:in `memoize_method_output'","lib/gitlab/repository_cache_adapter.rb:177:in `cache_method_output_asymmetrically'","lib/gitlab/repository_cache_adapter.rb:83:in `block in cache_method_asymmetrically'","app/models/repository.rb:527:in `empty?'","app/models/concerns/has_repository.rb:75:in `empty_repo?'","app/controllers/projects_controller.rb:429:in `repo_exists?'","app/controllers/projects_controller.rb:21:in `block in \u003cclass:ProjectsController\u003e'","app/controllers/application_controller.rb:552:in `block in allow_gitaly_ref_name_caching'","lib/gitlab/gitaly_client.rb:355:in `allow_ref_name_caching'","app/controllers/application_controller.rb:551:in `allow_gitaly_ref_name_caching'","app/controllers/application_controller.rb:497:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:488:in `set_session_storage'","app/controllers/application_controller.rb:482:in `set_locale'","lib/gitlab/error_tracking.rb:51:in `with_context'","app/controllers/application_controller.rb:547:in `sentry_context'","app/controllers/application_controller.rb:475:in `block in set_current_context'","lib/gitlab/application_context.rb:52:in `block in use'","lib/gitlab/application_context.rb:52:in `use'","lib/gitlab/application_context.rb:20:in `with_context'","app/controllers/application_controller.rb:468:in `set_current_context'"]}
3

It’s possible that at the time of reporting this, GitLab was experiencing an issue or outage of sorts, @kbreit . It’s sort of hard to be sure: status.gitlab.com.

Are you still getting 503 errors? If so, I’ll get you connected with a Support expert asap! Thanks!

4

This is an on-premises deployment. But yes, I am still seeing 503 errors.

5

Bump.