Currently installed version 15.11.9 ce.
Gitlab is installed as a package, without docker. Official documentation was used for backups.
A backup + secrets.json was made, moved to a new virtual machine in hetzner for testing.
The backup was deployed without errors, then edited gitlab.rb and updated gitlab to 16.1.2
After the upgrade I started getting 500 errors in the web version and 503 errors in the console
I could only find the information in production.log, if you can give me a hint I will look elsewhere.
part of production.log
ActionView::Template::Error (14:connections to all backends failing.):
22: - if project.creator && use_creator_avatar
23: = render Pajamas::AvatarComponent.new(project.creator, size: 48, alt: ‘’, class: ‘gl-mr-5’)
24: - else
25: = render Pajamas::AvatarComponent.new(project, size: 48, alt: ‘’, class: ‘gl-mr-5’)
26: .project-cell{ class: css_class }
27: .project-details.gl-pr-9.gl-sm-pr-0.gl-w-full.gl-display-flex.gl-flex-direction-column{ data: { qa_selector: ‘project_content’, qa_project_name: project.name } }
28: .gl-display-flex.gl-align-items-center.gl-flex-wrap
lib/gitlab/git/wraps_gitaly_errors.rb:24:in rescue in wrapped_gitaly_errors' lib/gitlab/git/wraps_gitaly_errors.rb:6:in wrapped_gitaly_errors’
lib/gitlab/git/repository.rb:1167:in uncached_has_local_branches?' lib/gitlab/git/repository.rb:190:in block in has_local_branches?’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' lib/gitlab/git/repository.rb:189:in has_local_branches?’
app/models/repository.rb:599:in has_visible_content?' lib/gitlab/repository_cache_adapter.rb:95:in block (2 levels) in cache_method_asymmetrically’
lib/gitlab/repository_cache.rb:44:in fetch_without_caching_false' lib/gitlab/repository_cache_adapter.rb:190:in block (2 levels) in cache_method_output_asymmetrically’
lib/gitlab/safe_request_store.rb:12:in fetch' lib/gitlab/repository_cache.rb:25:in fetch’
lib/gitlab/repository_cache_adapter.rb:189:in block in cache_method_output_asymmetrically' lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize’
lib/gitlab/repository_cache_adapter.rb:203:in block in memoize_method_output' lib/gitlab/repository_cache_adapter.rb:212:in no_repository_fallback’
lib/gitlab/repository_cache_adapter.rb:202:in memoize_method_output' lib/gitlab/repository_cache_adapter.rb:188:in cache_method_output_asymmetrically’
lib/gitlab/repository_cache_adapter.rb:94:in block in cache_method_asymmetrically' app/models/repository.rb:573:in empty?’
app/models/repository.rb:696:in tree' app/models/repository.rb:1084:in file_on_head’
app/models/repository.rb:607:in block in avatar' lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’
app/models/repository.rb:606:in avatar' lib/gitlab/repository_cache_adapter.rb:21:in block (2 levels) in cache_method’
lib/gitlab/repository_cache.rb:25:in fetch' lib/gitlab/repository_cache_adapter.rb:163:in block in cache_method_output’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' lib/gitlab/repository_cache_adapter.rb:203:in block in memoize_method_output’
lib/gitlab/repository_cache_adapter.rb:212:in no_repository_fallback' lib/gitlab/repository_cache_adapter.rb:202:in memoize_method_output’
lib/gitlab/repository_cache_adapter.rb:162:in cache_method_output' lib/gitlab/repository_cache_adapter.rb:20:in block in cache_method’
app/models/project.rb:1761:in avatar_in_git' app/models/project.rb:1765:in avatar_url’
app/models/concerns/avatarable.rb:36:in avatar_url' app/components/pajamas/avatar_component.rb:48:in block in src’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' app/components/pajamas/avatar_component.rb:37:in src’
app/components/pajamas/avatar_component.html.haml:1:in call' app/views/shared/projects/_project.html.haml:25 app/views/shared/projects/_project.html.haml:21 app/views/shared/projects/_list.html.haml:37 app/views/shared/projects/_list.html.haml:35:in each_with_index’
app/views/shared/projects/_list.html.haml:35
app/views/dashboard/projects/_projects.html.haml:1
app/views/dashboard/projects/index.html.haml:13
app/controllers/application_controller.rb:144:in render' app/controllers/dashboard/projects_controller.rb:64:in block in render_projects’
lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls' app/controllers/dashboard/projects_controller.rb:63:in render_projects’
app/controllers/dashboard/projects_controller.rb:23:in block (2 levels) in index' app/controllers/dashboard/projects_controller.rb:21:in index’
app/controllers/root_controller.rb:25:in block in index' lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’
app/controllers/root_controller.rb:23:in index' app/controllers/application_controller.rb:500:in set_current_admin’
lib/gitlab/session.rb:11:in with_session' app/controllers/application_controller.rb:491:in set_session_storage’
lib/gitlab/i18n.rb:107:in with_locale' lib/gitlab/i18n.rb:113:in with_user_locale’
app/controllers/application_controller.rb:482:in set_locale' app/controllers/application_controller.rb:475:in set_current_context’
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call' lib/gitlab/middleware/memory_report.rb:13:in call’
lib/gitlab/middleware/speedscope.rb:13:in call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call’
lib/gitlab/middleware/rails_queue_duration.rb:33:in call' lib/gitlab/metrics/rack_middleware.rb:16:in block in call’
lib/gitlab/metrics/web_transaction.rb:46:in run' lib/gitlab/metrics/rack_middleware.rb:16:in call’
lib/gitlab/jira/middleware.rb:19:in call' lib/gitlab/middleware/go.rb:20:in call’
lib/gitlab/etag_caching/middleware.rb:21:in call' lib/gitlab/middleware/query_analyzer.rb:11:in block in call’
lib/gitlab/database/query_analyzer.rb:37:in within' lib/gitlab/middleware/query_analyzer.rb:11:in call’
lib/gitlab/middleware/multipart.rb:173:in call' lib/gitlab/middleware/read_only/controller.rb:50:in call’
lib/gitlab/middleware/read_only.rb:18:in call' lib/gitlab/middleware/same_site_cookies.rb:27:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in call’
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call' lib/gitlab/middleware/request_context.rb:15:in call’
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/compressed_json.rb:44:in call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call’
lib/gitlab/middleware/sidekiq_web_static.rb:20:in call' lib/gitlab/metrics/requests_rack_middleware.rb:79:in call’
lib/gitlab/middleware/release_env.rb:13:in `call’
ActionView::Template::Error (14:connections to all backends failing.):
22: - if project.creator && use_creator_avatar
23: = render Pajamas::AvatarComponent.new(project.creator, size: 48, alt: ‘’, class: ‘gl-mr-5’)
24: - else
25: = render Pajamas::AvatarComponent.new(project, size: 48, alt: ‘’, class: ‘gl-mr-5’)
26: .project-cell{ class: css_class }
27: .project-details.gl-pr-9.gl-sm-pr-0.gl-w-full.gl-display-flex.gl-flex-direction-column{ data: { qa_selector: ‘project_content’, qa_project_name: project.name } }
28: .gl-display-flex.gl-align-items-center.gl-flex-wrap
lib/gitlab/git/wraps_gitaly_errors.rb:24:in rescue in wrapped_gitaly_errors' lib/gitlab/git/wraps_gitaly_errors.rb:6:in wrapped_gitaly_errors’
lib/gitlab/git/repository.rb:1167:in uncached_has_local_branches?' lib/gitlab/git/repository.rb:190:in block in has_local_branches?’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' lib/gitlab/git/repository.rb:189:in has_local_branches?’
app/models/repository.rb:599:in has_visible_content?' lib/gitlab/repository_cache_adapter.rb:95:in block (2 levels) in cache_method_asymmetrically’
lib/gitlab/repository_cache.rb:44:in fetch_without_caching_false' lib/gitlab/repository_cache_adapter.rb:190:in block (2 levels) in cache_method_output_asymmetrically’
lib/gitlab/safe_request_store.rb:12:in fetch' lib/gitlab/repository_cache.rb:25:in fetch’
lib/gitlab/repository_cache_adapter.rb:189:in block in cache_method_output_asymmetrically' lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize’
lib/gitlab/repository_cache_adapter.rb:203:in block in memoize_method_output' lib/gitlab/repository_cache_adapter.rb:212:in no_repository_fallback’
lib/gitlab/repository_cache_adapter.rb:202:in memoize_method_output' lib/gitlab/repository_cache_adapter.rb:188:in cache_method_output_asymmetrically’
lib/gitlab/repository_cache_adapter.rb:94:in block in cache_method_asymmetrically' app/models/repository.rb:573:in empty?’
app/models/repository.rb:696:in tree' app/models/repository.rb:1084:in file_on_head’
app/models/repository.rb:607:in block in avatar' lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’
app/models/repository.rb:606:in avatar' lib/gitlab/repository_cache_adapter.rb:21:in block (2 levels) in cache_method’
lib/gitlab/repository_cache.rb:25:in fetch' lib/gitlab/repository_cache_adapter.rb:163:in block in cache_method_output’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' lib/gitlab/repository_cache_adapter.rb:203:in block in memoize_method_output’
lib/gitlab/repository_cache_adapter.rb:212:in no_repository_fallback' lib/gitlab/repository_cache_adapter.rb:202:in memoize_method_output’
lib/gitlab/repository_cache_adapter.rb:162:in cache_method_output' lib/gitlab/repository_cache_adapter.rb:20:in block in cache_method’
app/models/project.rb:1761:in avatar_in_git' app/models/project.rb:1765:in avatar_url’
app/models/concerns/avatarable.rb:36:in avatar_url' app/components/pajamas/avatar_component.rb:48:in block in src’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' app/components/pajamas/avatar_component.rb:37:in src’
app/components/pajamas/avatar_component.html.haml:1:in call' app/views/shared/projects/_project.html.haml:25 app/views/shared/projects/_project.html.haml:21 app/views/shared/projects/_list.html.haml:37 app/views/shared/projects/_list.html.haml:35:in each_with_index’
app/views/shared/projects/_list.html.haml:35
app/views/dashboard/projects/_projects.html.haml:1
app/views/dashboard/projects/index.html.haml:13
app/controllers/application_controller.rb:144:in render' app/controllers/dashboard/projects_controller.rb:64:in block in render_projects’
lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls' app/controllers/dashboard/projects_controller.rb:63:in render_projects’
app/controllers/dashboard/projects_controller.rb:23:in block (2 levels) in index' app/controllers/dashboard/projects_controller.rb:21:in index’
app/controllers/root_controller.rb:25:in block in index' lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’
app/controllers/root_controller.rb:23:in index' app/controllers/application_controller.rb:500:in set_current_admin’
lib/gitlab/session.rb:11:in with_session' app/controllers/application_controller.rb:491:in set_session_storage’
lib/gitlab/i18n.rb:107:in with_locale' lib/gitlab/i18n.rb:113:in with_user_locale’
app/controllers/application_controller.rb:482:in set_locale' app/controllers/application_controller.rb:475:in set_current_context’
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call' lib/gitlab/middleware/memory_report.rb:13:in call’
lib/gitlab/middleware/speedscope.rb:13:in call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call’
lib/gitlab/middleware/rails_queue_duration.rb:33:in call' lib/gitlab/metrics/rack_middleware.rb:16:in block in call’
lib/gitlab/metrics/web_transaction.rb:46:in run' lib/gitlab/metrics/rack_middleware.rb:16:in call’
lib/gitlab/jira/middleware.rb:19:in call' lib/gitlab/middleware/go.rb:20:in call’
lib/gitlab/etag_caching/middleware.rb:21:in call' lib/gitlab/middleware/query_analyzer.rb:11:in block in call’
lib/gitlab/database/query_analyzer.rb:37:in within' lib/gitlab/middleware/query_analyzer.rb:11:in call’
lib/gitlab/middleware/multipart.rb:173:in call' lib/gitlab/middleware/read_only/controller.rb:50:in call’
lib/gitlab/middleware/read_only.rb:18:in call' lib/gitlab/middleware/same_site_cookies.rb:27:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in call’
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call' lib/gitlab/middleware/request_context.rb:15:in call’
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/compressed_json.rb:44:in call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call’
lib/gitlab/middleware/sidekiq_web_static.rb:20:in call' lib/gitlab/metrics/requests_rack_middleware.rb:79:in call’
lib/gitlab/middleware/release_env.rb:13:in `call’
ActionView::Template::Error (14:connections to all backends failing.):
22: - if project.creator && use_creator_avatar
23: = render Pajamas::AvatarComponent.new(project.creator, size: 48, alt: ‘’, class: ‘gl-mr-5’)
24: - else
25: = render Pajamas::AvatarComponent.new(project, size: 48, alt: ‘’, class: ‘gl-mr-5’)
26: .project-cell{ class: css_class }
27: .project-details.gl-pr-9.gl-sm-pr-0.gl-w-full.gl-display-flex.gl-flex-direction-column{ data: { qa_selector: ‘project_content’, qa_project_name: project.name } }
28: .gl-display-flex.gl-align-items-center.gl-flex-wrap
lib/gitlab/git/wraps_gitaly_errors.rb:24:in rescue in wrapped_gitaly_errors' lib/gitlab/git/wraps_gitaly_errors.rb:6:in wrapped_gitaly_errors’
lib/gitlab/git/repository.rb:1167:in uncached_has_local_branches?' lib/gitlab/git/repository.rb:190:in block in has_local_branches?’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' lib/gitlab/git/repository.rb:189:in has_local_branches?’
app/models/repository.rb:599:in has_visible_content?' lib/gitlab/repository_cache_adapter.rb:95:in block (2 levels) in cache_method_asymmetrically’
lib/gitlab/repository_cache.rb:44:in fetch_without_caching_false' lib/gitlab/repository_cache_adapter.rb:190:in block (2 levels) in cache_method_output_asymmetrically’
lib/gitlab/safe_request_store.rb:12:in fetch' lib/gitlab/repository_cache.rb:25:in fetch’
lib/gitlab/repository_cache_adapter.rb:189:in block in cache_method_output_asymmetrically' lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize’
lib/gitlab/repository_cache_adapter.rb:203:in block in memoize_method_output' lib/gitlab/repository_cache_adapter.rb:212:in no_repository_fallback’
lib/gitlab/repository_cache_adapter.rb:202:in memoize_method_output' lib/gitlab/repository_cache_adapter.rb:188:in cache_method_output_asymmetrically’
lib/gitlab/repository_cache_adapter.rb:94:in block in cache_method_asymmetrically' app/models/repository.rb:573:in empty?’
app/models/repository.rb:696:in tree' app/models/repository.rb:1084:in file_on_head’
app/models/repository.rb:607:in block in avatar' lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’
app/models/repository.rb:606:in avatar' lib/gitlab/repository_cache_adapter.rb:21:in block (2 levels) in cache_method’
lib/gitlab/repository_cache.rb:25:in fetch' lib/gitlab/repository_cache_adapter.rb:163:in block in cache_method_output’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' lib/gitlab/repository_cache_adapter.rb:203:in block in memoize_method_output’
lib/gitlab/repository_cache_adapter.rb:212:in no_repository_fallback' lib/gitlab/repository_cache_adapter.rb:202:in memoize_method_output’
lib/gitlab/repository_cache_adapter.rb:162:in cache_method_output' lib/gitlab/repository_cache_adapter.rb:20:in block in cache_method’
app/models/project.rb:1761:in avatar_in_git' app/models/project.rb:1765:in avatar_url’
app/models/concerns/avatarable.rb:36:in avatar_url' app/components/pajamas/avatar_component.rb:48:in block in src’
lib/gitlab/utils/strong_memoize.rb:34:in strong_memoize' app/components/pajamas/avatar_component.rb:37:in src’
app/components/pajamas/avatar_component.html.haml:1:in call' app/views/shared/projects/_project.html.haml:25 app/views/shared/projects/_project.html.haml:21 app/views/shared/projects/_list.html.haml:37 app/views/shared/projects/_list.html.haml:35:in each_with_index’
app/views/shared/projects/_list.html.haml:35
app/views/dashboard/projects/_projects.html.haml:1
app/views/dashboard/projects/index.html.haml:13
app/controllers/application_controller.rb:144:in render' app/controllers/dashboard/projects_controller.rb:64:in block in render_projects’
lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls' app/controllers/dashboard/projects_controller.rb:63:in render_projects’
app/controllers/dashboard/projects_controller.rb:23:in block (2 levels) in index' app/controllers/dashboard/projects_controller.rb:21:in index’
app/controllers/root_controller.rb:25:in block in index' lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’
app/controllers/root_controller.rb:23:in index' app/controllers/application_controller.rb:500:in set_current_admin’
lib/gitlab/session.rb:11:in with_session' app/controllers/application_controller.rb:491:in set_session_storage’
lib/gitlab/i18n.rb:107:in with_locale' lib/gitlab/i18n.rb:113:in with_user_locale’
app/controllers/application_controller.rb:482:in set_locale' app/controllers/application_controller.rb:475:in set_current_context’
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call' lib/gitlab/middleware/memory_report.rb:13:in call’
lib/gitlab/middleware/speedscope.rb:13:in call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call’
lib/gitlab/middleware/rails_queue_duration.rb:33:in call' lib/gitlab/metrics/rack_middleware.rb:16:in block in call’
lib/gitlab/metrics/web_transaction.rb:46:in run' lib/gitlab/metrics/rack_middleware.rb:16:in call’
lib/gitlab/jira/middleware.rb:19:in call' lib/gitlab/middleware/go.rb:20:in call’
lib/gitlab/etag_caching/middleware.rb:21:in call' lib/gitlab/middleware/query_analyzer.rb:11:in block in call’
lib/gitlab/database/query_analyzer.rb:37:in within' lib/gitlab/middleware/query_analyzer.rb:11:in call’
lib/gitlab/middleware/multipart.rb:173:in call' lib/gitlab/middleware/read_only/controller.rb:50:in call’
lib/gitlab/middleware/read_only.rb:18:in call' lib/gitlab/middleware/same_site_cookies.rb:27:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in call’
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call' lib/gitlab/middleware/request_context.rb:15:in call’
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/compressed_json.rb:44:in call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call’
lib/gitlab/middleware/sidekiq_web_static.rb:20:in call' lib/gitlab/metrics/requests_rack_middleware.rb:79:in call’
lib/gitlab/middleware/release_env.rb:13:in `call’