500 Server Error after upgrading gitlab from 13.8 to 14.4

florencebib · November 10, 2021, 12:04pm

I got an error after upgrading gitlab from version 13.8.8 to 14.4.4

here is the output from production.log after running

gitlab-ctl tail

Started GET “/” for at 2021-11-10 19:25:53 +0800
Processing by RootController#index as HTML
Redirected to http://example.org/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 571ms (ActiveRecord: 6.5ms | Elasticsearch: 0.0ms | Allocations: 260767)
Started GET “/” for 210.19.7.53 at 2021-11-10 19:25:58 +0800
Processing by RootController#index as HTML
Completed 500 Internal Server Error in 221ms (ActiveRecord: 120.7ms | Elasticsearch: 0.0ms | Allocations: 42994)

ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation “project_topics” does not exist
LINE 8: WHERE a.attrelid = ‘“project_topics”’::regclass
^
):

lib/gitlab/database/schema_cache_with_renamed_table.rb:25:in columns' lib/gitlab/database/schema_cache_with_renamed_table.rb:29:in columns_hash’
app/controllers/concerns/renders_member_access.rb:14:in preload_max_member_access_for_collection' app/controllers/concerns/renders_projects_list.rb:5:in prepare_projects_for_rendering’
app/controllers/dashboard/projects_controller.rb:79:in load_projects' ee/app/controllers/ee/dashboard/projects_controller.rb:40:in load_projects’
app/controllers/dashboard/projects_controller.rb:57:in projects' app/controllers/root_controller.rb:24:in block in index’
lib/gitlab/gitaly_client.rb:325:in allow_n_plus_1_calls' app/controllers/root_controller.rb:23:in index’
ee/lib/gitlab/ip_address_state.rb:10:in with' ee/app/controllers/ee/application_controller.rb:44:in set_current_ip_address’
app/controllers/application_controller.rb:503:in set_current_admin' lib/gitlab/session.rb:11:in with_session’
app/controllers/application_controller.rb:494:in set_session_storage' lib/gitlab/i18n.rb:105:in with_locale’
lib/gitlab/i18n.rb:111:in with_user_locale' app/controllers/application_controller.rb:488:in set_locale’
app/controllers/application_controller.rb:482:in set_current_context' lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call’
lib/gitlab/middleware/rails_queue_duration.rb:33:in call' lib/gitlab/middleware/speedscope.rb:13:in call’
lib/gitlab/request_profiler/middleware.rb:17:in call' lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call’
lib/gitlab/metrics/rack_middleware.rb:16:in block in call' lib/gitlab/metrics/web_transaction.rb:21:in run’
lib/gitlab/metrics/rack_middleware.rb:16:in call' lib/gitlab/jira/middleware.rb:19:in call’
lib/gitlab/middleware/go.rb:20:in call' lib/gitlab/etag_caching/middleware.rb:21:in call’
lib/gitlab/middleware/multipart.rb:173:in call' lib/gitlab/middleware/read_only/controller.rb:50:in call’
lib/gitlab/middleware/read_only.rb:18:in call' lib/gitlab/middleware/same_site_cookies.rb:27:in call’
lib/gitlab/middleware/handle_malformed_strings.rb:21:in call' lib/gitlab/middleware/basic_health_check.rb:25:in call’
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call' lib/gitlab/middleware/request_context.rb:21:in call’
config/initializers/fix_local_cache_middleware.rb:11:in call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call’
lib/gitlab/middleware/sidekiq_web_static.rb:20:in call' lib/gitlab/metrics/requests_rack_middleware.rb:75:in call’
lib/gitlab/middleware/release_env.rb:12:in `call’
Started GET “/favicon.ico” for 210.19.7.53 at 2021-11-10 19:25:59 +0800
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:26:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 20ms (Views: 7.1ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 2366)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:26:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 24ms (Views: 8.1ms | ActiveRecord: 2.7ms | Elasticsearch: 0.0ms | Allocations: 3420)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:26:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 760)
Started POST “/api/jsonws/invoke” for 45.146.164.110 at 2021-11-10 19:26:39 +0800
Started GET “/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php” for 45.146.164.110 at 2021-11-10 19:26:39 +0800

NoMethodError (undefined method `throttle_unauthenticated_api_enabled’ for #ApplicationSetting:0x00007fb8bafaef40
Did you mean? throttle_unauthenticated_enabled
throttle_unauthenticated_enabled?
throttle_unauthenticated_enabled=
throttle_unauthenticated_enabled_was
throttle_unauthenticated_packages_api_enabled
throttle_authenticated_api_enabled
throttle_authenticated_api_enabled?
throttle_authenticated_api_enabled=
throttle_authenticated_api_enabled_was
throttle_authenticated_web_enabled
throttle_authenticated_web_enabled?
throttle_authenticated_web_enabled=
throttle_authenticated_api_enabled_change
throttle_authenticated_web_enabled_was):

lib/gitlab/rack_attack/request.rb:76:in throttle_unauthenticated_api?' lib/gitlab/rack_attack/request.rb:67:in throttle?’
lib/gitlab/rack_attack.rb:90:in block (2 levels) in configure_throttles' lib/gitlab/middleware/read_only/controller.rb:50:in call’
lib/gitlab/middleware/read_only.rb:18:in call' lib/gitlab/middleware/same_site_cookies.rb:27:in call’
lib/gitlab/middleware/handle_malformed_strings.rb:21:in call' lib/gitlab/middleware/basic_health_check.rb:25:in call’
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call' lib/gitlab/middleware/request_context.rb:21:in call’
config/initializers/fix_local_cache_middleware.rb:11:in call' lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call’
lib/gitlab/middleware/sidekiq_web_static.rb:20:in call' lib/gitlab/metrics/requests_rack_middleware.rb:75:in call’
lib/gitlab/middleware/release_env.rb:12:in `call’
Processing by ApplicationController#route_not_found as /
Parameters: {“unmatched_route”=>“vendor/phpunit/phpunit/src/Util/PHP/eval-stdin”}
Redirected to https://47.254.198.232/users/sign_in
Completed 302 Found in 8ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 2245)
Started POST “/mifs/.;/services/LogService” for 45.146.164.110 at 2021-11-10 19:26:46 +0800
Started GET “/wp-content/plugins/wp-file-manager/readme.txt” for 45.146.164.110 at 2021-11-10 19:26:46 +0800

ActionController::RoutingError (No route matches [POST] “/mifs/.;/services/LogService”):

lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
lib/gitlab/middleware/request_context.rb:21:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:75:in call' lib/gitlab/middleware/release_env.rb:12:in call’
Processing by ApplicationController#route_not_found as TEXT
Parameters: {“unmatched_route”=>“wp-content/plugins/wp-file-manager/readme”}
Redirected to https://47.254.198.232/users/sign_in
Completed 302 Found in 5ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 2231)
Started GET “/console/” for 45.146.164.110 at 2021-11-10 19:26:50 +0800
Processing by ApplicationController#route_not_found as HTML
Parameters: {“unmatched_route”=>“console”}
Redirected to https://47.254.198.232/users/sign_in
Completed 302 Found in 10ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 4212)
Started GET “/” for 45.146.164.110 at 2021-11-10 19:26:50 +0800
Processing by RootController#index as HTML
Parameters: {“root”=>{}}
Redirected to https://47.254.198.232/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 10ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 3991)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:26:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 765)
Started POST “/Autodiscover/Autodiscover.xml” for 45.146.164.110 at 2021-11-10 19:26:53 +0800
Processing by ApplicationController#route_not_found as HTML
Parameters: {“namespace_id”=>“Autodiscover”, “project_id”=>“Autodiscover.xml”}
Can’t verify CSRF token authenticity.
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 301)

ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):

lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call' lib/gitlab/middleware/rails_queue_duration.rb:33:in call’
lib/gitlab/middleware/speedscope.rb:13:in call' lib/gitlab/request_profiler/middleware.rb:17:in call’
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call' lib/gitlab/metrics/rack_middleware.rb:16:in block in call’
lib/gitlab/metrics/web_transaction.rb:21:in run' lib/gitlab/metrics/rack_middleware.rb:16:in call’
lib/gitlab/jira/middleware.rb:19:in call' lib/gitlab/middleware/go.rb:20:in call’
lib/gitlab/etag_caching/middleware.rb:21:in call' lib/gitlab/middleware/multipart.rb:173:in call’
lib/gitlab/middleware/read_only/controller.rb:50:in call' lib/gitlab/middleware/read_only.rb:18:in call’
lib/gitlab/middleware/same_site_cookies.rb:27:in call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
lib/gitlab/middleware/request_context.rb:21:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:75:in call' lib/gitlab/middleware/release_env.rb:12:in call’
Started GET “/” for 45.146.164.110 at 2021-11-10 19:26:54 +0800
Processing by RootController#index as HTML
Redirected to https://47.254.198.232/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 10ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 3984)
Started POST “/vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php” for 45.146.164.110 at 2021-11-10 19:26:55 +0800
Processing by ApplicationController#route_not_found as /
Parameters: {"<?"=>"md5(\"phpunit\")?>", “namespace_id”=>“vendor/phpunit/phpunit/src/Util”, “project_id”=>“PHP”, “all”=>“eval-stdin”}
Can’t verify CSRF token authenticity.
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 283)

ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):

lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call' lib/gitlab/middleware/rails_queue_duration.rb:33:in call’
lib/gitlab/middleware/speedscope.rb:13:in call' lib/gitlab/request_profiler/middleware.rb:17:in call’
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call' lib/gitlab/metrics/rack_middleware.rb:16:in block in call’
lib/gitlab/metrics/web_transaction.rb:21:in run' lib/gitlab/metrics/rack_middleware.rb:16:in call’
lib/gitlab/jira/middleware.rb:19:in call' lib/gitlab/middleware/go.rb:20:in call’
lib/gitlab/etag_caching/middleware.rb:21:in call' lib/gitlab/middleware/multipart.rb:173:in call’
lib/gitlab/middleware/read_only/controller.rb:50:in call' lib/gitlab/middleware/read_only.rb:18:in call’
lib/gitlab/middleware/same_site_cookies.rb:27:in call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
lib/gitlab/middleware/request_context.rb:21:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:75:in call' lib/gitlab/middleware/release_env.rb:12:in call’
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:27:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 984)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:27:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 5ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 765)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:27:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 985)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:27:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 5ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:28:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/users/sign_in” for 62.76.41.46 at 2021-11-10 19:28:12 +0800
Processing by SessionsController#new as HTML
Rendered layout layouts/devise.html.haml (Duration: 351.1ms | Allocations: 195795)
Completed 200 OK in 516ms (Views: 350.7ms | ActiveRecord: 61.0ms | Elasticsearch: 0.0ms | Allocations: 250241)
Started GET “/users/sign_in” for 109.237.96.124 at 2021-11-10 19:28:18 +0800
Processing by SessionsController#new as HTML
Rendered layout layouts/devise.html.haml (Duration: 21.6ms | Allocations: 8106)
Completed 200 OK in 38ms (Views: 21.4ms | ActiveRecord: 1.7ms | Elasticsearch: 0.0ms | Allocations: 13371)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:28:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 766)
Started GET “/users/sign_in” for 109.237.96.124 at 2021-11-10 19:28:22 +0800
Processing by SessionsController#new as HTML
Rendered layout layouts/devise.html.haml (Duration: 274.0ms | Allocations: 185479)
Completed 200 OK in 304ms (Views: 279.4ms | ActiveRecord: 2.1ms | Elasticsearch: 0.0ms | Allocations: 198057)
Started GET “/users/sign_in” for 45.8.127.3 at 2021-11-10 19:28:25 +0800
Processing by SessionsController#new as HTML
Rendered layout layouts/devise.html.haml (Duration: 21.0ms | Allocations: 8080)
Completed 200 OK in 37ms (Views: 21.0ms | ActiveRecord: 1.5ms | Elasticsearch: 0.0ms | Allocations: 13319)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:28:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 984)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:28:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 9ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 765)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:29:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:29:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:29:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:29:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 14ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1117)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:30:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:30:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:30:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:30:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 983)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:31:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:31:58 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 1509ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1046)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:33:05 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 2784ms (Views: 189.0ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1196)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:33:23 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 2599ms (Views: 291.1ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 824)
Started GET “/users/sign_in” for 51.89.237.81 at 2021-11-10 19:33:34 +0800
Processing by SessionsController#new as /
Rendered layout layouts/devise.html.haml (Duration: 2188.3ms | Allocations: 7857)
Completed 200 OK in 3689ms (Views: 2392.2ms | ActiveRecord: 185.9ms | Elasticsearch: 0.0ms | Allocations: 11792)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:33:39 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 1199ms (Views: 97.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1162)
Started POST “/uploads/user/authorize” for 51.89.237.81 at 2021-11-10 19:34:31 +0800
Processing by UploadsController#authorize as /
Parameters: {“model”=>“user”}
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:34:35 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 391ms (Views: 1.0ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 983)

ActiveRecord::RecordNotFound (Couldn’t find User without an ID):
app/controllers/uploads_controller.rb:40:in find_model' app/controllers/concerns/uploads_actions.rb:154:in block in model’
lib/gitlab/utils/strong_memoize.rb:30:in strong_memoize' app/controllers/concerns/uploads_actions.rb:154:in model’
app/controllers/uploads_controller.rb:71:in authorize_create_access!' ee/lib/gitlab/ip_address_state.rb:10:in with’
ee/app/controllers/ee/application_controller.rb:44:in set_current_ip_address' app/controllers/application_controller.rb:503:in set_current_admin’
lib/gitlab/session.rb:11:in with_session' app/controllers/application_controller.rb:494:in set_session_storage’
lib/gitlab/i18n.rb:105:in with_locale' lib/gitlab/i18n.rb:111:in with_user_locale’
app/controllers/application_controller.rb:488:in set_locale' app/controllers/application_controller.rb:482:in set_current_context’
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call' lib/gitlab/middleware/rails_queue_duration.rb:33:in call’
lib/gitlab/middleware/speedscope.rb:13:in call' lib/gitlab/request_profiler/middleware.rb:17:in call’
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call' lib/gitlab/metrics/rack_middleware.rb:16:in block in call’
lib/gitlab/metrics/web_transaction.rb:21:in run' lib/gitlab/metrics/rack_middleware.rb:16:in call’
lib/gitlab/jira/middleware.rb:19:in call' lib/gitlab/middleware/go.rb:20:in call’
lib/gitlab/etag_caching/middleware.rb:21:in call' lib/gitlab/middleware/multipart.rb:173:in call’
lib/gitlab/middleware/read_only/controller.rb:50:in call' lib/gitlab/middleware/read_only.rb:18:in call’
lib/gitlab/middleware/same_site_cookies.rb:27:in call' lib/gitlab/middleware/handle_malformed_strings.rb:21:in call’
lib/gitlab/middleware/basic_health_check.rb:25:in call' lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
lib/gitlab/middleware/request_context.rb:21:in call' config/initializers/fix_local_cache_middleware.rb:11:in call’
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call' lib/gitlab/middleware/sidekiq_web_static.rb:20:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:75:in call' lib/gitlab/middleware/release_env.rb:12:in call’

Rendered layout layouts/errors.html.haml (Duration: 2306.5ms | Allocations: 16102)
Completed 404 Not Found in 6296ms (Views: 3422.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 20685)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:34:41 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 14ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 983)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:34:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 99ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:35:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:35:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 777)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:35:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:36:41 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 397ms (Views: 0.9ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:36:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:37:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:37:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:37:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:37:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 9ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1024)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:38:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:38:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:38:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:38:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 9ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:39:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:39:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 10ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:39:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:39:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:40:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 5ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:40:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 9ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 983)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:40:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:40:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 12ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:41:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:41:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 8ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:41:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 12ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:41:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 5ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:42:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:42:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:42:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:42:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:43:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:43:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:43:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:43:52 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:44:07 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:44:22 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 6ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 764)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:44:37 +0800
Processing by MetricsController#index as HTML
Completed 200 OK in 7ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1025)
Started GET “/-/metrics” for 127.0.0.1 at 2021-11-10 19:44:52 +0800
Processing by MetricsController#index as HTML

iwalker · November 10, 2021, 12:21pm

As per the upgrade docs: Upgrading GitLab | GitLab

you cannot upgrade from 13.8.8 to 14.4 - you have to follow the upgrade path and upgrade to each version in between 13.8.8 and 14.4 as shown in the upgrade path.

13.8.8 → latest 13.12.Z → latest 14.0.Z → latest 14.Y.Z

so from 13.8.8 you need to go to latest 13.12.x release, then latest 14.0.x release, and then to 14.4.x. You need to restore your server and start again properly.

florencebib · November 12, 2021, 3:53pm

Thank you very much for your reply. I rolled back to version 13.8.8 again and now I upgraded to 13.10.3 first. The error 500 has gone. However, now I face this error : ERROR: relation “services” does not exist after upgrade to 14.2.0. I found some solutions that need to revert back the data migration to rename the table name from integrations back to services, but I don’t know how to do that? Could anyone advise?

iwalker · November 12, 2021, 4:20pm

If you rolled back using a backup, then the database should have also been rolled back. Unless you haven’t completely rolled back the server? But you would need to explain in more detail of what you exactly did.

Normally under /var/opt/gitlab/backups you should find the backup made before you attempted the upgrade from 13.8.8. However, since you have tried this multiple times, you might need to find the exact backup that was taken before you started any of the upgrades. You would need to follow Gitlab’s backup/restore documents to ensure restoring it properly.

Ideally, you want to remove Gitlab from that server, install Gitlab 13.8.8, restore /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json and the appropriate backup before you started any of your upgrades. That would get you to a sane Gitlab 13.8.8 which you can then properly start again without having to hack your partially upgraded database that hasn’t been restored properly from your apparent rollback.

florencebib · November 12, 2021, 4:56pm

Thanks @iwalker for your reply. Yes, I could find the backup at the path you mentioned. However, when i saw this Restoring from backup after a failed upgrade | GitLab it is recommended that

In some cases after a failed upgrade, the fastest solution is to roll back to the previous version you were using. We recommend this path because the failed upgrade might have made database changes that cannot be readily reverted.

Hence, what I have done is revert back to the old one first by installing back the 13.8.8 and it works fine except for the error instead of restoring the backup. I further found out that the table has been renamed from “Services” to “Integrations” db/post_migrate/20210621223242_finalize_rename_services_to_integrations.rb · master · GitLab.org / GitLab · GitLab

So I try my luck to revert it back to services by running the following command in PostgreSQL

sudo gitlab-psql
You will see the psql command line started. Then type the command:
ALTER TABLE Integrations RENAME TO Services;
After that \q to exit
After that run : sudo gitlab-ctl reconfigure
Problem solved.

Hopefully, it is resolved and no more new/other error arises.