Upgrading container from 14.2.6 to 14.3.0

Hello, I tried to upgrade gitlab from version 13.7.1 to the latest one, upgraded to 14.2.6 and now I can’t upgrade it to 14.3.0. When I’m deploying a new version of docker container, gitlab returns 500 error on every web request. Logs attached below. What am I doing wrong?

 ==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <==
 {"type":"gitaly-ruby","grpc.start_time":"2021-11-16T18:02:10Z","grpc.time_ms":0.354,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5646,"correlation_id":"02a0f0df3323e967ce15e67cc46867b1","time":"2021-11-16T18:02:10.024Z"}
 {"type":"gitaly-ruby","grpc.start_time":"2021-11-16T18:02:10Z","grpc.time_ms":9.08,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5645,"correlation_id":"0e1094fd1462e80d8a631c8114a6c81b","time":"2021-11-16T18:02:10.035Z"}

 ==> /var/log/gitlab/puma/puma_stdout.log <==
 {"timestamp":"2021-11-16T18:02:10.349Z","pid":5635,"message":"PumaWorkerKiller: Consuming 1379.44140625 mb with master and 1 workers."}

 ==> /var/log/gitlab/gitlab-rails/production.log <==
 Started GET "/" for 176.215.114.129 at 2021-11-16 18:02:10 +0000
 Processing by RootController#index as HTML
 Completed 500 Internal Server Error in 8ms (ActiveRecord: 1.0ms | Elasticsearch: 0.0ms | Allocations: 2104)

 ==> /var/log/gitlab/gitlab-rails/production_json.log <==
 {"method":"GET","path":"/","format":"html","controller":"RootController","action":"index","status":500,"time":"2021-11-16T18:02:10.773Z","params":[],"remote_ip":"176.215.114.129","user_id":26,"username":"FMarkov","ua":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36 OPR/80.0.4170.16","correlation_id":"01FMMYKNXPN87VGTF9Q3H8GQDH","meta.caller_id":"RootController#index","meta.remote_ip":"176.215.114.129","meta.feature_category":"projects","meta.client_id":"ip/176.215.114.129","redis_calls":4,"redis_duration_s":0.0010350000000000001,"redis_read_bytes":583,"redis_write_bytes":1482,"redis_cache_calls":1,"redis_cache_duration_s":0.0002,"redis_cache_read_bytes":203,"redis_cache_write_bytes":58,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000835,"redis_shared_state_read_bytes":380,"redis_shared_state_write_bytes":1424,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":3,"db_primary_cached_count":0,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"cpu_s":0.026796,"mem_objects":9219,"mem_bytes":907920,"mem_mallocs":1886,"mem_total_bytes":1276680,"pid":5869,"queue_duration_s":0.016548,"exception.class":"OpenSSL::Cipher::CipherError","exception.message":"","exception.backtrace":["lib/gitlab/current_settings.rb:32:in `method_missing'","lib/gitlab/gon_helper.rb:25:in `add_gon_variables'","app/controllers/application_controller.rb:482:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:21:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.00103,"view_duration_s":0.0,"duration_s":0.00826}

 ==> /var/log/gitlab/gitlab-rails/production.log <==

 OpenSSL::Cipher::CipherError ():

 lib/gitlab/current_settings.rb:32:in `method_missing'
 lib/gitlab/gon_helper.rb:25:in `add_gon_variables'
 app/controllers/application_controller.rb:482:in `set_current_context'
 lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
 lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
 lib/gitlab/middleware/speedscope.rb:13:in `call'
 lib/gitlab/request_profiler/middleware.rb:17:in `call'
 lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
 lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
 lib/gitlab/metrics/web_transaction.rb:21:in `run'
 lib/gitlab/metrics/rack_middleware.rb:16:in `call'
 lib/gitlab/jira/middleware.rb:19:in `call'
 lib/gitlab/middleware/go.rb:20:in `call'
 lib/gitlab/etag_caching/middleware.rb:21:in `call'
 lib/gitlab/middleware/multipart.rb:173:in `call'
 lib/gitlab/middleware/read_only/controller.rb:50:in `call'
 lib/gitlab/middleware/read_only.rb:18:in `call'
 lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
 lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
 lib/gitlab/middleware/basic_health_check.rb:25:in `call'
 lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
 lib/gitlab/middleware/request_context.rb:21:in `call'
 config/initializers/fix_local_cache_middleware.rb:11:in `call'
 lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
 lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
 lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
 lib/gitlab/middleware/release_env.rb:12:in `call'

 ==> /var/log/gitlab/gitlab-workhorse/current <==
 {"content_type":"text/html; charset=utf-8","correlation_id":"01FMMYKNXPN87VGTF9Q3H8GQDH","duration_ms":33,"host":"git.supl.biz","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"176.215.114.129:0","remote_ip":"176.215.114.129","route":"","status":500,"system":"http","time":"2021-11-16T18:02:10Z","ttfb_ms":33,"uri":"/","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36 OPR/80.0.4170.16","written_bytes":2926}

 ==> /var/log/gitlab/nginx/gitlab_access.log <==
 10.100.0.180 - - [16/Nov/2021:18:02:10 +0000] "GET / HTTP/1.0" 500 2926 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36 OPR/80.0.4170.16" -

After a lot of work I found out a reason: corrupted secret file. Everything works okay but there’s a trouble with ApplicationSetting values, as gitlab:doctor:secrets says. These troubles occures when you’re upgrading to gitlab 14.3.0+ or when you’re opening admin/application_settings/reporting page.
Here’s the block of output which identifies my problem:

I, [2021-12-23T08:34:53.965372 #181825]  INFO -- : - ApplicationSetting failures: 1
D, [2021-12-23T08:34:53.965525 #181825] DEBUG -- :   - ApplicationSetting[1]: recaptcha_private_key, recaptcha_site_key

I tried to follow these instructions but they don’t affect recaptcha settings. I’m also confused about this error because I’ve never tried to setup captcha. How can I fix it?

Can you check under Admin → Settings → Reporting → Spam and Anti-bot Protection and see if any of the options are enabled? Also see if recaptcha field have the site id and key, perhaps remove this if they are supplied with data, and then save and also disable the options if enabled.

I know you said you haven’t attempted to set it up, but worth verifying just to make sure in case something else has gone awry.

As I said before, I can’t, it returns 500 error with CipherError in logs.

Have you attempted setting SSL Ciphers in gitlab.rb other than what is the defaults? I’m curious as to why you are receiving these errors. If so, you might want to comment this out so Gitlab can use the defaults. If not, then I don’t know why so many problems. Seems weird.

No, SSL encryption works on another Nginx instance which proxies requests to unencrypted gitlab instance
The reason is that recaptcha keys seems to be encrypted and because of corrupted key file they cannot be decrypted and when you try to get these keys (like in settings) it raises CipherError because decryption fails.

Checked out the fields in database which are responisble for captcha settings, and they’re all NULL. Now I’m confused what causes a decryption error.

The only way I could deal with this problem is to reset application_settings table in database. Made it this way:

1. Started a clear gitlab instance with my gitlab_secrets.json file. Took a backup from clear gitlab instance, found SQL script to restore DB and found a command to set values to application_settings table;
2. Took a backup of my current gitlab installation;
3. Replaced commands in backup which restore application_settings table with clear one;
4. This actions broke encrypted_ci_jwt_signing_key and runners_registration_token_encrypted so I had to clear it and reinstall the runners;
5. Restored all the application settings manually.