500 error when updating application settings

tdpu · April 26, 2021, 1:54am

I’ve migrated my repository to a new server. The original migration was (broadly):

gitlab-ce=12.2.4 on Debian 9.13 ---> gitlab-ce=12.2.4 on Debian 10

After the original migration, everything appeared to be working.

I then did the following upgrade path:
12.2.4-ce.0 => 12.10.14-ce.0 => 13.0.14-ce.0 => 13.1.11-ce.0 => 13.10.3-ce.0 => 13.11.1-ce.0

At each step, I created a backup using
sudo gitlab-backup create

After testing some things out, I noticed that every time I try to change a configuration setting, I get a 500 error.

I realized that I missed copying over my gitlab-secrets.json file prior to restoring the backup on the new server. I copied it over after the subsequent upgrades.

My sudo gitlab-ctl tail gives:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started PATCH "/admin/application_settings/general" for 10.210.10.7 at 2021-04-25 10:52:32 -0700
Processing by Admin::ApplicationSettingsController#general as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "application_setting"=>{"signup_enabled"=>"0", "require_admin_approval_after_user_signup"=>"0", "send_user_confirmation_email"=>"0", "new_user_signups_cap"=>"", "minimum_password_length"=>"[FILTERED]", "domain_allowlist_raw"=>"", "domain_denylist_enabled"=>"0", "domain_denylist_raw"=>"", "email_restrictions_enabled"=>"0", "email_restrictions"=>"", "after_sign_up_text"=>""}, "denylist_type"=>"raw"}
Completed 500 Internal Server Error in 21ms (ActiveRecord: 3.3ms | Elasticsearch: 0.0ms | Allocations: 11223)==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"PATCH","path":"/admin/application_settings/general","format":"html","controller":"Admin::ApplicationSettingsController","action":"general","status":500,"time":"2021-04-25T17:52:32.125Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"patch"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"application_setting","value":{"signup_enabled":"0","require_admin_approval_after_user_signup":"0","send_user_confirmation_email":"0","new_user_signups_cap":"","minimum_password_length":"[FILTERED]","domain_allowlist_raw":"","domain_denylist_enabled":"0","domain_denylist_raw":"","email_restrictions_enabled":"0","email_restrictions":"","after_sign_up_text":""}},{"key":"denylist_type","value":"raw"}],"remote_ip":"10.210.10.7","user_id":1,"username":"root","ua":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0","correlation_id":"01F452JNTVRWKGR8EK0KS1HG7X","meta.user":"root","meta.caller_id":"Admin::ApplicationSettingsController#general","meta.remote_ip":"10.210.10.7","meta.feature_category":"not_owned","meta.client_id":"user/1","redis_calls":1,"redis_duration_s":0.000255,"redis_read_bytes":181,"redis_write_bytes":700,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000255,"redis_shared_state_read_bytes":181,"redis_shared_state_write_bytes":700,"db_count":5,"db_write_count":0,"db_cached_count":1,"cpu_s":0.026488,"mem_objects":14447,"mem_bytes":1652715,"mem_mallocs":4294,"exception.class":"OpenSSL::Cipher::CipherError","exception.message":"","exception.backtrace":["lib/gitlab/database.rb:342:in `block in transaction'","lib/gitlab/database.rb:341:in `transaction'","app/services/application_settings/update_service.rb:50:in `update_settings'","app/services/application_settings/update_service.rb:12:in `execute'","app/controllers/admin/application_settings_controller.rb:258:in `perform_update'","app/controllers/admin/application_settings_controller.rb:46:in `block (2 levels) in <class:ApplicationSettingsController>'","app/controllers/application_controller.rb:485:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:476:in `set_session_storage'","lib/gitlab/i18n.rb:73:in `with_locale'","lib/gitlab/i18n.rb:79:in `with_user_locale'","app/controllers/application_controller.rb:470:in `set_locale'","app/controllers/application_controller.rb:463:in `block in set_current_context'","lib/gitlab/application_context.rb:70:in `block in use'","lib/gitlab/application_context.rb:70:in `use'","lib/gitlab/application_context.rb:27:in `with_context'","app/controllers/application_controller.rb:454:in `set_current_context'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"db_duration_s":0.00327,"view_duration_s":0.0,"duration_s":0.02104}==> /var/log/gitlab/gitlab-rails/production.log <==OpenSSL::Cipher::CipherError ():lib/gitlab/database.rb:342:in `block in transaction'
lib/gitlab/database.rb:341:in `transaction'
app/services/application_settings/update_service.rb:50:in `update_settings'
app/services/application_settings/update_service.rb:12:in `execute'
app/controllers/admin/application_settings_controller.rb:258:in `perform_update'
app/controllers/admin/application_settings_controller.rb:46:in `block (2 levels) in <class:ApplicationSettingsController>'
app/controllers/application_controller.rb:485:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:476:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:470:in `set_locale'
app/controllers/application_controller.rb:463:in `block in set_current_context'
lib/gitlab/application_context.rb:70:in `block in use'
lib/gitlab/application_context.rb:70:in `use'
lib/gitlab/application_context.rb:27:in `with_context'
app/controllers/application_controller.rb:454:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:172:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/html; charset=utf-8","correlation_id":"01F452JNTVRWKGR8EK0KS1HG7X","duration_ms":36,"host":"yvr-gitlab-01.coanda.local","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://yvr-gitlab-01.coanda.local/admin/application_settings/general","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2021-04-25T10:52:32-07:00","ttfb_ms":36,"uri":"/admin/application_settings/general","user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0","written_bytes":2926}==> /var/log/gitlab/nginx/gitlab_access.log <==
10.210.10.7 - - [25/Apr/2021:10:52:32 -0700] "POST /admin/application_settings/general HTTP/2.0" 500 2926 "https://yvr-gitlab-01.coanda.local/admin/application_settings/general" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0" -==> /var/log/gitlab/crond/current <==
2021-04-25_17:52:32.25932 time="2021-04-25T10:52:32-07:00" level=error msg="unknown flag `no-auto'"
2021-04-25_17:52:33.26872 time="2021-04-25T10:52:33-07:00" level=error msg="unknown flag `no-auto'"==> /var/log/gitlab/redis/current <==
2021-04-25_17:52:34.03885 29805:M 25 Apr 2021 10:52:34.038 * 10 changes in 300 seconds. Saving...
2021-04-25_17:52:34.03972 29805:M 25 Apr 2021 10:52:34.039 * Background saving started by pid 3352
2021-04-25_17:52:34.05397 3352:C 25 Apr 2021 10:52:34.053 * DB saved on disk
2021-04-25_17:52:34.05439 3352:C 25 Apr 2021 10:52:34.054 * RDB: 2 MB of memory used by copy-on-write
2021-04-25_17:52:34.14029 29805:M 25 Apr 2021 10:52:34.140 * Background saving terminated with success==> /var/log/gitlab/crond/current <==
2021-04-25_17:52:34.27816 time="2021-04-25T10:52:34-07:00" level=error msg="unknown flag `no-auto'"
^CTraceback (most recent call last):
	5: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
	4: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
	3: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
	2: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
	1: from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:584:in `tail'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:584:in `system': Interrupt

and sudo gitlab-rake gitlab:doctor:secrets gives:

I, [2021-04-25T17:56:02.333056 #6764]  INFO -- : Checking encrypted values in the database
I, [2021-04-25T17:56:08.865717 #6764]  INFO -- : - ApplicationSetting failures: 1
I, [2021-04-25T17:56:08.930602 #6764]  INFO -- : - User failures: 0
I, [2021-04-25T17:56:08.951621 #6764]  INFO -- : - Ci::InstanceVariable failures: 0
I, [2021-04-25T17:56:08.965009 #6764]  INFO -- : - Clusters::Platforms::Kubernetes failures: 0
I, [2021-04-25T17:56:08.984972 #6764]  INFO -- : - Snippet failures: 0
I, [2021-04-25T17:56:08.988139 #6764]  INFO -- : - PersonalSnippet failures: 0
I, [2021-04-25T17:56:08.991007 #6764]  INFO -- : - ProjectSnippet failures: 0
I, [2021-04-25T17:56:09.011290 #6764]  INFO -- : - Ci::PipelineScheduleVariable failures: 0
I, [2021-04-25T17:56:09.023729 #6764]  INFO -- : - Ci::GroupVariable failures: 0
I, [2021-04-25T17:56:09.035764 #6764]  INFO -- : - Clusters::Applications::Helm failures: 0
I, [2021-04-25T17:56:09.047791 #6764]  INFO -- : - Clusters::Applications::Prometheus failures: 0
I, [2021-04-25T17:56:09.061583 #6764]  INFO -- : - Ci::Variable failures: 0
I, [2021-04-25T17:56:09.075104 #6764]  INFO -- : - AlertManagement::HttpIntegration failures: 0
I, [2021-04-25T17:56:09.087117 #6764]  INFO -- : - Alerting::ProjectAlertingSetting failures: 0
I, [2021-04-25T17:56:09.099585 #6764]  INFO -- : - Atlassian::Identity failures: 0
I, [2021-04-25T17:56:09.111885 #6764]  INFO -- : - BulkImports::Configuration failures: 0
I, [2021-04-25T17:56:09.125513 #6764]  INFO -- : - Ci::JobVariable failures: 0
I, [2021-04-25T17:56:09.139035 #6764]  INFO -- : - Ci::PipelineVariable failures: 0
I, [2021-04-25T17:56:09.152957 #6764]  INFO -- : - Clusters::KubernetesNamespace failures: 0
I, [2021-04-25T17:56:09.166493 #6764]  INFO -- : - Clusters::Providers::Aws failures: 0
I, [2021-04-25T17:56:09.181419 #6764]  INFO -- : - Clusters::Providers::Gcp failures: 0
I, [2021-04-25T17:56:09.196609 #6764]  INFO -- : - ErrorTracking::ProjectErrorTrackingSetting failures: 0
I, [2021-04-25T17:56:09.208571 #6764]  INFO -- : - GrafanaIntegration failures: 0
I, [2021-04-25T17:56:09.221527 #6764]  INFO -- : - WebHook failures: 0
I, [2021-04-25T17:56:09.224791 #6764]  INFO -- : - ProjectHook failures: 0
I, [2021-04-25T17:56:09.226895 #6764]  INFO -- : - ServiceHook failures: 0
I, [2021-04-25T17:56:09.228855 #6764]  INFO -- : - SystemHook failures: 0
I, [2021-04-25T17:56:09.240051 #6764]  INFO -- : - IncidentManagement::ProjectIncidentManagementSetting failures: 0
I, [2021-04-25T17:56:09.251475 #6764]  INFO -- : - JiraConnectInstallation failures: 0
I, [2021-04-25T17:56:09.264890 #6764]  INFO -- : - Packages::Debian::GroupDistribution failures: 0
I, [2021-04-25T17:56:09.281007 #6764]  INFO -- : - Packages::Debian::ProjectDistribution failures: 0
I, [2021-04-25T17:56:09.293700 #6764]  INFO -- : - PagesDomain failures: 0
I, [2021-04-25T17:56:09.306748 #6764]  INFO -- : - PagesDomainAcmeOrder failures: 0
I, [2021-04-25T17:56:09.319672 #6764]  INFO -- : - ProjectImportData failures: 0
I, [2021-04-25T17:56:09.349154 #6764]  INFO -- : - IssueTrackerData failures: 1
I, [2021-04-25T17:56:09.362360 #6764]  INFO -- : - JiraTrackerData failures: 0
I, [2021-04-25T17:56:09.373901 #6764]  INFO -- : - OpenProjectTrackerData failures: 0
I, [2021-04-25T17:56:09.389506 #6764]  INFO -- : - RemoteMirror failures: 0
I, [2021-04-25T17:56:09.401839 #6764]  INFO -- : - Serverless::DomainCluster failures: 0
I, [2021-04-25T17:56:09.403931 #6764]  INFO -- : - Gitlab::BackgroundMigration::BackfillJiraTrackerDeploymentType2::JiraTrackerDataTemp failures: 0
I, [2021-04-25T17:56:09.414455 #6764]  INFO -- : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::IssueTrackerData failures: 1
I, [2021-04-25T17:56:09.415949 #6764]  INFO -- : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData failures: 0
I, [2021-04-25T17:56:10.169569 #6764]  INFO -- : - Group failures: 0
I, [2021-04-25T17:56:10.276471 #6764]  INFO -- : - Project failures: 1
I, [2021-04-25T17:56:10.311230 #6764]  INFO -- : - Ci::Runner failures: 0
I, [2021-04-25T17:56:10.504856 #6764]  INFO -- : - Ci::Build failures: 0
I, [2021-04-25T17:56:10.544044 #6764]  INFO -- : - DeployToken failures: 3
I, [2021-04-25T17:56:10.562837 #6764]  INFO -- : - Clusters::AgentToken failures: 0
I, [2021-04-25T17:56:10.575814 #6764]  INFO -- : - Operations::FeatureFlagsClient failures: 0
I, [2021-04-25T17:56:10.575931 #6764]  INFO -- : Total: 7 row(s) affected
I, [2021-04-25T17:56:10.575971 #6764]  INFO -- : Done!

I think the line

I, [2021-04-25T17:56:08.865717 #6764] INFO -- : - ApplicationSetting failures: 1
is telling me my gitlab-secrets for application settings are messed up, and that’s why I can’t change any settings. Is there a way to fix this? do I have to do the migration/upgrade path again?

tdpu · April 26, 2021, 3:59am

I think I’ve resolved it… I found the appropriate refs in the documentation.

In addition to the recommended steps of:

DELETE FROM ci_group_variables;
DELETE FROM ci_variables;
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE ci_runners SET token = null, token_encrypted = null;
UPDATE ci_builds SET token = null, token_encrypted = null;
TRUNCATE web_hooks CASCADE;

I had to reset the secret for the application_settings:

DELETE FROM application_settings;

Now most of my 500 errors have been resolved, I’m able to change Application settings. I still have a few errors in my gitlab-rake gitlab:doctor:secrets but I’m working through these.

bzanchi · February 26, 2024, 10:54pm

For those trying to find the linked documentation, it has been reorganized and can be found here:

Topic		Replies	Views
Lib/gitlab/current_settings.rb:32:in `method_missing' Upgrade	2	1101	November 16, 2023
Internal Server Error 500 while accessing /admin/application_settings/general after upgrading to 13.12.8 Migrations	1	1365	July 26, 2021
500 Error while setup in Gitlab MIgration to New server GitLab CI/CD	0	31	September 24, 2024
Projects pages gives HTTP 500 error after data migration to another server Upgrade	10	2642	October 27, 2023
Error 500 on Admin area Upgrade upgrade , admin	5	156	October 15, 2024

500 error when updating application settings

Related topics