The `Feature Flags` page cannot be opened normally, and a 500 error is returned in 14.5.0

How to Use GitLab
1

Replace this template with your information

Describe your question in as much detail as possible:

  • What are you seeing, and how does it differ from what you expect to see?
    When I tried to open the Feature Flags page, it returned a 500 error. Other pages did not have this problem.
    image
    image1862×782 35.6 KB

image
image1858×848 28.9 KB

  • Consider including screenshots, error messages, and/or other helpful visuals

  • What version are you on (Hint: /help) ? and are you using self-managed or gitlab.com?
    I am using self-managed docker image to deploy gitlab instances. The version is 14.5.0-ce.0. From Docker Hub .

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
    At first I thought it was an authentication problem. According to Back up and restore GitLab | GitLab , I tried the following in the docker container.
    Refer to the above connection, the first is to reset CI/CD variables and enter the gitlab container.

root@113:/# gitlab-rails dbconsole --database main
psql (12.7)
Type "help" for help.

gitlabhq_production=> SELECT * FROM public."ci_group_variables";
 id | key | value | encrypted_value | encrypted_value_salt | encrypted_value_iv | group_id | protected | created_at | updated_at | masked 
| variable_type | environment_scope 
----+-----+-------+-----------------+----------------------+--------------------+----------+-----------+------------+------------+--------
+---------------+-------------------
(0 rows)

gitlabhq_production=> SELECT * FROM public."ci_variables";
 id | key | value | encrypted_value | encrypted_value_salt | encrypted_value_iv | project_id | protected | environment_scope | masked | va
riable_type 
----+-----+-------+-----------------+----------------------+--------------------+------------+-----------+-------------------+--------+---
------------
(0 rows)
gitlabhq_production=> DELETE FROM ci_group_variables;
DELETE 0
gitlabhq_production=> DELETE FROM ci_variables;
DELETE 0
gitlabhq_production=> exit
could not save history to file "/var/opt/gitlab/.psql_history": Permission denied

Then restart the gitlab container instance.

Enter the ssh window of the gitlab container again, in order to reset the runner registration tokens and pending pipeline jobs, and repair the project integrations.

root@113:/# gitlab-rails dbconsole --database main
psql (12.7)
Type "help" for help.

gitlabhq_production=> UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE 3
gitlabhq_production=> UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE 6
gitlabhq_production=> UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE 1
gitlabhq_production=> UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
UPDATE 1
gitlabhq_production=> UPDATE ci_runners SET token = null, token_encrypted = null;
UPDATE 1
gitlabhq_production=> UPDATE ci_builds SET token = null, token_encrypted = null;
UPDATE 0
gitlabhq_production=> TRUNCATE web_hooks CASCADE;
NOTICE:  truncate cascades to table "web_hook_logs"
NOTICE:  truncate cascades to table "web_hook_logs_202108"
NOTICE:  truncate cascades to table "web_hook_logs_202109"
NOTICE:  truncate cascades to table "web_hook_logs_202110"
NOTICE:  truncate cascades to table "web_hook_logs_202111"
NOTICE:  truncate cascades to table "web_hook_logs_202112"
NOTICE:  truncate cascades to table "web_hook_logs_202201"
NOTICE:  truncate cascades to table "web_hook_logs_202202"
NOTICE:  truncate cascades to table "web_hook_logs_202203"
NOTICE:  truncate cascades to table "web_hook_logs_202204"
NOTICE:  truncate cascades to table "web_hook_logs_202205"
TRUNCATE TABLE
gitlabhq_production=> exit
could not save history to file "/var/opt/gitlab/.psql_history": Permission denied

The last step is that I reconfigured gitlab-runner, and I still can’t access the Feature Flags page at this time.
Through docker log, I got the following log.


127.0.0.1 - - [25/Nov/2021:08:57:26 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 14.5.0 (14-5-stable; go1.13.8; linux/amd64)" -


==> /var/log/gitlab/sidekiq/current <==

{"severity":"INFO","time":"2021-11-25T08:57:27.617Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"retry","retry":0}

{"severity":"INFO","time":"2021-11-25T08:57:27.617Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"retry","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0008484319987474009,"retry":0}

{"severity":"INFO","time":"2021-11-25T08:57:27.618Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"schedule","retry":0}

{"severity":"INFO","time":"2021-11-25T08:57:27.619Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0012072810022800695,"retry":0}


==> /var/log/gitlab/gitlab-rails/production.log <==

Started GET "/linghengqian/test-springboot/-/feature_flags" for 127.0.0.1 at 2021-11-25 08:57:29 +0000

Processing by Projects::FeatureFlagsController#index as HTML

  Parameters: {"namespace_id"=>"linghengqian", "project_id"=>"test-springboot"}

  Rendered layout layouts/project.html.haml (Duration: 24.2ms | Allocations: 2811)

Completed 500 Internal Server Error in 74ms (ActiveRecord: 11.1ms | Elasticsearch: 0.0ms | Allocations: 12711)


==> /var/log/gitlab/gitlab-rails/production_json.log <==

{"method":"GET","path":"/linghengqian/test-springboot/-/feature_flags","format":"html","controller":"Projects::FeatureFlagsController","action":"index","status":500,"time":"2021-11-25T08:57:29.230Z","params":[{"key":"namespace_id","value":"linghengqian"},{"key":"project_id","value":"test-springboot"}],"correlation_id":"01FNB50S908T6SC75C2C9C3X4C","meta.user":"linghengqian","meta.project":"linghengqian/test-springboot","meta.root_namespace":"linghengqian","meta.caller_id":"Projects::FeatureFlagsController#index","meta.remote_ip":"127.0.0.1","meta.feature_category":"feature_flags","meta.client_id":"user/2","remote_ip":"127.0.0.1","user_id":2,"username":"linghengqian","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 Edg/96.0.1054.29","queue_duration_s":0.013948,"request_urgency":"default","target_duration_s":1,"redis_calls":8,"redis_duration_s":0.003102,"redis_read_bytes":1294,"redis_write_bytes":1142,"redis_cache_calls":5,"redis_cache_duration_s":0.002181,"redis_cache_read_bytes":1013,"redis_cache_write_bytes":281,"redis_shared_state_calls":3,"redis_shared_state_duration_s":0.000921,"redis_shared_state_read_bytes":281,"redis_shared_state_write_bytes":861,"db_count":14,"db_write_count":0,"db_cached_count":1,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":14,"db_primary_cached_count":1,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.015,"cpu_s":0.094044,"mem_objects":21253,"mem_bytes":2368968,"mem_mallocs":5056,"mem_total_bytes":3219088,"pid":650,"exception.class":"ActionView::Template::Error","exception.message":"","exception.backtrace":["lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'","app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'","app/models/concerns/token_authenticatable.rb:35:in `block in add_authentication_token_field'","app/models/project.rb:2672:in `feature_flags_client_token'","app/helpers/feature_flags_helper.rb:11:in `unleash_api_instance_id'","app/views/projects/feature_flags/index.html.haml:13","app/controllers/application_controller.rb:136:in `render'","app/controllers/application_controller.rb:504:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:495:in `set_session_storage'","lib/gitlab/i18n.rb:105:in `with_locale'","lib/gitlab/i18n.rb:111:in `with_user_locale'","app/controllers/application_controller.rb:489:in `set_locale'","app/controllers/application_controller.rb:483:in `set_current_context'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:26:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"db_duration_s":0.0111,"view_duration_s":0.0,"duration_s":0.07401}


==> /var/log/gitlab/gitlab-rails/production.log <==

  

ActionView::Template::Error ():

    10:   "feature-flags-limit-exceeded" => @project.actual_limits.exceeded?(:project_feature_flags, @project.operations_feature_flags.count),

    11:   "feature-flags-limit" => @project.actual_limits.project_feature_flags,

    12:   "unleash-api-url" => (unleash_api_url(@project) if can?(current_user, :admin_feature_flag, @project)),

    13:   "unleash-api-instance-id" => (unleash_api_instance_id(@project) if can?(current_user, :admin_feature_flag, @project)),

    14:   "can-user-admin-feature-flag" => can?(current_user, :admin_feature_flag, @project),

    15:   "new-feature-flag-path" => can?(current_user, :create_feature_flag, @project) ? new_project_feature_flag_path(@project): nil,

    16:   "rotate-instance-id-path" => can?(current_user, :admin_feature_flags_client, @project) ? reset_token_project_feature_flags_client_path(@project, format: :json) : nil,

  

lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'

app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'

app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'

app/models/concerns/token_authenticatable.rb:35:in `block in add_authentication_token_field'

app/models/project.rb:2672:in `feature_flags_client_token'

app/helpers/feature_flags_helper.rb:11:in `unleash_api_instance_id'

app/views/projects/feature_flags/index.html.haml:13

app/controllers/application_controller.rb:136:in `render'

app/controllers/application_controller.rb:504:in `set_current_admin'

lib/gitlab/session.rb:11:in `with_session'

app/controllers/application_controller.rb:495:in `set_session_storage'

lib/gitlab/i18n.rb:105:in `with_locale'

lib/gitlab/i18n.rb:111:in `with_user_locale'

app/controllers/application_controller.rb:489:in `set_locale'

app/controllers/application_controller.rb:483:in `set_current_context'

lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'

lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'

lib/gitlab/middleware/speedscope.rb:13:in `call'

lib/gitlab/request_profiler/middleware.rb:17:in `call'

lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'

lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'

lib/gitlab/metrics/web_transaction.rb:46:in `run'

lib/gitlab/metrics/rack_middleware.rb:16:in `call'

lib/gitlab/jira/middleware.rb:19:in `call'

lib/gitlab/middleware/go.rb:20:in `call'

lib/gitlab/etag_caching/middleware.rb:21:in `call'

lib/gitlab/middleware/multipart.rb:173:in `call'

lib/gitlab/middleware/read_only/controller.rb:50:in `call'

lib/gitlab/middleware/read_only.rb:18:in `call'

lib/gitlab/middleware/same_site_cookies.rb:27:in `call'

lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'

lib/gitlab/middleware/basic_health_check.rb:25:in `call'

lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'

lib/gitlab/middleware/request_context.rb:21:in `call'

config/initializers/fix_local_cache_middleware.rb:11:in `call'

lib/gitlab/middleware/compressed_json.rb:26:in `call'

lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'

lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'

lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'

lib/gitlab/middleware/release_env.rb:13:in `call'


==> /var/log/gitlab/gitlab-workhorse/current <==

{"content_type":"text/html; charset=utf-8","correlation_id":"01FNB50S908T6SC75C2C9C3X4C","duration_ms":115,"host":"gitlab.lingh.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://gitlab.lingh.com/linghengqian/test-springboot","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":500,"system":"http","time":"2021-11-25T08:57:29Z","ttfb_ms":115,"uri":"/linghengqian/test-springboot/-/feature_flags","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 Edg/96.0.1054.29","written_bytes":2926}


==> /var/log/gitlab/nginx/gitlab_access.log <==

127.0.0.1 - - [25/Nov/2021:08:57:29 +0000] "GET /linghengqian/test-springboot/-/feature_flags HTTP/1.1" 500 2926 "http://gitlab.lingh.com/linghengqian/test-springboot" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 Edg/96.0.1054.29" -


==> /var/log/gitlab/gitlab-workhorse/current <==

{"content_type":"","correlation_id":"01FNB50SWESBB8PR9V886TBPG2","duration_ms":0,"host":"gitlab.lingh.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2021-11-25T08:57:29Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 14.5.0 (14-5-stable; go1.13.8; linux/amd64)","written_bytes":0}


==> /var/log/gitlab/nginx/gitlab_access.log <==

127.0.0.1 - - [25/Nov/2021:08:57:29 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 14.5.0 (14-5-stable; go1.13.8; linux/amd64)" -


==> /var/log/gitlab/gitlab-exporter/current <==

2021-11-25_08:57:30.91522 127.0.0.1 - - [25/Nov/2021:08:57:30 UTC] "GET /database HTTP/1.1" 200 1222

2021-11-25_08:57:30.91539 - -> /database

2021-11-25_08:57:32.10531 Passing 'exists?' command to redis as is; blind passthrough has been deprecated and will be removed in redis-namespace 2.0 (at /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sidekiq-6.2.2/lib/sidekiq/api.rb:959:in `block (3 levels) in each')

2021-11-25_08:57:32.14823 127.0.0.1 - - [25/Nov/2021:08:57:31 UTC] "GET /sidekiq HTTP/1.1" 200 68880

2021-11-25_08:57:32.14839 - -> /sidekiq


==> /var/log/gitlab/sidekiq/current <==

{"severity":"INFO","time":"2021-11-25T08:57:32.626Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"retry","retry":0}

{"severity":"INFO","time":"2021-11-25T08:57:32.627Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"retry","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0009575539988873061,"retry":0}

{"severity":"INFO","time":"2021-11-25T08:57:32.627Z","message":"Enqueuing scheduled jobs","status":"start","sorted_set":"schedule","retry":0}

{"severity":"INFO","time":"2021-11-25T08:57:32.628Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":0,"redundant_jobs_count":0,"duration_s":0.0006173080000735354,"retry":0}


==> /var/log/gitlab/gitlab-workhorse/current <==

{"content_type":"","correlation_id":"01FNB50WT6CWVTZ7X80S9A2N2A","duration_ms":0,"host":"gitlab.lingh.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2021-11-25T08:57:32Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 14.5.0 (14-5-stable; go1.13.8; linux/amd64)","written_bytes":0}


==> /var/log/gitlab/nginx/gitlab_access.log <==

127.0.0.1 - - [25/Nov/2021:08:57:32 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 14.5.0 (14-5-stable; go1.13.8; linux/amd64)" -

Thanks for taking the time to be thorough in your request, it really helps! :blush:

2

This issue was resolved after yum update.