500 error when trying to archive a project

How to Use GitLab
1

Our local GitLab instance works fine except for a few 500 errors. One of them happens consistently when trying to archive a project.

Here’s the production.log for the request. The problem seesm to be related to OpenSSL::Cipher::CipherError:

Started POST "/pipe/CERTAIN-PROJECT-I-WANT-ARCHIVED/archive" for 192.168.1.1 at 2019-12-10 22:00:36 +0200
Processing by ProjectsController#archive as HTML
Parameters: {"authenticity_token"=>"[FILTERED]", "namespace_id"=>"pipe", "id"=>"CERTAIN-PROJECT-I-WANT-ARCHIVED"}
Completed 500 Internal Server Error in 58ms (ActiveRecord: 14.3ms | Elasticsearch: 0.0ms)
OpenSSL::Cipher::CipherError ():
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
app/services/projects/update_service.rb:23:in `execute'
app/controllers/projects_controller.rb:168:in `archive'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Started GET "/favicon.ico" for 192.168.1.1 at 2019-12-10 22:00:36 +0200

Any idea what I could try to get archiving to work? I’ve kept GitLab up to date - it is now @ 12.5.4 (latest version - and otherwise it works without other major problems day to day.

2

Also happens when trying to DELETE a similar project

Started DELETE "/pipe/CERTAIN-PROJECT-I-WANT-REMOVED" for 192.168.1.1 at 2019-12-10 22:28:01 +0200
Processing by ProjectsController#destroy as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "namespace_id"=>"pipe", "id"=>"CERTAIN-PROJECT-I-WANT-REMOVED"}
Completed 500 Internal Server Error in 70ms (ActiveRecord: 2.8ms | Elasticsearch: 0.0ms)
OpenSSL::Cipher::CipherError ():
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
app/services/projects/destroy_service.rb:13:in `async_execute'
app/controllers/projects_controller.rb:152:in `destroy'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Error is very similar.

3

Solved! Leaving the solution below in case someone has the same issue:

After reading this similar ticket I realized my groups and projects were restored from a backup a few months ago in a new GitLab instance without also restoring the /etc/gitlab/gitlab-secrets.json file.

So I followed the documentation on When The Secrets File Is Lost and cleared the tokens for projects, groups and application_settings as explained under Resetting runner registration tokens .

All of my 500 erros are now gone as far as I can tell.

1 Like