Web ide -> 500

vaschthestampede · November 20, 2024, 4:40pm

If I try to use the WEB IDE it gives me the following page:

I’m on a self-managed 17.5.2

root@myGitLab ~# gitlab-rake gitlab:env:info

System information
System:         Debian 11
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.5.17
Bundler Version:2.5.11
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.2.4
Go Version:     unknown

GitLab information
Version:        17.5.2
Revision:       cebb958cb73
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.11
URL:            https://gitlab.eagle.local
HTTP Clone URL: https://gitlab.eagle.local/some-group/some-project.git
SSH Clone URL:  git@gitlab.eagle.local:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        14.39.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.5.2
- default Git Version:  2.46.2

Let me know where to look for more information to analyze the issue.

vaschthestampede · November 21, 2024, 5:15pm

The problem persists even after updating to 17.6.0

root@myGitLab ~# gitlab-rake gitlab:env:info

System information
System:         Debian 11
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.5.23
Bundler Version:2.5.11
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.2.4
Go Version:     unknown

GitLab information
Version:        17.6.0
Revision:       746f630acb1
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.11
URL:            https://gitlab.eagle.local
HTTP Clone URL: https://gitlab.eagle.local/some-group/some-project.git
SSH Clone URL:  git@gitlab.eagle.local:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        14.39.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.6.0
- default Git Version:  2.47.0

vaschthestampede · November 29, 2024, 9:03am

The problem persists even after updating to 17.6.0

gitlab-rake gitlab:env:info

System information
System:         Debian 11
Current User:   git
Using RVM:      no
Ruby Version:   3.2.5
Gem Version:    3.5.23
Bundler Version:2.5.11
Rake Version:   13.0.6
Redis Version:  7.0.15
Sidekiq Version:7.2.4
Go Version:     unknown

GitLab information
Version:        17.6.1
Revision:       8a31863db02
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     14.11
URL:            https://gitlab.eagle.local
HTTP Clone URL: https://gitlab.eagle.local/some-group/some-project.git
SSH Clone URL:  git@gitlab.eagle.local:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        14.39.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      17.6.1
- default Git Version:  2.47.0

Please let me know where to look for more information to analyze the issue.

dnsmichi · November 29, 2024, 11:52am

500 errors usually refer to backend errors, which should be visible in the production log. Try tailing the log file while making the browser request to the Web IDE and correlate the error.

vaschthestampede · November 29, 2024, 2:11pm

Thank you very much.

This is the log:

OpenSSL::Cipher::CipherError ():
  
encryptor (3.0.0) lib/encryptor.rb:98:in `final'
encryptor (3.0.0) lib/encryptor.rb:98:in `crypt'
encryptor (3.0.0) lib/encryptor.rb:49:in `decrypt'
lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:16:in `decrypt_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:77:in `get_encrypted_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:112:in `token_set?'
app/models/concerns/token_authenticatable_strategies/base.rb:46:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:37:in `ensure_token'
app/models/concerns/token_authenticatable.rb:56:in `block in add_authentication_token_field'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:686:in `catch'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:686:in `block in default_terminator'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:200:in `block in halting'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:106:in `run_callbacks'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
activerecord (7.0.8.4) lib/active_record/callbacks.rb:455:in `create_or_update'
activerecord (7.0.8.4) lib/active_record/timestamp.rb:126:in `create_or_update'
activerecord (7.0.8.4) lib/active_record/persistence.rb:648:in `save!'
activerecord (7.0.8.4) lib/active_record/validations.rb:53:in `save!'
activerecord (7.0.8.4) lib/active_record/transactions.rb:302:in `block in save!'
activerecord (7.0.8.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
activerecord (7.0.8.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
activerecord (7.0.8.4) lib/active_record/transactions.rb:302:in `save!'
activerecord (7.0.8.4) lib/active_record/suppressor.rb:54:in `save!'
activerecord (7.0.8.4) lib/active_record/persistence.rb:782:in `block in update!'
activerecord (7.0.8.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
activerecord (7.0.8.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
activerecord (7.0.8.4) lib/active_record/persistence.rb:780:in `update!'
lib/web_ide/default_oauth_application.rb:51:in `block in ensure_oauth_application!'
app/models/concerns/cross_database_modification.rb:91:in `block in transaction'
activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
activerecord (7.0.8.4) lib/active_record/transactions.rb:209:in `transaction'
lib/gitlab/database.rb:394:in `block in transaction'
activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'
activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'
lib/gitlab/database.rb:393:in `transaction'
app/models/concerns/cross_database_modification.rb:82:in `transaction'
activerecord (7.0.8.4) lib/active_record/transactions.rb:290:in `transaction'
lib/web_ide/default_oauth_application.rb:39:in `ensure_oauth_application!'
app/controllers/ide_controller.rb:48:in `ensure_web_ide_oauth_application!'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:180:in `block (2 levels) in halting_and_conditional'
actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:181:in `block in halting_and_conditional'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:116:in `block in run_callbacks'
app/controllers/application_controller.rb:503:in `set_current_admin'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:494:in `set_session_storage'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
lib/gitlab/i18n.rb:114:in `with_locale'
lib/gitlab/i18n.rb:120:in `with_user_locale'
app/controllers/application_controller.rb:485:in `set_locale'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
app/controllers/application_controller.rb:478:in `set_current_context'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:32:in `block in sentry_around_action'
sentry-ruby (5.21.0) lib/sentry/hub.rb:108:in `with_child_span'
sentry-ruby (5.21.0) lib/sentry-ruby.rb:499:in `with_child_span'
sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'
actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'
actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'
actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'
activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'
activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'
actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'
actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'
activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'
actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'
actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'
actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `block in dispatch'
lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'
actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `dispatch'
actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'
actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'
actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'
actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'
gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'
flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'
flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'
lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'
lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:83:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/ci/job_token/middleware.rb:11:in `call'
batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'
rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'
apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'
warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
warden (1.2.9) lib/warden/manager.rb:34:in `catch'
warden (1.2.9) lib/warden/manager.rb:34:in `call'
rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'
rack (2.2.10) lib/rack/tempfile_reaper.rb:15:in `call'
rack (2.2.10) lib/rack/etag.rb:27:in `call'
rack (2.2.10) lib/rack/conditional_get.rb:27:in `call'
rack (2.2.10) lib/rack/head.rb:12:in `call'
actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'
actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
rack (2.2.10) lib/rack/session/abstract/id.rb:266:in `context'
rack (2.2.10) lib/rack/session/abstract/id.rb:260:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'
lib/gitlab/middleware/strip_cookies.rb:29:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'
actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
sentry-rails (5.21.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'
sentry-ruby (5.21.0) lib/sentry/hub.rb:265:in `with_session_tracking'
sentry-ruby (5.21.0) lib/sentry-ruby.rb:412:in `with_session_tracking'
sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'
sentry-ruby (5.21.0) lib/sentry/hub.rb:59:in `with_scope'
sentry-ruby (5.21.0) lib/sentry-ruby.rb:392:in `with_scope'
sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'
activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'
activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'
activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'
railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:15:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'
rack (2.2.10) lib/rack/method_override.rb:24:in `call'
rack (2.2.10) lib/rack/runtime.rb:22:in `call'
rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'
rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'
rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:44:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
rack (2.2.10) lib/rack/sendfile.rb:110:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call'
gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context'
gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'
actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
railties (7.0.8.4) lib/rails/engine.rb:530:in `call'
railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'
railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'
lib/gitlab/middleware/release_env.rb:12:in `call'
rack (2.2.10) lib/rack/urlmap.rb:74:in `block in call'
rack (2.2.10) lib/rack/urlmap.rb:58:in `each'
rack (2.2.10) lib/rack/urlmap.rb:58:in `call'
puma (6.4.3) lib/puma/configuration.rb:272:in `call'
puma (6.4.3) lib/puma/request.rb:100:in `block in handle_request'
puma (6.4.3) lib/puma/thread_pool.rb:378:in `with_force_shutdown'
puma (6.4.3) lib/puma/request.rb:99:in `handle_request'
puma (6.4.3) lib/puma/server.rb:464:in `process_client'
puma (6.4.3) lib/puma/server.rb:245:in `block in run'
puma (6.4.3) lib/puma/thread_pool.rb:155:in `block in spawn_thread'

It appears to indicate a failure in the encryption or decryption process.

Could it be an openssl version issue?

openssl version 
OpenSSL 1.1.1w  11 Sep 2023

dnsmichi · November 29, 2024, 6:44pm

I’ve found a few issues in the issue tracker with the error message, for example Unable to Delete or Rename Projects in GitLab (“Internal Error 500”) (#30885) · Issues · GitLab.org / GitLab · GitLab All of them mention that the secrets file might have gone missing, causing this error. In this case, it might affect project authentication.

'When the secrets file list lost" describes the scenarios and steps to take. Proceed with caution and first verify whether the secrets file is there.

vaschthestampede · December 2, 2024, 10:08am

Referring to secrets file, in my system the file /etc/gitlab/gitlab-secrets.json doesn’t exist…
There is the file /etc/gitlab/gitlab-secrets.json.bak, I tried copying and renaming it but every time I reboot the file gets deleted…

This is an excerpt from the file /etc/gitlab/gitlab-secrets.json:

{
  "gitlab_workhorse": {
    "secret_token": "*****"
  },
  "gitlab_shell": {
    "secret_token": "*****"
  },
  "gitlab_rails": {
    "secret_key_base": "*****",
    "db_key_base": "*****",
    "otp_key_base": "*****",
    "encrypted_settings_key_base": "*****",
    "openid_connect_signing_key": "-----BEGIN RSA PRIVATE KEY-----*****>
  },
  "gitlab_pages": {
    "gitlab_secret": ""*****"",
    "gitlab_id": ""*****"",
    "auth_secret": null,
    "api_secret_key": ""*****"",
    "register_as_oauth_app": null
  },
  "gitlab_kas": {
    "api_secret_key": ""*****"",
    "private_api_secret_key": ""*****""
  },
  "suggested_reviewers": {
    "api_secret_key": null
  },
  "registry": {
    "http_secret": ""*****"",
    "internal_certificate": "-----BEGIN CERTIFICATE-----*****>
    "internal_key": "-----BEGIN RSA PRIVATE KEY-----\*****>
  },
  "letsencrypt": {
    "auto_enabled": null
  },
  "mattermost": {
    "email_invite_salt": ""*****"",
    "file_public_link_salt": ""*****"",
    "sql_at_rest_encrypt_key": ""*****"",
    "register_as_oauth_app": null
  },
  "postgresql": {
    "internal_certificate": "-----BEGIN CERTIFICATE-----\*****>
    "internal_key": "-----BEGIN RSA PRIVATE KEY-----\*****>
  },
  "mailroom": {
    "incoming_email_auth_token": null,
    "service_desk_email_auth_token": null
  },
  "gitaly": {
    "gitlab_secret": ""*****""
  }
}

From what I can see there are no problems.

dnsmichi · December 2, 2024, 12:31pm

Maybe there is a cronjob on the system that tries to backup everything, including the secrets.json file. Or the system has been modified by someone, and something weird is going on in the background. My debugging strategy would be to check all startup scripts after boot, and trace the file access to /etc/gitlab.

Eventually, /etc/gitlab is mounted as a tmpfs which gets purged after reboot, or otherwise on a non-functional network filesystem. Or the setup runs in Docker, and the bind mounts do not work properly.

Clearly, the error you are seeing sources from a non-existent secrets.json file, and outside of the Web IDE, it may also affect other functionality that requires authentication (CI/CD, etc.).

lreisoliveira · December 2, 2024, 7:01pm

Hi, did you manage to solve it?

vaschthestampede · December 3, 2024, 7:52am

No.
I don’t even have a clue how to apply dnssmichi’s advice yet.

dnsmichi · December 3, 2024, 10:43am

Assuming you have SSH root access on the Linux VM or server, you’ll need some basic Linux tools and analytics knowledge.

For example, you can increase the log level, or use log history inspection (lastlog, history, cat ~/.bash_history, less /var/log/syslog, etc.) to figure out if there was a script or command that deleted that file. For example, Any way to determine how files were deleted on Linux/Apache server? - Server Fault - I googled for linux figure out how a file is deleted on boot.

Cronjobs - linux cronjobs logs Where is the cron / crontab log? - Ask Ubuntu can help see if there is a script running that eventually manipulates the files.

grep CRON /var/log/syslog

Mounts - run the mount command as root to see all mounted nodes. Filter the results for tmpfs - mount | grep tmp. It should only show something like this on an Ubuntu node:

root@legendiary:~# mount | grep tmp
udev on /dev type devtmpfs (rw,nosuid,relatime,size=941204k,nr_inodes=235301,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=196192k,mode=755,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=196188k,nr_inodes=49047,mode=700,inode64)

Before giving Docker bind mount details, I’d like to know if the setup is run in Docker at all, or using the raw Omnibus package.

vaschthestampede · December 3, 2024, 11:53am

The system was built using turnkeylinux. In particular the debian system is an LXC container.
So the given indications should be applicable.

root@myGitLab ~# mount | grep tmp
none on /dev type tmpfs (rw,relatime,size=492k,mode=755,uid=100000,gid=100000,inode64)
udev on /dev/full type devtmpfs (rw,nosuid,relatime,size=32908288k,nr_inodes=8227072,mode=755,inode64)
udev on /dev/null type devtmpfs (rw,nosuid,relatime,size=32908288k,nr_inodes=8227072,mode=755,inode64)
udev on /dev/random type devtmpfs (rw,nosuid,relatime,size=32908288k,nr_inodes=8227072,mode=755,inode64)
udev on /dev/tty type devtmpfs (rw,nosuid,relatime,size=32908288k,nr_inodes=8227072,mode=755,inode64)
udev on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=32908288k,nr_inodes=8227072,mode=755,inode64)
udev on /dev/zero type devtmpfs (rw,nosuid,relatime,size=32908288k,nr_inodes=8227072,mode=755,inode64)
none on /proc/sys/kernel/random/boot_id type tmpfs (ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=100000,gid=100000,inode64)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=100000,gid=100000,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=13177164k,nr_inodes=819200,mode=755,uid=100000,gid=100000,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=100000,gid=100000,inode64)

and:

root@myGitLab ~# grep CRON /var/log/syslog
Dec  1 00:56:01 myGitLab CRON[71929]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 01:56:01 myGitLab CRON[73277]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 02:56:01 myGitLab CRON[74630]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 03:06:01 myGitLab CRON[74859]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /usr/lib/x86_64-linux-gnu/e2fsprogs/e2scrub_all_cron)
Dec  1 03:55:01 myGitLab CRON[75974]: (root) CMD (test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt)
Dec  1 03:55:01 myGitLab CRON[75975]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Dec  1 03:56:01 myGitLab CRON[76011]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 04:16:22 myGitLab cron-apt: CRON-APT RUN [/etc/cron-apt/config]: Sun Dec  1 03:55:01 UTC 2024
Dec  1 04:16:23 myGitLab cron-apt: CRON-APT SLEEP: 1278, Sun Dec  1 04:16:19 UTC 2024
Dec  1 04:16:23 myGitLab cron-apt: CRON-APT ACTION: 0-update
Dec  1 04:16:23 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 update -o quiet=2
Dec  1 04:16:23 myGitLab cron-apt: CRON-APT ACTION: 5-install
Dec  1 04:16:23 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 autoclean -y
Dec  1 04:16:30 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -y     -o APT::Get::Show-Upgraded=true     -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list     -o Dir::Etc::sourceparts=nonexistent     -o DPkg::Options::=--force-confdef     -o DPkg::Options::=--force-confold
Dec  1 04:56:01 myGitLab CRON[78498]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 05:56:01 myGitLab CRON[79833]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 06:00:01 myGitLab CRON[79919]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly ))
Dec  1 06:13:02 myGitLab CRON[80212]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Dec  1 06:29:01 myGitLab CRON[80714]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ))
Dec  1 06:56:01 myGitLab CRON[81318]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 07:56:01 myGitLab CRON[82735]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 08:56:01 myGitLab CRON[84084]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 09:56:01 myGitLab CRON[85443]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 10:56:01 myGitLab CRON[86767]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 11:56:01 myGitLab CRON[88141]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 12:56:01 myGitLab CRON[89514]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 13:56:01 myGitLab CRON[90870]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 14:56:01 myGitLab CRON[92210]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 15:56:01 myGitLab CRON[93584]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 16:56:01 myGitLab CRON[94949]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 17:56:01 myGitLab CRON[96288]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 18:56:01 myGitLab CRON[97649]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 19:56:01 myGitLab CRON[99002]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 20:56:01 myGitLab CRON[100420]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 21:56:01 myGitLab CRON[101754]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 22:56:01 myGitLab CRON[103101]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  1 23:56:01 myGitLab CRON[104450]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 00:56:01 myGitLab CRON[105799]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 01:56:01 myGitLab CRON[107139]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 02:56:01 myGitLab CRON[108445]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 03:55:01 myGitLab CRON[109793]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Dec  2 03:55:01 myGitLab CRON[109792]: (root) CMD (test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt)
Dec  2 03:56:01 myGitLab CRON[109829]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 04:03:47 myGitLab cron-apt: CRON-APT RUN [/etc/cron-apt/config]: Mon Dec  2 03:55:01 UTC 2024
Dec  2 04:03:47 myGitLab cron-apt: CRON-APT SLEEP: 523, Mon Dec  2 04:03:44 UTC 2024
Dec  2 04:03:47 myGitLab cron-apt: CRON-APT ACTION: 0-update
Dec  2 04:03:47 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 update -o quiet=2
Dec  2 04:03:48 myGitLab cron-apt: CRON-APT ACTION: 5-install
Dec  2 04:03:48 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 autoclean -y
Dec  2 04:03:55 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -y     -o APT::Get::Show-Upgraded=true     -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list     -o Dir::Etc::sourceparts=nonexistent     -o DPkg::Options::=--force-confdef     -o DPkg::Options::=--force-confold
Dec  2 04:56:01 myGitLab CRON[112368]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 05:56:01 myGitLab CRON[113718]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 06:13:01 myGitLab CRON[114114]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Dec  2 06:56:02 myGitLab CRON[115218]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 07:56:01 myGitLab CRON[116566]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 08:56:01 myGitLab CRON[117905]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 09:41:34 myGitLab cron[147]: (CRON) INFO (pidfile fd = 3)
Dec  2 09:41:34 myGitLab cron[147]: (CRON) INFO (Running @reboot jobs)
Dec  2 09:56:25 myGitLab cron[104]: (CRON) INFO (pidfile fd = 3)
Dec  2 09:56:25 myGitLab cron[104]: (CRON) INFO (Running @reboot jobs)
Dec  2 10:56:01 myGitLab CRON[5013]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 11:56:01 myGitLab CRON[6373]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 12:56:01 myGitLab CRON[7779]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 13:56:01 myGitLab CRON[9126]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 14:56:01 myGitLab CRON[10506]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 15:56:01 myGitLab CRON[11962]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 16:56:01 myGitLab CRON[13341]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 17:56:01 myGitLab CRON[14680]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 18:56:01 myGitLab CRON[16056]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 19:56:01 myGitLab CRON[17403]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 20:56:01 myGitLab CRON[18754]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 21:56:01 myGitLab CRON[20125]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 22:56:01 myGitLab CRON[21489]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  2 23:56:01 myGitLab CRON[22869]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 00:56:01 myGitLab CRON[24270]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 01:56:01 myGitLab CRON[25621]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 02:56:01 myGitLab CRON[26993]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 03:55:01 myGitLab CRON[28329]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Dec  3 03:55:01 myGitLab CRON[28328]: (root) CMD (test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt)
Dec  3 03:56:01 myGitLab CRON[28365]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT RUN [/etc/cron-apt/config]: Tue Dec  3 03:55:01 UTC 2024
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT SLEEP: 2607, Tue Dec  3 04:38:28 UTC 2024
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT ACTION: 0-update
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 update -o quiet=2
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT ACTION: 5-install
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 autoclean -y
Dec  3 04:38:32 myGitLab cron-apt: CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -y     -o APT::Get::Show-Upgraded=true     -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list     -o Dir::Etc::sourceparts=nonexistent     -o DPkg::Options::=--force-confdef     -o DPkg::Options::=--force-confold
Dec  3 04:56:01 myGitLab CRON[30105]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 05:56:01 myGitLab CRON[31508]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 06:13:01 myGitLab CRON[31898]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Dec  3 06:56:01 myGitLab CRON[32948]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 07:56:01 myGitLab CRON[34325]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 08:56:01 myGitLab CRON[35686]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 09:56:01 myGitLab CRON[37044]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Dec  3 10:56:01 myGitLab CRON[38470]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

At first glance there doesn’t seem to be any activity in the /etc/gitlab/gitlab-secrets.json file, am I wrong?

dnsmichi · December 3, 2024, 12:41pm

Hmmm I have no experience with LXC, but it uses a similar container abstraction layer as with Docker.

When you say that you restart the server, are you restarting the LXC container, or the entire turnkeylinux host system?

vaschthestampede · December 3, 2024, 1:05pm

I probably explained myself badly.

The LXC container is an entire Debian system with gitlab installed.
So, as far as gitlab is concerned, there are no differences with a system installed on Debian.

I am referring to the LXC container.

lreisoliveira · December 3, 2024, 1:32pm

A small contribution: I did a new, clean installation, without any registered project, only with the root user from https://hub.docker.com/r/gitlab/gitlab-ee/tags/?name=17.6.1 and error 500 happens when saving a configuration as root here /admin/application_settings/general

vaschthestampede · December 3, 2024, 2:30pm

I’m also trying with a fresh installation, the problem even occurs at login.

dnsmichi · December 3, 2024, 3:59pm

Can you share a little more on how GitLab is installed into the Debian LXC container. Is it using the Omnibus packages, source installation, something else, any custom scripts involved?

By default, the Omnibus package installation will create the secrets.json file. You can also re-create it through rotation.

dnsmichi · December 3, 2024, 4:00pm

That sounds like a different problem, likely with how Docker is executed and files are mounted. Suggest creating a new topic.

vaschthestampede · December 3, 2024, 4:24pm

It’s using the Omnibus packages.
No custom scripts involved by me, I don’t know if turnkey has them.

I think there is some problem in decoding the DB

root@myGitLab ~# gitlab-rake gitlab:doctor:secrets VERBOSE=1
I, [2024-12-03T16:18:53.610289 #48374]  INFO -- : Checking encrypted values in the database
I, [2024-12-03T16:19:03.944763 #48374]  INFO -- : - Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration failures: 0
I, [2024-12-03T16:19:03.950948 #48374]  INFO -- : - VirtualRegistries::Packages::Maven::Upstream failures: 0
I, [2024-12-03T16:19:03.956984 #48374]  INFO -- : - Packages::Debian::ProjectDistributionKey failures: 0
I, [2024-12-03T16:19:03.962885 #48374]  INFO -- : - Packages::Debian::GroupDistributionKey failures: 0
I, [2024-12-03T16:19:03.973944 #48374]  INFO -- : - Integrations::Instance::Integration failures: 0
I, [2024-12-03T16:19:03.980292 #48374]  INFO -- : - Clusters::Providers::Gcp failures: 0
I, [2024-12-03T16:19:03.986714 #48374]  INFO -- : - Clusters::Providers::Aws failures: 0
I, [2024-12-03T16:19:03.988711 #48374]  INFO -- : - Clusters::Integrations::Prometheus failures: 0
I, [2024-12-03T16:19:03.994619 #48374]  INFO -- : - ServiceDesk::CustomEmailVerification failures: 0
I, [2024-12-03T16:19:04.000504 #48374]  INFO -- : - ServiceDesk::CustomEmailCredential failures: 0
I, [2024-12-03T16:19:04.006439 #48374]  INFO -- : - Integrations::ZentaoTrackerData failures: 0
I, [2024-12-03T16:19:04.013129 #48374]  INFO -- : - Integrations::JiraTrackerData failures: 0
I, [2024-12-03T16:19:04.019184 #48374]  INFO -- : - Integrations::IssueTrackerData failures: 0
I, [2024-12-03T16:19:04.025091 #48374]  INFO -- : - IncidentManagement::ProjectIncidentManagementSetting failures: 0
I, [2024-12-03T16:19:04.030837 #48374]  INFO -- : - ErrorTracking::ProjectErrorTrackingSetting failures: 0
I, [2024-12-03T16:19:04.062093 #48374]  INFO -- : - Clusters::KubernetesNamespace failures: 0
I, [2024-12-03T16:19:04.067840 #48374]  INFO -- : - CloudConnector::ServiceAccessToken failures: 0
I, [2024-12-03T16:19:04.073744 #48374]  INFO -- : - BulkImports::Configuration failures: 0
I, [2024-12-03T16:19:04.079565 #48374]  INFO -- : - Atlassian::Identity failures: 0
I, [2024-12-03T16:19:04.084950 #48374]  INFO -- : - Alerting::ProjectAlertingSetting failures: 0
I, [2024-12-03T16:19:04.090975 #48374]  INFO -- : - SlackIntegration failures: 0
I, [2024-12-03T16:19:04.096440 #48374]  INFO -- : - RemoteMirror failures: 0
I, [2024-12-03T16:19:04.137270 #48374]  INFO -- : - ProjectSetting failures: 0
I, [2024-12-03T16:19:04.143783 #48374]  INFO -- : - PagesDomainAcmeOrder failures: 0
I, [2024-12-03T16:19:04.176519 #48374]  INFO -- : - PagesDomain failures: 0
I, [2024-12-03T16:19:04.182493 #48374]  INFO -- : - JiraConnectInstallation failures: 0
I, [2024-12-03T16:19:04.188291 #48374]  INFO -- : - GrafanaIntegration failures: 0
I, [2024-12-03T16:19:04.194181 #48374]  INFO -- : - ChatName failures: 0
I, [2024-12-03T16:19:04.202950 #48374]  INFO -- : - AlertManagement::HttpIntegration failures: 0
I, [2024-12-03T16:19:04.239366 #48374]  INFO -- : - WebHook failures: 0
I, [2024-12-03T16:19:04.244889 #48374]  INFO -- : - Integration failures: 0
I, [2024-12-03T16:19:04.252581 #48374]  INFO -- : - ProjectImportData failures: 0
I, [2024-12-03T16:19:04.260726 #48374]  INFO -- : - Clusters::Platforms::Kubernetes failures: 0
I, [2024-12-03T16:19:04.293626 #48374]  INFO -- : - Snippet failures: 0
I, [2024-12-03T16:19:04.525760 #48374]  INFO -- : - User failures: 4
D, [2024-12-03T16:19:04.525870 #48374] DEBUG -- :   - User[1]: static_object_token
D, [2024-12-03T16:19:04.525924 #48374] DEBUG -- :   - User[2]: static_object_token
D, [2024-12-03T16:19:04.525981 #48374] DEBUG -- :   - User[3]: static_object_token
D, [2024-12-03T16:19:04.526040 #48374] DEBUG -- :   - User[4]: static_object_token
I, [2024-12-03T16:19:04.563867 #48374]  INFO -- : - ApplicationSetting failures: 1
D, [2024-12-03T16:19:04.563970 #48374] DEBUG -- :   - ApplicationSetting[1]: error_tracking_access_token
I, [2024-12-03T16:19:04.569352 #48374]  INFO -- : - SystemHook failures: 0
I, [2024-12-03T16:19:04.573623 #48374]  INFO -- : - ServiceHook failures: 0
I, [2024-12-03T16:19:04.577144 #48374]  INFO -- : - ProjectHook failures: 0
I, [2024-12-03T16:19:04.581209 #48374]  INFO -- : - Integrations::BeyondIdentity failures: 0
I, [2024-12-03T16:19:04.585043 #48374]  INFO -- : - Integrations::Pushover failures: 0
I, [2024-12-03T16:19:04.588557 #48374]  INFO -- : - Integrations::BaseMonitoring failures: 0
I, [2024-12-03T16:19:04.592424 #48374]  INFO -- : - Integrations::Pivotaltracker failures: 0
I, [2024-12-03T16:19:04.595622 #48374]  INFO -- : - Integrations::PipelinesEmail failures: 0
I, [2024-12-03T16:19:04.599673 #48374]  INFO -- : - Integrations::Packagist failures: 0
I, [2024-12-03T16:19:04.603467 #48374]  INFO -- : - Integrations::Irker failures: 0
I, [2024-12-03T16:19:04.606648 #48374]  INFO -- : - Integrations::EmailsOnPush failures: 0
I, [2024-12-03T16:19:04.610396 #48374]  INFO -- : - Integrations::SquashTm failures: 0
I, [2024-12-03T16:19:04.613600 #48374]  INFO -- : - Integrations::BaseSlashCommands failures: 0
I, [2024-12-03T16:19:04.617409 #48374]  INFO -- : - Integrations::JiraCloudApp failures: 0
I, [2024-12-03T16:19:04.621168 #48374]  INFO -- : - Integrations::Harbor failures: 0
I, [2024-12-03T16:19:04.624341 #48374]  INFO -- : - Integrations::GooglePlay failures: 0
I, [2024-12-03T16:19:04.628145 #48374]  INFO -- : - Integrations::ExternalWiki failures: 0
I, [2024-12-03T16:19:04.631806 #48374]  INFO -- : - Integrations::BaseChatNotification failures: 0
I, [2024-12-03T16:19:04.635659 #48374]  INFO -- : - Integrations::DiffblueCover failures: 0
I, [2024-12-03T16:19:04.639721 #48374]  INFO -- : - Integrations::Datadog failures: 0
I, [2024-12-03T16:19:04.642902 #48374]  INFO -- : - Integrations::BaseThirdPartyWiki failures: 0
I, [2024-12-03T16:19:04.646704 #48374]  INFO -- : - Integrations::Campfire failures: 0
I, [2024-12-03T16:19:04.650216 #48374]  INFO -- : - Integrations::BaseIssueTracker failures: 0
I, [2024-12-03T16:19:04.654314 #48374]  INFO -- : - Integrations::BaseCi failures: 0
I, [2024-12-03T16:19:04.658092 #48374]  INFO -- : - Integrations::Assembla failures: 0
I, [2024-12-03T16:19:04.661888 #48374]  INFO -- : - Integrations::Asana failures: 0
I, [2024-12-03T16:19:04.665603 #48374]  INFO -- : - Integrations::AppleAppStore failures: 0
I, [2024-12-03T16:19:04.668849 #48374]  INFO -- : - Integrations::MockMonitoring failures: 0
I, [2024-12-03T16:19:04.672727 #48374]  INFO -- : - Integrations::Prometheus failures: 0
I, [2024-12-03T16:19:04.676023 #48374]  INFO -- : - Integrations::SlackSlashCommands failures: 0
I, [2024-12-03T16:19:04.679646 #48374]  INFO -- : - Integrations::MattermostSlashCommands failures: 0
I, [2024-12-03T16:19:04.683369 #48374]  INFO -- : - Integrations::UnifyCircuit failures: 0
I, [2024-12-03T16:19:04.686503 #48374]  INFO -- : - Integrations::Pumble failures: 0
I, [2024-12-03T16:19:04.690168 #48374]  INFO -- : - Integrations::MicrosoftTeams failures: 0
I, [2024-12-03T16:19:04.693284 #48374]  INFO -- : - Integrations::HangoutsChat failures: 0
I, [2024-12-03T16:19:04.696936 #48374]  INFO -- : - Integrations::WebexTeams failures: 0
I, [2024-12-03T16:19:04.700095 #48374]  INFO -- : - Integrations::Telegram failures: 0
I, [2024-12-03T16:19:04.703793 #48374]  INFO -- : - Integrations::Mattermost failures: 0
I, [2024-12-03T16:19:04.707441 #48374]  INFO -- : - Integrations::Matrix failures: 0
I, [2024-12-03T16:19:04.710688 #48374]  INFO -- : - Integrations::BaseSlackNotification failures: 0
I, [2024-12-03T16:19:04.714403 #48374]  INFO -- : - Integrations::Discord failures: 0
I, [2024-12-03T16:19:04.717531 #48374]  INFO -- : - Integrations::Slack failures: 0
I, [2024-12-03T16:19:04.721231 #48374]  INFO -- : - Integrations::GitlabSlackApplication failures: 0
I, [2024-12-03T16:19:04.724351 #48374]  INFO -- : - Integrations::Confluence failures: 0
I, [2024-12-03T16:19:04.728052 #48374]  INFO -- : - Integrations::Zentao failures: 0
I, [2024-12-03T16:19:04.731225 #48374]  INFO -- : - Integrations::Jira failures: 0
I, [2024-12-03T16:19:04.734827 #48374]  INFO -- : - Integrations::Clickup failures: 0
I, [2024-12-03T16:19:04.738479 #48374]  INFO -- : - Integrations::Youtrack failures: 0
I, [2024-12-03T16:19:04.741575 #48374]  INFO -- : - Integrations::Ewm failures: 0
I, [2024-12-03T16:19:04.745128 #48374]  INFO -- : - Integrations::Redmine failures: 0
I, [2024-12-03T16:19:04.748240 #48374]  INFO -- : - Integrations::Phorge failures: 0
I, [2024-12-03T16:19:04.751869 #48374]  INFO -- : - Integrations::CustomIssueTracker failures: 0
I, [2024-12-03T16:19:04.754965 #48374]  INFO -- : - Integrations::Bugzilla failures: 0
I, [2024-12-03T16:19:04.758573 #48374]  INFO -- : - Integrations::MockCi failures: 0
I, [2024-12-03T16:19:04.761692 #48374]  INFO -- : - Integrations::Teamcity failures: 0
I, [2024-12-03T16:19:04.765504 #48374]  INFO -- : - Integrations::Jenkins failures: 0
I, [2024-12-03T16:19:04.768651 #48374]  INFO -- : - Integrations::DroneCi failures: 0
I, [2024-12-03T16:19:04.772187 #48374]  INFO -- : - Integrations::Buildkite failures: 0
I, [2024-12-03T16:19:04.775681 #48374]  INFO -- : - Integrations::Bamboo failures: 0
I, [2024-12-03T16:19:04.778209 #48374]  INFO -- : - ProjectSnippet failures: 0
I, [2024-12-03T16:19:04.780640 #48374]  INFO -- : - PersonalSnippet failures: 0
I, [2024-12-03T16:19:04.805932 #48374]  INFO -- : - Gitlab::BackgroundMigration::EncryptCiTriggerToken::CiTrigger failures: 0
I, [2024-12-03T16:19:04.845163 #48374]  INFO -- : - Ci::JobVariable failures: 0
I, [2024-12-03T16:19:04.848589 #48374]  INFO -- : - Ci::Trigger failures: 0
I, [2024-12-03T16:19:04.858461 #48374]  INFO -- : - Ci::GroupVariable failures: 0
I, [2024-12-03T16:19:04.867754 #48374]  INFO -- : - Ci::Variable failures: 0
I, [2024-12-03T16:19:04.877148 #48374]  INFO -- : - Ci::PipelineScheduleVariable failures: 0
I, [2024-12-03T16:19:04.886302 #48374]  INFO -- : - Ci::InstanceVariable failures: 0
I, [2024-12-03T16:19:04.891193 #48374]  INFO -- : - Ci::PipelineVariable failures: 0
I, [2024-12-03T16:19:04.902922 #48374]  INFO -- : - Operations::FeatureFlagsClient failures: 1
D, [2024-12-03T16:19:04.902989 #48374] DEBUG -- :   - Operations::FeatureFlagsClient[1]: token
I, [2024-12-03T16:19:04.932967 #48374]  INFO -- : - Clusters::AgentToken failures: 0
I, [2024-12-03T16:19:05.250917 #48374]  INFO -- : - Project failures: 0
I, [2024-12-03T16:19:05.287312 #48374]  INFO -- : - DeployToken failures: 0
I, [2024-12-03T16:19:05.296990 #48374]  INFO -- : - Group failures: 0
I, [2024-12-03T16:19:05.332193 #48374]  INFO -- : - Ci::Runner failures: 0
I, [2024-12-03T16:19:05.349875 #48374]  INFO -- : - Ci::Build failures: 0
I, [2024-12-03T16:19:05.349944 #48374]  INFO -- : Total: 6 row(s) affected
I, [2024-12-03T16:19:05.349973 #48374]  INFO -- : Done!

But trying to reset encrypted tokens I get this error:

root@myGitLab ~# VERBOSE=true MODEL_NAMES=Project,Group TOKEN_NAMES=runners_token bundle exec rake gitlab:doctor:reset_encrypted_tokens
-bash: bundle: command not found

I feel stupid because I can’t create a new instance but I have the same problem when logging in.

lreisoliveira · December 3, 2024, 4:35pm

Docker is not the problem.

What I can’t understand is that if you search here on the forum, the same problem with OpenSSL::Cipher::CipherError occurs in different versions and apparently for different reasons and the gitlab team didn’t solve it.

So, the community looks for a way to solve it and creates different ways to fix it, which will not necessarily be the solution for everyone.

And worse, with commands that don’t work, such as in my case

> VERBOSE=true MODEL_NAMES=ApplicationSetting TOKEN_NAMES=ci_jwt_signing_key,runners_registration_token,error_tracking_access_token bundle exec rake gitlab:doctor:reset_encrypted
_tokens
rake aborted!
LoadError: cannot load such file -- toogle

wtf fail toogle?

Again, my install is clean.

There is a lack of a definitive solution.

Topic		Replies	Views
Error 500 after upgrade to gitlab 13.0.3 from 12.10 How to Use GitLab	4	2649	October 7, 2020
Error 404 when accessing all projects Self-managed	0	518	November 22, 2021
HTTP(S) Error 500. JWT::DecodeError (Nil JSON web token): Self-managed	1	1681	January 23, 2019
500 Error after upgrade to Gitlab 14.7 from 14.2.2 gitlab-ce omnibus install General	5	1589	February 5, 2022
Error 500 on Admin area Upgrade upgrade , admin	5	117	October 15, 2024

Web ide -> 500

Related topics