The problem with set of CI/CD Variables

GitLab CI/CD
1

Problem to solve

I use the latest version (17.6) of self-managed GitLab in Docker.
I’m not sure how long, but I cannot create CI/CD Variable in any projects.
The request about creating return error 500.

gitlab    | ==> /var/log/gitlab/gitlab-rails/production.log <==
gitlab    |   
gitlab    | OpenSSL::Cipher::CipherError ():
gitlab    |   
gitlab    | encryptor (3.0.0) lib/encryptor.rb:98:in `final'
gitlab    | encryptor (3.0.0) lib/encryptor.rb:98:in `crypt'
gitlab    | encryptor (3.0.0) lib/encryptor.rb:49:in `decrypt'
gitlab    | lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
gitlab    | app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in `decrypt_token'
gitlab    | app/models/concerns/token_authenticatable_strategies/encrypted.rb:77:in `get_encrypted_token'
gitlab    | app/models/concerns/token_authenticatable_strategies/encrypted.rb:112:in `token_set?'
gitlab    | app/models/concerns/token_authenticatable_strategies/base.rb:46:in `ensure_token'
gitlab    | app/models/concerns/token_authenticatable_strategies/encrypted.rb:37:in `ensure_token'
gitlab    | app/models/concerns/token_authenticatable.rb:56:in `block in add_authentication_token_field'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:686:in `catch'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:686:in `block in default_terminator'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:200:in `block in halting'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:116:in `block in run_callbacks'
gitlab    | lib/gitlab/database/query_analyzers/prevent_cross_database_modification.rb:29:in `temporary_ignore_tables_in_transaction'
gitlab    | app/models/concerns/cross_database_ignored_tables.rb:8:in `temporary_ignore_cross_database_tables'
gitlab    | app/models/concerns/cross_database_ignored_tables.rb:50:in `temporary_ignore_cross_database_tables'
gitlab    | app/models/concerns/cross_database_ignored_tables.rb:30:in `block in register_ignored_cross_database_event'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `instance_exec'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:929:in `_run_save_callbacks'
gitlab    | activerecord (7.0.8.4) lib/active_record/callbacks.rb:455:in `create_or_update'
gitlab    | activerecord (7.0.8.4) lib/active_record/timestamp.rb:126:in `create_or_update'
gitlab    | activerecord (7.0.8.4) lib/active_record/persistence.rb:615:in `save'
gitlab    | activerecord (7.0.8.4) lib/active_record/validations.rb:47:in `save'
gitlab    | activerecord (7.0.8.4) lib/active_record/transactions.rb:298:in `block in save'
gitlab    | activerecord (7.0.8.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
gitlab    | activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:314:in `transaction'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
gitlab    | lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
gitlab    | lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
gitlab    | lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
gitlab    | activerecord (7.0.8.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
gitlab    | activerecord (7.0.8.4) lib/active_record/transactions.rb:298:in `save'
gitlab    | activerecord (7.0.8.4) lib/active_record/suppressor.rb:50:in `save'
gitlab    | activerecord (7.0.8.4) lib/active_record/persistence.rb:771:in `block in update'
gitlab    | activerecord (7.0.8.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
gitlab    | activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
gitlab    | activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
gitlab    | activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
gitlab    | activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
gitlab    | activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
gitlab    | activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
gitlab    | activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
gitlab    | lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
gitlab    | lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
gitlab    | lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
gitlab    | lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
gitlab    | activerecord (7.0.8.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
gitlab    | activerecord (7.0.8.4) lib/active_record/persistence.rb:769:in `update'
gitlab    | app/services/ci/change_variables_service.rb:6:in `execute'
gitlab    | app/controllers/projects/variables_controller.rb:23:in `update'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
gitlab    | actionpack (7.0.8.4) lib/abstract_controller/base.rb:215:in `process_action'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal/rendering.rb:165:in `process_action'
gitlab    | actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:234:in `block in process_action'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:118:in `block in run_callbacks'
gitlab    | app/controllers/application_controller.rb:503:in `set_current_admin'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | lib/gitlab/session.rb:11:in `with_session'
gitlab    | app/controllers/application_controller.rb:494:in `set_session_storage'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | lib/gitlab/i18n.rb:114:in `with_locale'
gitlab    | lib/gitlab/i18n.rb:120:in `with_user_locale'
gitlab    | app/controllers/application_controller.rb:485:in `set_locale'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | app/controllers/application_controller.rb:478:in `set_current_context'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:32:in `block in sentry_around_action'
gitlab    | sentry-ruby (5.21.0) lib/sentry/hub.rb:108:in `with_child_span'
gitlab    | sentry-ruby (5.21.0) lib/sentry-ruby.rb:499:in `with_child_span'
gitlab    | sentry-rails (5.21.0) lib/sentry/rails/controller_transaction.rb:18:in `sentry_around_action'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'
gitlab    | actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'
gitlab    | activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'
gitlab    | activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'
gitlab    | activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'
gitlab    | activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'
gitlab    | actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'
gitlab    | actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `block in dispatch'
gitlab    | lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `dispatch'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:48:in `serve'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'
gitlab    | gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'
gitlab    | omniauth (2.1.0) lib/omniauth/strategy.rb:470:in `call_app!'
gitlab    | omniauth-saml (2.2.1) lib/omniauth/strategies/saml.rb:86:in `other_phase'
gitlab    | omniauth (2.1.0) lib/omniauth/strategy.rb:195:in `call!'
gitlab    | omniauth (2.1.0) lib/omniauth/strategy.rb:169:in `call'
gitlab    | flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'
gitlab    | flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'
gitlab    | lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
gitlab    | lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'
gitlab    | lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'
gitlab    | lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'
gitlab    | lib/gitlab/middleware/memory_report.rb:13:in `call'
gitlab    | lib/gitlab/middleware/speedscope.rb:13:in `call'
gitlab    | lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
gitlab    | lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
gitlab    | lib/gitlab/etag_caching/middleware.rb:21:in `call'
gitlab    | lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
gitlab    | lib/gitlab/metrics/web_transaction.rb:46:in `run'
gitlab    | lib/gitlab/metrics/rack_middleware.rb:16:in `call'
gitlab    | lib/gitlab/middleware/go.rb:21:in `call'
gitlab    | lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
gitlab    | lib/gitlab/database/query_analyzer.rb:83:in `within'
gitlab    | lib/gitlab/middleware/query_analyzer.rb:11:in `call'
gitlab    | lib/ci/job_token/middleware.rb:11:in `call'
gitlab    | batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'
gitlab    | rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'
gitlab    | apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'
gitlab    | lib/gitlab/middleware/multipart.rb:173:in `call'
gitlab    | rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'
gitlab    | warden (1.2.9) lib/warden/manager.rb:36:in `block in call'
gitlab    | warden (1.2.9) lib/warden/manager.rb:34:in `catch'
gitlab    | warden (1.2.9) lib/warden/manager.rb:34:in `call'
gitlab    | rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'
gitlab    | rack (2.2.10) lib/rack/tempfile_reaper.rb:15:in `call'
gitlab    | rack (2.2.10) lib/rack/etag.rb:27:in `call'
gitlab    | rack (2.2.10) lib/rack/conditional_get.rb:40:in `call'
gitlab    | rack (2.2.10) lib/rack/head.rb:12:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'
gitlab    | lib/gitlab/middleware/read_only/controller.rb:50:in `call'
gitlab    | lib/gitlab/middleware/read_only.rb:18:in `call'
gitlab    | lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'
gitlab    | rack (2.2.10) lib/rack/session/abstract/id.rb:266:in `context'
gitlab    | rack (2.2.10) lib/rack/session/abstract/id.rb:260:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'
gitlab    | lib/gitlab/middleware/strip_cookies.rb:29:in `call'
gitlab    | lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
gitlab    | activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'
gitlab    | sentry-rails (5.21.0) lib/sentry/rails/rescued_exception_interceptor.rb:14:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
gitlab    | lib/gitlab/middleware/path_traversal_check.rb:35:in `call'
gitlab    | lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
gitlab    | sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'
gitlab    | sentry-ruby (5.21.0) lib/sentry/hub.rb:265:in `with_session_tracking'
gitlab    | sentry-ruby (5.21.0) lib/sentry-ruby.rb:412:in `with_session_tracking'
gitlab    | sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'
gitlab    | sentry-ruby (5.21.0) lib/sentry/hub.rb:59:in `with_scope'
gitlab    | sentry-ruby (5.21.0) lib/sentry-ruby.rb:392:in `with_scope'
gitlab    | sentry-ruby (5.21.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
gitlab    | lib/gitlab/middleware/basic_health_check.rb:25:in `call'
gitlab    | lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'
gitlab    | railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'
gitlab    | activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'
gitlab    | activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'
gitlab    | activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'
gitlab    | railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
gitlab    | lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
gitlab    | lib/gitlab/middleware/request_context.rb:15:in `call'
gitlab    | lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
gitlab    | request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'
gitlab    | rack (2.2.10) lib/rack/method_override.rb:24:in `call'
gitlab    | rack (2.2.10) lib/rack/runtime.rb:22:in `call'
gitlab    | rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'
gitlab    | rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'
gitlab    | rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'
gitlab    | config/initializers/fix_local_cache_middleware.rb:11:in `call'
gitlab    | lib/gitlab/middleware/compressed_json.rb:44:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'
gitlab    | lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
gitlab    | rack (2.2.10) lib/rack/sendfile.rb:110:in `call'
gitlab    | lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
gitlab    | lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
gitlab    | gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call'
gitlab    | gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context'
gitlab    | gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'
gitlab    | actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
gitlab    | railties (7.0.8.4) lib/rails/engine.rb:530:in `call'
gitlab    | railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'
gitlab    | railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'
gitlab    | lib/gitlab/middleware/release_env.rb:12:in `call'
gitlab    | rack (2.2.10) lib/rack/urlmap.rb:74:in `block in call'
gitlab    | rack (2.2.10) lib/rack/urlmap.rb:58:in `each'
gitlab    | rack (2.2.10) lib/rack/urlmap.rb:58:in `call'
gitlab    | puma (6.4.3) lib/puma/configuration.rb:272:in `call'
gitlab    | puma (6.4.3) lib/puma/request.rb:100:in `block in handle_request'
gitlab    | puma (6.4.3) lib/puma/thread_pool.rb:378:in `with_force_shutdown'
gitlab    | puma (6.4.3) lib/puma/request.rb:99:in `handle_request'
gitlab    | puma (6.4.3) lib/puma/server.rb:464:in `process_client'
gitlab    | puma (6.4.3) lib/puma/server.rb:245:in `block in run'
gitlab    | puma (6.4.3) lib/puma/thread_pool.rb:155:in `block in spawn_thread'

image
image368×1030 49.5 KB

I have tried:

  • gitlab-rake gitlab:doctor:secrets and I got
    • Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration failures: 2
    • WebHook failures: 5
    • Integration failures: 2
    • User failures: 1
    • ApplicationSetting failures: 1
    • ServiceHook failures: 1
    • ProjectHook failures: 4
    • Integrations::EmailsOnPush failures: 1
    • Integrations::BaseCi failures: 1
    • Integrations::Jenkins failures: 1
    • Ci::JobVariable failures: 332
    • Ci::PipelineVariable failures: 2
    • Operations::FeatureFlagsClient failures: 1
    • Ci::Build failures: 2105

I have continued by this Troubleshooting GitLab backups | GitLab
By this

DELETE FROM ci_group_variables;
DELETE FROM ci_variables;

But it does not help.

Another features (e.g. creating access tokens) works fine.

Steps to reproduce

  • Create CI/CD Variable by WebUI.

Versions

  • Self-managed
  • Self-hosted Runners

System information
System:
Current User: git
Using RVM: no
Ruby Version: 3.2.5
Gem Version: 3.5.23
Bundler Version:2.5.11
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.2.4
Go Version: unknown

GitLab information
Version: 17.6.0
Revision: 746f630acb1
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.11
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: sam

2

I have the same issue too. It’s causing me to fail the upgrade to 17.7.1, causing the error on a migration task:

main: == 20241017160505 RegenerateCiJobTokenSigningKey: migrating ===================
main: == [advisory_lock_connection] object_id: 48320, pg_backend_pid: 12271
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
/opt/gitlab/gitlab/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:244:in `attr_decrypt'
/opt/gitlab/gitlab/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:333:in `attr_decrypt'
/opt/gitlab/gitlab/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:163:in `block (2 levels) in attr_encrypted'
/opt/gitlab/gitlab/db/migrate/20241017160505_regenerate_ci_job_token_signing_key.rb:23:in `block in up'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:71:in `each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:229:in `loop'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:70:in `find_each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/querying.rb:22:in `find_each'
/opt/gitlab/gitlab/db/migrate/20241017160505_regenerate_ci_job_token_signing_key.rb:22:in `up'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:908:in `public_send'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:908:in `exec_migration'
/opt/gitlab/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/gitlab/lib/gitlab/database/query_analyzer.rb:83:in `within'
/opt/gitlab/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/gitlab/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:889:in `block (2 levels) in migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:888:in `block in migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:887:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1095:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1409:in `block in execute_migration_in_transaction'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1382:in `each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1331:in `block in migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1331:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1166:in `up'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1141:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <top (required)>'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

Caused by:
OpenSSL::Cipher::CipherError:
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
/opt/gitlab/gitlab/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:244:in `attr_decrypt'
/opt/gitlab/gitlab/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:333:in `attr_decrypt'
/opt/gitlab/gitlab/vendor/gems/attr_encrypted/lib/attr_encrypted.rb:163:in `block (2 levels) in attr_encrypted'
/opt/gitlab/gitlab/db/migrate/20241017160505_regenerate_ci_job_token_signing_key.rb:23:in `block in up'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:71:in `each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:229:in `loop'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/relation/batches.rb:70:in `find_each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/querying.rb:22:in `find_each'
/opt/gitlab/gitlab/db/migrate/20241017160505_regenerate_ci_job_token_signing_key.rb:22:in `up'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:908:in `public_send'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:908:in `exec_migration'
/opt/gitlab/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
/opt/gitlab/gitlab/lib/gitlab/database/query_analyzer.rb:83:in `within'
/opt/gitlab/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
/opt/gitlab/gitlab/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:889:in `block (2 levels) in migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:888:in `block in migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:887:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1095:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1409:in `block in execute_migration_in_transaction'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activesupport-7.0.8.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/opt/gitlab/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1382:in `each'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1331:in `block in migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/opt/gitlab/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1331:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1166:in `up'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/migration.rb:1141:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8.6/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <top (required)>'
/opt/gitlab/gitlab/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'