ERROR no partition of relation "audit_events" found for row

GitLab Version: V174.1

Deployment Method: docker-compose

Operating System: CentOS 7.9

My deployed GitLab has been running normally for over 5 years. Early this morning, while executing the backup command docker exec -t gitlab gitlab-backup create via a scheduled task script, the following error occurred.

''' # Omitted the normal output above

{"command":"create","gl_project_path":"GROUP_NAME/PROJECT_NAME-1","level":"info","msg":"completed create","pid":26753,"relative_path":"@hashed/d6/f0/d6f0c71ef0c88e45e4b3a2118fcb83b0def392d759c901e9d755d0e879028727.git","storage_name":"default","time":"2024-10-07T19:19:14.748Z"}

{"level":"error","msg":"create: pipeline: 1 failures encountered:\n - @hashed/c8/37/c837649cce43f2729138e72cc315207057ac82599a59be72765a477f22d14a54.git (GROUP_NAME/PROJECT_NAME-2): manager: write bundle: remote repository: create bundle: rpc error: code = Internal desc = unexpected EOF\n","pid":26753,"time":"2024-10-07T19:19:14.753Z"}

e[36m2024-10-08 03:19:14 +0800 -- Deleting tar staging files ... e[0m

e[36m2024-10-08 03:19:14 +0800 -- Cleaning up /var/opt/gitlab/backups/dbe[0m

e[36m2024-10-08 03:19:14 +0800 -- Cleaning up /var/opt/gitlab/backups/repositoriese[0m

e[36m2024-10-08 03:19:15 +0800 -- Deleting tar staging files ... donee[0m

e[36m2024-10-08 03:19:15 +0800 -- Deleting backups/tmp ... e[0m

e[36m2024-10-08 03:19:15 +0800 -- Deleting backups/tmp ... donee[0m

2024-10-08 03:19:15 +0800 e[34m-- Deleting backup and restore PID file at [e[0m/opt/gitlab/embedded/service/gitlab-rails/tmp/backup_restore.pide[34m] ... e[0me[32mdonee[0m

rake aborted!

Backup::Error: gitaly-backup exit status 1

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:49:in `finish!'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/targets/repositories.rb:33:in `dump'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/tasks/task.rb:25:in `backup!'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:57:in `run_create_task'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:145:in `block in run_all_create_tasks'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:145:in `each_value'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:145:in `run_all_create_tasks'

/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:32:in `create'

/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `block in create_backup'

/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:75:in `lock_backup'

/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `create_backup'

/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:117:in `block (3 levels) in <top (required)>'

/opt/gitlab/embedded/bin/bundle:25:in `load'

/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Tasks: TOP => gitlab:backup:create

(See full trace by running task with --trace)

Since I enabled two-factor authentication, when logging in, after entering the two-factor code, the page displays the following error:

500: We're sorry, something went wrong on our end

Request ID: 01J9P20SGP91HV03ZA2312MCRR

Try refreshing the page, or going back and attempting the action again.

Please contact your GitLab administrator if this problem persists.

The program log shows the following:

==> /var/log/gitlab/gitlab-rails/application_json.log <==

{"severity":"INFO","time":"2024-10-08T11:35:52.299Z","correlation_id":"01J9NYM1NYD8XATQJ7X8V7EDGP","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.remote_ip":"MY_IP","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","message":"Successful Login: username=root ip=MY_IP method=two-factor admin=true"}

==> /var/log/gitlab/gitlab-rails/audit_json.log <==

{"severity":"INFO","time":"2024-10-08T11:35:52.299Z","correlation_id":"01J9NYM1NYD8XATQJ7X8V7EDGP","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.remote_ip":"MY_IP","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","author_id":1,"author_name":"Administrator","entity_id":1,"entity_type":"User","created_at":"2024-10-08T19:35:52.299+08:00","with":"two-factor","target_id":1,"target_type":"User","target_details":"Administrator"}

==> /var/log/gitlab/postgresql/current <==

2024-10-08_11:35:52.30460 ERROR: no partition of relation "audit_events" found for row

2024-10-08_11:35:52.30461 DETAIL: Partition key of the failing row contains (created_at) = (2024-10-08 11:35:52.299554).

2024-10-08_11:35:52.30462 STATEMENT: /*application:web,correlation_id:01J9NYM1NYD8XATQJ7X8V7EDGP,endpoint_id:SessionsController#create,db_config_name:main*/ INSERT INTO "audit_events" ("author_id", "entity_id", "entity_type", "details", "author_name", "target_details", "created_at", "target_type", "target_id") VALUES (1, 1, 'User', '---

2024-10-08_11:35:52.30463 :with: two-factor

2024-10-08_11:35:52.30463 :target_id: 1

2024-10-08_11:35:52.30464 :target_type: User

2024-10-08_11:35:52.30464 :target_details: Administrator

2024-10-08_11:35:52.30464 :author_name: Administrator

2024-10-08_11:35:52.30464 ', 'Administrator', 'Administrator', '2024-10-08 11:35:52.299554', 'User', 1) RETURNING "id"

==> /var/log/gitlab/gitlab-rails/exceptions_json.log <==

{"severity":"ERROR","time":"2024-10-08T11:35:52.306Z","correlation_id":"01J9NYM1NYD8XATQJ7X8V7EDGP","meta.caller_id":"SessionsController#create","meta.feature_category":"system_access","meta.remote_ip":"MY_IP","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","exception.class":"ActiveRecord::StatementInvalid","exception.message":"PG::CheckViolation: ERROR: no partition of relation \"audit_events\" found for row\nDETAIL: Partition key of the failing row contains (created_at) = (2024-10-08 11:35:52.299554).\n","exception.backtrace":["activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params'","activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'","activesupport (7.0.8.4) lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'","activesupport (7.0.8.4) lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'","activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'","activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'","activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'","activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'","marginalia (1.11.1) lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'","activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `exec_query'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:132:in `exec_insert'","activerecord (7.0.8.4) lib/active_record/connection_adapters/postgresql/database_statements.rb:88:in `exec_insert'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:167:in `insert'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","activerecord (7.0.8.4) lib/active_record/persistence.rb:496:in `_insert_record'","activerecord (7.0.8.4) lib/active_record/persistence.rb:1098:in `_create_record'","activerecord (7.0.8.4) lib/active_record/counter_cache.rb:166:in `_create_record'","activerecord (7.0.8.4) lib/active_record/locking/optimistic.rb:84:in `_create_record'","activerecord (7.0.8.4) lib/active_record/attribute_methods/dirty.rb:222:in `_create_record'","activerecord (7.0.8.4) lib/active_record/callbacks.rb:459:in `block in _create_record'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:929:in `_run_create_callbacks'","activerecord (7.0.8.4) lib/active_record/callbacks.rb:459:in `_create_record'","activerecord (7.0.8.4) lib/active_record/timestamp.rb:108:in `_create_record'","activerecord (7.0.8.4) lib/active_record/persistence.rb:1069:in `create_or_update'","activerecord (7.0.8.4) lib/active_record/callbacks.rb:455:in `block in create_or_update'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:107:in `run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:929:in `_run_save_callbacks'","activerecord (7.0.8.4) lib/active_record/callbacks.rb:455:in `create_or_update'","activerecord (7.0.8.4) lib/active_record/timestamp.rb:126:in `create_or_update'","activerecord (7.0.8.4) lib/active_record/persistence.rb:648:in `save!'","activerecord (7.0.8.4) lib/active_record/validations.rb:53:in `save!'","activerecord (7.0.8.4) lib/active_record/transactions.rb:302:in `block in save!'","activerecord (7.0.8.4) lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'","activesupport (7.0.8.4) lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'","activerecord (7.0.8.4) lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'","lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'","activerecord (7.0.8.4) lib/active_record/transactions.rb:350:in `with_transaction_returning_status'","activerecord (7.0.8.4) lib/active_record/transactions.rb:302:in `save!'","activerecord (7.0.8.4) lib/active_record/suppressor.rb:54:in `save!'","app/services/audit_event_service.rb:149:in `save_or_track'","app/services/audit_event_service.rb:126:in `log_security_event_to_database'","app/services/audit_event_service.rb:54:in `security_event'","app/controllers/sessions_controller.rb:282:in `log_audit_event'","app/controllers/sessions_controller.rb:85:in `block in create'","devise (4.9.3) app/controllers/devise/sessions_controller.rb:22:in `create'","app/controllers/sessions_controller.rb:69:in `create'","actionpack (7.0.8.4) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'","actionpack (7.0.8.4) lib/abstract_controller/base.rb:215:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/rendering.rb:165:in `process_action'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:234:in `block in process_action'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:118:in `block in run_callbacks'","app/controllers/application_controller.rb:484:in `set_current_admin'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:475:in `set_session_storage'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_controller.rb:466:in `set_locale'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:459:in `set_current_context'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","sentry-rails (5.19.0) lib/sentry/rails/controller_transaction.rb:30:in `block in sentry_around_action'","sentry-ruby (5.19.0) lib/sentry/hub.rb:102:in `with_child_span'","sentry-ruby (5.19.0) lib/sentry-ruby.rb:498:in `with_child_span'","sentry-rails (5.19.0) lib/sentry/rails/controller_transaction.rb:16:in `sentry_around_action'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'","activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'","actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'","actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'","actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'","actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `block in dispatch'","lib/gitlab/middleware/action_controller_static_context.rb:23:in `call'","actionpack (7.0.8.4) lib/action_controller/metal.rb:249:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:18:in `block in \u003cclass:Constraints\u003e'","actionpack (7.0.8.4) lib/action_dispatch/routing/mapper.rb:48:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/go.rb:21:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.2) lib/rack/cors.rb:102:in `call'","rack (2.2.9) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.9) lib/rack/etag.rb:27:in `call'","rack (2.2.9) lib/rack/conditional_get.rb:40:in `call'","rack (2.2.9) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.9) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.9) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/strip_cookies.rb:29:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.19.0) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:35:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.19.0) lib/sentry/rack/capture_exceptions.rb:30:in `block (2 levels) in call'","sentry-ruby (5.19.0) lib/sentry/hub.rb:258:in `with_session_tracking'","sentry-ruby (5.19.0) lib/sentry-ruby.rb:411:in `with_session_tracking'","sentry-ruby (5.19.0) lib/sentry/rack/capture_exceptions.rb:21:in `block in call'","sentry-ruby (5.19.0) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.19.0) lib/sentry-ruby.rb:391:in `with_scope'","sentry-ruby (5.19.0) lib/sentry/rack/capture_exceptions.rb:20:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'","rack (2.2.9) lib/rack/method_override.rb:24:in `call'","rack (2.2.9) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:154:in `block in call'","rack-timeout (0.7.0) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.7.0) lib/rack/timeout/core.rb:153:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.9) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.36.1) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.36.1) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","railties (7.0.8.4) lib/rails/engine.rb:530:in `call'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:13:in `call'","rack (2.2.9) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.9) lib/rack/urlmap.rb:58:in `each'","rack (2.2.9) lib/rack/urlmap.rb:58:in `call'","puma (6.4.0) lib/puma/configuration.rb:272:in `call'","puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'","puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'","puma (6.4.0) lib/puma/request.rb:99:in `handle_request'","puma (6.4.0) lib/puma/server.rb:443:in `process_client'","puma (6.4.0) lib/puma/server.rb:241:in `block in run'","puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"],"exception.cause_class":"PG::CheckViolation","exception.sql":"/*application:web,correlation_id:01J9NYM1NYD8XATQJ7X8V7EDGP,endpoint_id:SessionsController#create,db_config_name:main*/ INSERT INTO \"audit_events\" (\"author_id\", \"entity_id\", \"entity_type\", \"details\", \"author_name\", \"target_details\", \"created_at\", \"target_type\", \"target_id\") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING \"id\"","user.username":"root","tags.program":"web","tags.locale":"zh_CN","tags.feature_category":"system_access","tags.correlation_id":"01J9NYM1NYD8XATQJ7X8V7EDGP","extra.audit_event_type":"AuditEvent"}

How should I fix this issue?

Searching for this error message led me to Issue with upgrading from 13.3.6 to 13.4 because of PG migration error (#5671) · Issues · GitLab.org / omnibus-gitlab · GitLab which says that a record in that table might be in the future. Suggest reviewing the steps, queries and see if that helps fix the problem.

Maybe the system’s host time drifted into the future, too.

Indeed, I accidentally changed the system time to 2019 using date -s @1571277985. I have since corrected the time, but I still cannot log in using two-factor authentication.

Additionally, I also checked the content in the link above. I queried the data in PostgreSQL, and there are no audit events from the future. What should I investigate next?

Indeed, I accidentally changed the system time to 2019 using date -s @1571277985. I have since corrected the time, but I still cannot log in using two-factor authentication.

Additionally, I also checked the content in the link above. I queried the data in PostgreSQL, and there are no audit events from the future. What should I investigate next?

Maybe the host time is still out of sync. I’d suggest investigating whether NTP or systemd-timesyncd is configured and running. More modern Linux distributions switched to systemd by default AFAIK.

I’m also curious why you manually changed the time on the host? Was that a debugging experiment?

Create a DB backup. Then find this row and delete it, try again to see if it helps solve the 2FA problem.

1 Like

Changing the time was a mistake. I originally just wanted to use date -d to convert the timestamp into a normal date for easier viewing, but I accidentally executed date -s, which caused the system time to be modified.

After noticing the incorrect time, the NTPD service did not synchronize the time back to the correct one. So, I manually set the time to the current time and then restarted the NTPD service. Fortunately, after several hours of automatic synchronization, I checked again after receiving your reply. Two-factor authentication is now working correctly again. Thank you so much!

Ah, thanks for sharing. I am not alone in that nightmare :slight_smile:

Awesome, glad it solved itself.

Tip: If you have not already, add a monitoring check for NTP time drifts.