Upgrade 17.2.2 -> 17.3 fails at DB migration

lug-gh · August 16, 2024, 7:31am

Hi, I am using Gitlab-CE as Docker using docker-compose file.
I always update the latest update within a few days, which worked well for years, but this time it fails.
I have already executed the permission update command, it runs through without errors, but does not fix the problem.

services:
  gitlab-ce:
    hostname: gitlab.example.org
    ports:
      - '443:443'
      - '80:80'
      - '23:22'
    container_name: gitlab
    restart: always
    volumes:
      - './config:/etc/gitlab'
      - './logs:/var/log/gitlab'
      - './data:/var/opt/gitlab'
    image: 'gitlab/gitlab-ce:17.2.2'
#    command: update-permissions

Here is the log from the container start:

gitlab   | Recipe: gitlab::database_migrations
gitlab   |   * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
gitlab   |   * rails_migration[gitlab-rails] action run[2024-08-16T06:47:17+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
gitlab   | 
gitlab   |     * bash_hide_env[migrate gitlab-rails database] action run
gitlab   |       [execute] rake aborted!
gitlab   |                 StandardError: An error has occurred, this and all later migrations canceled:
gitlab   |                 
gitlab   |                 PG::InternalError: ERROR:  no unpinned buffers available
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:92:in `queue_batched_background_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240704030747_queue_backfill_new_audit_event_tables.rb:13:in `up'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
gitlab   |                 /opt/gitlab/embedded/bin/bundle:25:in `load'
gitlab   |                 /opt/gitlab/embedded/bin/bundle:25:in `<main>'
gitlab   |                 
gitlab   |                 Caused by:
gitlab   |                 ActiveRecord::StatementInvalid: PG::InternalError: ERROR:  no unpinned buffers available
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:92:in `queue_batched_background_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240704030747_queue_backfill_new_audit_event_tables.rb:13:in `up'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
gitlab   |                 /opt/gitlab/embedded/bin/bundle:25:in `load'
gitlab   |                 /opt/gitlab/embedded/bin/bundle:25:in `<main>'
gitlab   |                 
gitlab   |                 Caused by:
gitlab   |                 PG::InternalError: ERROR:  no unpinned buffers available
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/batched_background_migration_helpers.rb:92:in `queue_batched_background_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20240704030747_queue_backfill_new_audit_event_tables.rb:13:in `up'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/query_analyzer.rb:40:in `within'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:21:in `exec_migration'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:97:in `run'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:28:in `block in with_advisory_lock_connection'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/pg_backend_pid.rb:25:in `with_advisory_lock_connection'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:138:in `configure_database'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
gitlab   |                 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
gitlab   |                 /opt/gitlab/embedded/bin/bundle:25:in `load'
gitlab   |                 /opt/gitlab/embedded/bin/bundle:25:in `<main>'
gitlab   |                 Tasks: TOP => db:migrate
gitlab   |                 (See full trace by running task with --trace)
gitlab   |                 Running db:migrate rake task
gitlab   |                 main: == [advisory_lock_connection] object_id: 49720, pg_backend_pid: 394
gitlab   |                 main: == 20240704030747 QueueBackfillNewAuditEventTables: migrating =================
gitlab   |                 main: == [advisory_lock_connection] object_id: 49720, pg_backend_pid: 394
gitlab   |       
gitlab   |       ================================================================================
gitlab   |       Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]'
gitlab   |       ================================================================================
gitlab   |       
gitlab   |       Mixlib::ShellOut::ShellCommandFailed
gitlab   |       ------------------------------------
gitlab   |       Expected process to exit with [0], but received '1'

lug-gh · August 16, 2024, 7:54am

PostgreSQL’s shared memory buffers is set to 256 MB by default: files/gitlab-config-template/gitlab.rb.template · master · GitLab.org / omnibus-gitlab · GitLab

Set postgresql['shared_buffers'] = "512MB" in gitlab.rb.

This should be enough. If not: Increase.

c-holtermann · August 20, 2024, 6:08pm

I came across this issue today as well. Implementing the suggested fix worked for me like this:

docker run --env GITLAB_OMNIBUS_CONFIG='postgresql["shared_buffers"] = "256MB";'

All other parameters need to be added. I wanted to highlight the --env GITLAB_OMNIBUS_CONFIG which I had no been aware of.

razor7 · August 21, 2024, 5:37pm

Thanks! in my case using docker-compose looks like this

        environment:
            GITLAB_OMNIBUS_CONFIG: |
...
...
...
                postgresql['shared_buffers'] = '512MB'
...
...
...

ilya.evseev · September 18, 2024, 2:48am

In my own case:

Not works, ignored:
postgresql[‘shared_buffers’] = “512MB” in gitlab.rb

Not works, ignored:
GITLAB_OMNIBUS_CONFIG in docker-compose.yml

Not works, always overwritten by “1M” value:
shared_buffers = 512MB in postgresql.conf

Works:
shared_buffers = 512MB in postgresql.auto.conf