Gitlab-ee:latest, postgres db doesn't seem to start?

How to Use GitLab Self-managed
1

Hi guys,

Recently started having an issue. I always use latest gitlab image and on recent update it seems that db is not starting at all and the update is failing.

Got a stacktrace with the error included, not sure if anything else is needed to get some help

Generated at 2023-05-13 20:48:47 +0000
Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:90:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:109:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::ConnectionBad: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:90:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:109:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:686:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:681:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:681:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:705:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:90:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:109:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::ConnectionBad: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:90:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:109:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:300:in `invalid!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:130:in `shell_out_compacted!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/helper.rb:54:in `shell_out!'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:52:in `block (2 levels) in <class:Execute>'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/why_run.rb:51:in `add_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:293:in `converge_by'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider/execute.rb:50:in `block in <class:Execute>'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection.rb:64:in `insert'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/declare_resource.rb:267:in `declare_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/resources.rb:36:in `bash_hide_env'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:18:in `block in class_from_file'
(eval):2:in `block in action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:304:in `compile_and_converge_action'
(eval):2:in `action_run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/provider.rb:245:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:601:in `block in run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:628:in `with_umask'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource.rb:600:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:74:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:108:in `run_all_actions'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:132:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:96:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/stepable_iterator.rb:54:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/resource_collection/resource_list.rb:94:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/runner.rb:130:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:686:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:681:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:681:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:705:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:285:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'

anyone can help me solve this?

2

Hi @arthurcloud :wave: Welcome to the GitLab community forum!

This error message:

Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'

is related to a failed database migration in GitLab.

Can you check for any pending and failed migrations:

Can you also check the PostgreSQL /var/log/gitlab/postgresql/current log (/srv/gitlab/logs/postgresql/current for Docker) log and reconfigure logs for any relevant errors?

Please share a sanitized copy of anything you find here for help interpreting it and troubleshooting further.

Thanks

1 Like
3

hey @gitlab-greg I tried checking for migrations to which I get errors that postgres is not running

docker exec gitlab gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/activerecord-6.1.7.2/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in `rescue in new_client': could not connect to server: Connection refused (ActiveRecord::ConnectionNotEstablished)
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

regarding postgres logs it gives me only:

docker exec gitlab cat /var/log/gitlab/postgresql/current
2023-05-19_20:01:15.24965 FATAL:  lock file "postmaster.pid" is empty
2023-05-19_20:01:15.24977 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.

for the reconfigure logs I get:

# Logfile created on 2023-05-19 20:02:40 +0000 by logger.rb/v1.4.3
[2023-05-19T20:02:41+00:00] INFO: Started Cinc Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded (One version per cookbook)
[2023-05-19T20:02:41+00:00] INFO: *** Cinc Client 17.10.0 ***
[2023-05-19T20:02:41+00:00] INFO: Platform: x86_64-linux
[2023-05-19T20:02:41+00:00] INFO: Cinc-client pid: 34
[2023-05-19T20:02:43+00:00] INFO: Setting the run_list to ["recipe[gitlab-ee]"] from CLI options
[2023-05-19T20:02:43+00:00] INFO: Run List is [recipe[gitlab-ee]][2023-05-19T20:04:24+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
[2023-05-19T20:02:43+00:00] INFO: Run List expands to [gitlab-ee]
[2023-05-19T20:02:43+00:00] INFO: Starting Cinc Client Run for my.domain
[2023-05-19T20:02:43+00:00] INFO: Running start handlers
[2023-05-19T20:02:43+00:00] INFO: Start handlers complete.
[2023-05-19T20:02:45+00:00] INFO: Loading cookbooks [gitlab-ee@0.0.1, package@0.1.0, gitlab@0.0.1, consul@0.1.0, patroni@0.1.0, pgbouncer@0.1.0, spamcheck@0.1.0, runit@5.1.7, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, acme@4.1.6, crond@0.1.0]
[2023-05-19T20:02:57+00:00] INFO: Skipped selecting an init system because it was explicitly disabled
[2023-05-19T20:02:57+00:00] WARN: logrotate does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: logrotate does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: puma does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: gitlab-shell does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: gitlab-workhorse does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: gitlab-pages does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: gitlab-kas does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: gitaly does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:02:57+00:00] WARN: mailroom does not have a log_group or default logdir mode defined. Setting to 0700.[2023-05-19T20:04:24+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
[2023-05-19T20:02:57+00:00] WARN: gitaly does not have a log_group or default logdir mode defined. Setting to 0700.
[2023-05-19T20:03:04+00:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2023-05-19T20:03:04+00:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2023-05-19T20:03:06+00:00] INFO: link[/opt/gitlab/service/logrotate] created
[2023-05-19T20:03:06+00:00] INFO: link[/opt/gitlab/service/redis] created
[2023-05-19T20:03:07+00:00] INFO: link[/opt/gitlab/service/gitaly] created
[2023-05-19T20:03:13+00:00] INFO: link[/opt/gitlab/service/postgresql] created
[2023-05-19T20:03:13+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] owner changed to 996
[2023-05-19T20:03:13+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] group changed to 996
[2023-05-19T20:03:13+00:00] INFO: file[/opt/gitlab/service/postgresql/supervise/status] updated atime and mtime to 2023-05-19 20:03:13 +0000
[2023-05-19T20:03:13+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] owner changed to 996
[2023-05-19T20:03:13+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] group changed to 996
[2023-05-19T20:03:13+00:00] INFO: file[/opt/gitlab/service/postgresql/log/supervise/status] updated atime and mtime to 2023-05-19 20:03:13 +0000
[2023-05-19T20:03:14+00:00] INFO: link[/opt/gitlab/service/gitlab-kas] created
[2023-05-19T20:04:24+00:00] INFO: Running queued delayed notifications before re-raising exception
[2023-05-19T20:04:24+00:00] ERROR: Running exception handlers
[2023-05-19T20:04:24+00:00] ERROR: Exception handlers complete
[2023-05-19T20:04:24+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-05-19T20:04:24+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-05-19T20:04:24+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-05-19T20:04:24+00:00] FATAL: ---------------------------------------------------------------------------------------
[2023-05-19T20:04:24+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 18) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:90:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:109:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::ConnectionBad: could not connect to server: Connection refused
	Is the server running locally and accepting
	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:90:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:109:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:95:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  ----
Ran "bash"  returned 1

does that help in any way?