Our Gitlab installation ran out of disk space, and after more space was added, it cannot run again. It’s probably a postgresql issue, as the log shows these lines:
==> /var/log/gitlab/postgresql/current <==
2023-06-01_07:49:44.61610 FATAL: lock file "postmaster.pid" is empty
2023-06-01_07:49:44.61612 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2023-06-01T07:49:47.682Z","message":"A worker terminated, shutting down the cluster"}
==> /var/log/gitlab/postgres-exporter/current <==
2023-06-01_07:49:48.58160 ts=2023-06-01T07:49:48.580Z caller=collector.go:196 level=error msg="collector failed" name=bgwriter duration_seconds=0.000115105 err="dial unix /var/opt/gitlab/postgresql/.s.PGSQL.5432: connect: no such file or directory"
2023-06-01_07:49:48.58164 ts=2023-06-01T07:49:48.580Z caller=server.go:74 level=info msg="Established new database connection" fingerprint=/var/opt/gitlab/postgresql:5432
2023-06-01_07:49:48.58166 ts=2023-06-01T07:49:48.581Z caller=collector.go:196 level=error msg="collector failed" name=database duration_seconds=6.7367e-05 err="dial unix /var/opt/gitlab/postgresql/.s.PGSQL.5432: connect: no such file or directory"
==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-06-01T07:49:49Z","uri":""}
==> /var/log/gitlab/postgres-exporter/current <==
2023-06-01_07:49:49.58175 ts=2023-06-01T07:49:49.581Z caller=postgres_exporter.go:714 level=error err="Error opening connection to database (host=/var/opt/gitlab/postgresql%20user=gitlab-psql%20database=gitlabhq_production): dial unix /var/opt/gitlab/postgresql/.s.PGSQL.5432: connect: no such file or directory"
==> /var/log/gitlab/gitlab-exporter/current <==
2023-06-01_07:49:49.84500 E, [2023-06-01T07:49:49.844946 #370] ERROR -- : Error connecting to the database: could not connect to server: No such file or directory
2023-06-01_07:49:49.84502 Is the server running locally and accepting
2023-06-01_07:49:49.84503 connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
2023-06-01_07:49:49.84504
2023-06-01_07:49:49.84737 E, [2023-06-01T07:49:49.847334 #370] ERROR -- : Error connecting to the database: could not connect to server: No such file or directory
2023-06-01_07:49:49.84738 Is the server running locally and accepting
2023-06-01_07:49:49.84739 connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
2023-06-01_07:49:49.84740
2023-06-01_07:49:49.84924 E, [2023-06-01T07:49:49.849201 #370] ERROR -- : Error connecting to the database: could not connect to server: No such file or directory
2023-06-01_07:49:49.84925 Is the server running locally and accepting
2023-06-01_07:49:49.84926 connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
2023-06-01_07:49:49.84926
2023-06-01_07:49:49.88796 127.0.0.1 - - [01/Jun/2023:07:49:49 UTC] "GET /database HTTP/1.1" 200 0
2023-06-01_07:49:49.88834 - -> /database
==> /var/log/gitlab/gitlab-exporter/current <==
2023-06-01_07:49:52.22971 127.0.0.1 - - [01/Jun/2023:07:49:51 UTC] "GET /sidekiq HTTP/1.1" 200 85552
2023-06-01_07:49:52.23024 - -> /sidekiq
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2023-06-01T07:49:57.234Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2023-06-01T07:49:57.393Z","message":"Starting metrics server on port 8082"}
==> /var/log/gitlab/postgresql/current <==
2023-06-01_07:49:57.93183 FATAL: lock file "postmaster.pid" is empty
2023-06-01_07:49:57.93185 HINT: Either another server is starting, or the lock file is the remnant of a previous server startup crash.
==> /var/log/gitlab/gitlab-exporter/current <==
2023-06-01_07:49:58.73363 127.0.0.1 - - [01/Jun/2023:07:49:58 UTC] "GET /ruby HTTP/1.1" 200 1081
2023-06-01_07:49:58.73505 - -> /ruby
==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-06-01T07:49:59Z","uri":""}
==> /var/log/gitlab/postgres-exporter/current <==
2023-06-01_07:50:04.58133 ts=2023-06-01T07:50:04.580Z caller=postgres_exporter.go:714 level=error err="Error opening connection to database (host=/var/opt/gitlab/postgresql%20user=gitlab-psql%20database=gitlabhq_production): dial unix /var/opt/gitlab/postgresql/.s.PGSQL.5432: connect: no such file or directory"
==> /var/log/gitlab/gitlab-exporter/current <==
2023-06-01_07:50:04.85154 E, [2023-06-01T07:50:04.849432 #370] ERROR -- : Error connecting to the database: could not connect to server: No such file or directory
2023-06-01_07:50:04.85157 Is the server running locally and accepting
2023-06-01_07:50:04.85158 connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"WARN","time":"2023-06-01T07:50:05.918Z","correlation_id":null,"message":"Cached record for ApplicationSetting couldn't be loaded, falling back to uncached record: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/opt/gitlab/postgresql/.s.PGSQL.5432\"?\n"}
The version of Gitlab is 15.11.2-ce.0, im using the docker version, run from a docker-compose file that looks like this:
version: '3.3'
services:
gitlab-ce:
ports:
- '443:443'
- '80:80'
#- '2222:22'
restart: always
environment:
GITLAB_SKIP_UNMIGRATED_DATA_CHECK: "true"
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab.example.local'
letsencrypt['enable'] = false
#letsencrypt['contact_emails'] = ['gitlab@example.local']
nginx['redirect_http_to_https'] = false
gitlab_rails['time_zone'] = 'Europe/Zagreb'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'gitlab@example.local'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.local'
registry['enable'] = false
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main:
label: 'Gitlab'
host: '10.10.0.10'
port: 389
uid: 'sAMAccountName'
method: 'plain'
bind_dn: 'CN=Gitlab,OU=Users,DC=example,DC=local'
password: 'passw0rd'
active_directory: true
allow_username_or_email_login: true
block_auto_created_users: false
base: 'DC=example,DC=local'
user_filter: ''
EOS
gitlab_rails['omniauth_enabled'] = true
gitlab_rails['omniauth_allow_single_sign_on'] = false
gitlab_rails['omniauth_block_auto_created_users'] = true
gitlab_rails['omniauth_providers'] = []
gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.example.local"
gitlab_rails['smtp_domain'] = "mail.example.local"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "gitlab@example.local"
gitlab_rails['smtp_password'] = "passw0rd"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['gitlab_shell_ssh_port'] = 2222
gitlab_rails['smtp_openssl_verify_mode'] = 'none'
volumes:
- '/srv/gitlab/config:/etc/gitlab'
- '/srv/gitlab/logs:/var/log/gitlab'
- '/srv/gitlab/data:/var/opt/gitlab'
- '/srv/gitlab/backup:/var/opt/gitlab/backups'
image: 'gitlab/gitlab-ce:15.11.2-ce.0'
From looking at the web it seemed like postgre thinks there is another one running, so I tried renaming the .s.PGSQL.5432 and the .s.PGSQL.5432.lock file in the /var/opt/gitlab/postgresql/ folder, but accidentally managed to delete the .s.PGSQL.5432 file. It seems that is now also creating errors.
Any ideas of how to come back from this?
Thanks.