Error:could not read block 963 in file "base/X/Z": Cannot allocate memory

Community
1

I am getting the below error in our self managed gitlab instance:

==> /var/log/gitlab/postgresql/current <==
ERROR: could not read block 0 in file “base/X/Y”: read only 4096 of 8192 bytes
ERROR: could not read block 963 in file “base/X/Z”: Cannot allocate memory

After searching several solutions, we found that the postgres table X which is merge_request_diff_files in our case is corrupted.
To fix it, we truncated that table. However, the newly created data also seems to be corrupted.

Please let me know if the corrupted table needs be created again or what is the issue here?

2

Hi @Hepzi

from the Cannot allocate memory error I would say Postgresql cannot allocate memory, I would suggest to check free memory on the server or look into Postresql troubleshooting and configuration.

3

Hi @balonik

Thank you for your response.

Server space is available.

Filesystem Size Used Avail Use% Mounted on
overlay 124G 36G 89G 29% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sdc 69G 25G 45G 36% /etc/gitlab
/dev/sda1 124G 36G 89G 29% /etc/hosts
shm 64M 16M 49M 24% /dev/shm
tmpfs 16G 12K 16G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 16G 0 16G 0% /proc/acpi
tmpfs 16G 0 16G 0% /proc/scsi
tmpfs 16G 0 16G 0% /sys/firmware

I have allocated shared_buffers = 8GB in postgresql.conf file as well.

can you please help me if I have to check anywhere else?

4

Please check free server memory (RAM), not disk space.

6

@balonik/ @ScrubsCanada : Hope this is sufficient memory.

  total        used        free      shared  buff/cache   available

Mem: 31G 7.6G 18G 267M 5.4G 22G
Swap: 0B 0B 0B

Thanks for your help

7

what version of Gitlab and PostreSQL is this on?

8

@balonik :

Gitlab version - 12.10.6

PostgreSQL version - 11.7

9

Backup is also failing now .

gitlab-backup create SKIP=builds,artifacts,registry
rake aborted!
Errno::ENOMEM: Cannot allocate memory @ io_fillbuf - fd:5 /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/actionview-6.0.2/lib/action_view/template/handlers/builder.rb
/opt/gitlab/embedded/service/gitlab-rails/config/application.rb:14:in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/Rakefile:5:in require’
/opt/gitlab/embedded/service/gitlab-rails/Rakefile:5:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
(See full trace by running task with --trace)