Dumping the contents of table "merge_request_diff_files" failed: PQgetResult() failed

How to Use GitLab
1

Problem to solve

We have noticed a persistent error when we are trying to update/upgrade our GitLab installation.

The update procedure is performing a dump of the PostgreSQL database which unfortunately fails with the following error:

Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: Dumping the contents of table "merge_request_diff_files" failed: PQgetResult() failed.
pg_dump: error: Error message from server: ERROR:  missing chunk number 0 for toast value 382085 in pg_toast_80097
pg_dump: error: The command was: COPY public.merge_request_diff_files (merge_request_diff_id, relative_order, new_file, renamed_file, deleted_file, too_large, a_mode, b_mode, new_path, old_path, diff, "binary", external_diff_offset, external_diff_size) TO stdout;
2024-01-12 16:38:40 UTC -- Dumping database failed: Failed to create compressed file '/var/opt/gitlab/backups/db/database.sql.gz' when trying to backup the main database:
 - host: '/var/opt/gitlab/postgresql'
 - port: '5432'
 - database: 'gitlabhq_production'

Steps to reproduce

We have done some research on the error and found many possible solutions but none of them have worked for us so far.

We have done a REINDEX as well as a VACUUM of the affected tables without success.

REINDEX table merge_request_diff_files;

REINDEX table pg_toast.pg_toast_80097;

VACUUM analyze merge_request_diff_files;

Since many solutions we have found suggested corrupted rows, we have also done a search for these corrupted rows and removed them but the error still appears.

This is the script we ran to iterate through the table to find the corrupted rows:

DO
$$DECLARE
   v_pk integer;
   v_t text;
BEGIN
   FOR v_pk IN SELECT merge_request_diff_id FROM merge_request_diff_files LOOP
      BEGIN
         SELECT merge_request_diff_files::text INTO v_t FROM merge_request_diff_files WHERE merge_request_diff_id = v_pk;
      EXCEPTION
         WHEN OTHERS THEN
            RAISE NOTICE 'Broken row with id = %', v_pk;  -- or go ahead and delete it
      END;
   END LOOP;
END;$$;

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • GitLab v16.11.6

Any help is appreciated because we are currently trying to upgrade the operating system our GitLab is running on due to upcoming EOL of the operating system, but cannot do so because the dump of the database fails.