Upgrade Preinstall Unexpected Chunk Size when backing up the Database

Hi,

I’m getting the following error when trying to upgrade GitLab CE:

Dumping database …
Dumping PostgreSQL database gitlabhq_production … pg_dump: Dumping the contents of table “events” failed: PQgetResult() failed.
pg_dump: Error message from server: ERROR: unexpected chunk size 1893 (expected 1010) in final chunk 1 for toast value 36854 in pg_toast_16418
pg_dump: The command was: COPY public.events (id, target_type, target_id, title, data, project_id, created_at, updated_at, action, author_id) TO stdout;
[FAILED]

Which appears to be a corrupted public.events table. I don’t think I have a backup of the database where the public.events table is not corrupted.

I’ve seen a couple of posts online on how to fix the corruption (I know nothing about PostgreSQL) but I don’t know what data this table contains.

Does anyone know what this table contains, the corruption doesn’t seem to be affecting the operation of our GitLab environment?

What would be the best way to fix the corruption?

Thanks

Hello

Did you find a solution for above issue. I am also facing the same issue while taking gitlab backup

Hi Udaybhanud

I partly resolved my issue, I managed to clear the corruption and upgrade GitLab.

What you need to do is to log into the Postgres Server and find the corrupt record and remove the link to the toast table. I followed the instructions here.

Obviously backup before doing this and backup again before doing the GitLab upgrade.

My corruption was in the ‘Events’ table, ‘data’ column. The corresponding toast table was: pg_toast_16418.

Connect to the GitLab Postgres Server:
sudo gitlab-psql -d gitlabhq_production

You will need to amend and run following script to match the database table that has the corruption:

DO $f$
declare
    curid INT := 0;
    vdata TEXT;
    badid INT;
begin
FOR badid IN SELECT id FROM events LOOP
    curid = curid + 1;
    if curid % 100000 = 0 then
        raise notice '% rows inspected', curid;
    end if;
    begin
        SELECT data
        INTO vdata
        FROM events where id = badid;
        vdata := substr(vdata,1000,2000);
    exception
        when others then
            raise notice 'data for event % is corrupt', badid;
            continue;
    end;
end loop;
end;
$f$;

In my case the result was:
data for event 9035 is corrupt

I reindexed the toast table:
REINDEX TABLE pg_toast.pg_toast_16418;

And broke the link to the corrupt toast data:
UPDATE events SET data = '' WHERE id = 9035;

This cleared the corruption and allowed me to upgrade GitLab.

What I’ve not managed to do yet is install and run pg_repack to clean up the toast tables. I know nothing about Postgres but this is apparently required otherwise the Postgres VACUUM will fail. I found the following link about pg_repack and GitLab but that is as far as I’ve got.

Good Luck