I partly resolved my issue, I managed to clear the corruption and upgrade GitLab.
What you need to do is to log into the Postgres Server and find the corrupt record and remove the link to the toast table. I followed the instructions here.
Obviously backup before doing this and backup again before doing the GitLab upgrade.
My corruption was in the ‘Events’ table, ‘data’ column. The corresponding toast table was: pg_toast_16418.
Connect to the GitLab Postgres Server: sudo gitlab-psql -d gitlabhq_production
You will need to amend and run following script to match the database table that has the corruption:
DO $f$
declare
curid INT := 0;
vdata TEXT;
badid INT;
begin
FOR badid IN SELECT id FROM events LOOP
curid = curid + 1;
if curid % 100000 = 0 then
raise notice '% rows inspected', curid;
end if;
begin
SELECT data
INTO vdata
FROM events where id = badid;
vdata := substr(vdata,1000,2000);
exception
when others then
raise notice 'data for event % is corrupt', badid;
continue;
end;
end loop;
end;
$f$;
In my case the result was: data for event 9035 is corrupt
I reindexed the toast table: REINDEX TABLE pg_toast.pg_toast_16418;
And broke the link to the corrupt toast data: UPDATE events SET data = '' WHERE id = 9035;
This cleared the corruption and allowed me to upgrade GitLab.
What I’ve not managed to do yet is install and run pg_repack to clean up the toast tables. I know nothing about Postgres but this is apparently required otherwise the Postgres VACUUM will fail. I found the following link about pg_repack and GitLab but that is as far as I’ve got.