I have an instance of gitlab-ce 9.4.2 running with various failures from admin UI to repositories not accessible from the web interface or git clone. I tried to run a backup using gitlab-rake gitlab:backup:create but it fails with the following error
Dumping database ..
Dumping PostgreSQL database gitlabhg production ... pg dump: [archiver (db) ] query failed: ERROR: invalid page in block 84 of relation base/16385/2608
pg_dump: [archiver (db)] query was: SELECT c.tableoid, c.oid, c.relname, (SELECT pg_catalog.array_agg (acl) FROM (SELECT pg_catalog.unnest (coalesce (c.relacl, pg_catalog.acldefault (CASE WHEN c.relkind = 'S' THEN '3' ELSE 'T' END: :"char",c.r elowner))) AS acl EXCEPT SELECT pg_catalog.unnest (coalesce (pip.initprivs, pg_catalog.acldefault (CASE WHEN c.relkind = 'S' THEN '3' ELSE 'r' END::"char",c.relowner)))) as foo) AS relacl, (SELECT Pg_catalog.array_agg (acl) FROM (SELECT pg_ca talog.unnest (coalesce (pip.initprivs, pg_catalog.acldefault (CASE WHEN c.relkind = 'S' THEN '3' ELSE 'T' END: :"char",c.relowner))) AS acl EXCEPT SELECT pg_catalog.unnest (coalesce (c.relacl,pg_catalog.acldefault (CASE WHEN c.relkind = 'S' THEN
'3' ELSE 'r' END: :"char",c.relowner)))) as foo) as rrelacl, NULL AS initrelacl, NULL as initrrelacl, c.relkind, c.relnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = c.relowner) AS rolname, c.relchecks, c.relhastriggers, c.relhasindex, c.relhasrules, c.relhasoids, c.relrowsecurity, c.relforcerowsecurity, c.relfrozenxid, c.relminmxid, tc.oid AS toid, tc.relfrozenxid AS tfrozenxid, tc.relminmxid AS tminmxid, c.relpersistence, c.relispopulated, c.relreplide nt, c.relpages, CASE WHEN c.reloftype => 0 THEN c.reloftype: :pg_catalog.regtype ELSE NULL END AS reloftype, d.refobjid AS owning_tab, d.refobjsubid As owning_col, (SELECT spcname FROM pg_tablespace t WHERE t.oid = c.reltablespace) AS rel tablespace, array_remove (array_remove (c.reloptions, 'check_option=local'), 'check_option=cascaded') AS reloptions, CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text WHEN 'check_option=cascaded' = ANY (c.reloptions) THE N 'CASCADED'::text ELSE NULL END AS checkoption, tc.reloptions AS toast_reloptions, EXISTS (SELECT 1 FROM pg_attribute at LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = at.at tnum) WHERE at.attrelid = c.oid AND ((SELECT pg_catalog.array_agg (acl) FROM (SELECT pg_catalog.unnest (coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) AS acl EXCEPT SELECT pg_catalog.unnest (coalesce (pip.initprivs, pg_catalog.acid efault('c',c.relowner)))) as foo) IS NOT NULL OR (SELECT pg_catalog.array_agg (acl) FROM (SELECT pg_catalog.unnest (coalesce (pip.initprivs,pg_catalog.acldefault('c', c.relowner))) AS acl EXCEPT SELECT pg_catalog.unnest (coalesce (at.attacl,pg =
catalog.acldefault('c', c.relowner)))) as foo) IS NOT NULL OR NULL IS NOT NULL OR NULL IS NOT NULL)) AS changed_acl FROM pg_class c LEFT JOIN pg_depend d ON (c.relkind = 'S' AND d.classid = c.tableoid AND d.objid = c.oid AND d.objsubid = O AND d. refclassid = c.tableoid AND d. deptype = 'a') LEFT JOIN pg_class tc ON (c.reltoastrelid = tc.oid) LEFT JOIN pg_init privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = 0) WHERE c.relkind in ('r', 'S', 'v', 'c', 'm', 'f') ORDER BY c.oid
[FAILED]
Backup failed
I diagnosed it as a problem with PostgreSQL and went forward to see if I can fix the database before proceeding.
Here is the output when I try to reinded the database.
$ reindexdb -p 5432 -h /var/opt/gitlab/postgresql gitlabhq_production reindexdb
: reindexing of database "gitlabhq_production" failed: ERROR: failed to find parent tuple for heap-only tuple at (83,2) in table "pg depend"
I really need another opinion on how to go about fixing this error.