Upgrade to GitLab 13.4.0 (b0481767fe4) killed all repositories

Hi, that was not the case :frowning:

I had gitlab running on a single node, simple yum install gitlab-ci, was running for a while, then it started having some issues with upgrade (I think “registry” was added so the https keys were not loading properly) and I had to manually after every upgrade “fix” the config by pointing registry to proper https keys… then I decided to move the gitlab from host to vm for easier backup as the whole system started to be super important, so what we do is backup-etc + backup, installed it on vm, restored etc and backup (so rb files are copied to new system), restored backup and everything worked ok, did few upgrades, everything worked ok, and then this upgrade came trying to do a filesystem migration that crashed big time. I had ~25 repos there, 2 of them had wiki pages, 2-3 of them were already on the hash system and all others on the old system. After migration, all repos in the “old system” were empty (empty folders) but no repos except those that were on hashed before were visible. starting migration again was crashing with this crypto error, I then

UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE ci_runners SET token = null, token_encrypted = null;

and started the migrate again
all repos became visible now, the 2 repos with wiki had empty wiki
I checked filesystem, those 2 repos that had wiki now had repo.wiki (empty) folder in the old filesystem repo format, I tried few things to restore wiki but could not do it, the repos were also in the old format and migrate were not moving them to hashed. then I deleted the .wiki folder and run migration again, the repos migrated successfully to hashed and wiki automagically got restored…

so I’m now fully functional but can’t say I’m very confident in gitlab after the whole ordeal :frowning: so I’m manually backing up the whole VM non stop and storing those backups… not very optimal but…

1 Like