I was using an older version of GitLab before encountering this issue. I was attempting to upgrade GitLab to the latest version and was manually updating every few releases as I was using one of the 14.x versions. During this process I noticed that I could no longer access some repositories on some projects, but not all. I tried reverting and the repositories were still erroring out (error list below). I searched for the ‘Git server, gitaly, is unavailable’ errors and the only thing that I could see was that it was updated in newer versions, so I continued to update until I reached the latest docker version. I’m still running into the errors. Please note that these errors are being encountered in the Web Gui. I am also having issues pushing to the repository. It’s as if the repository doesn’t exist.
Error List (also attaching images):
An error occurred while fetching branches. retry the search.
An error occurred while fetching tags. retry the search.
404 Page Not Found (when clicking Code->Repository)
The git server, Gitaly, is not available at this time. Please contact your administrator. (Code->Branches)
I have the repository cloned somewhere else but I’d like to not lose the Project records including issues, releases, etc. It might be easier to just clear out the repository and start from scratch but is there a way to do so without just deleting the project? As if I had just created the project and want to make an initial commit.
How “fast” were you upgrading the versions? Gitlab runs some migration tasks after upgrading to a new version, maybe you didn’t let those finish properly before upgrading again?
Possibly. How fast is too fast? I waited until I could access the web portal. Does it create backups during this process? Is there away to remove the repository from the project and then push up my own backup?
So this is just what I remember from last time when I upgraded a few versions, but the website was definitely available before alll migrations had been done. Check out this link to see how you might be able to fix the failed migrations. (If those are even the problem).
In the meantime, is there a way to just remove the repository completely from the project and then re-push the repository? As if I was creating a new project? I’d like to avoid having to recreate all of my issues, releases, etc.
Possibly. How fast is too fast? I waited until I could access the web portal.
It sounds likely that background migrations did not complete before you moved on to the next step in your upgrade path.
I suggest you check the status of background migrations - both batched background migrations and pending/failed background migrations. Upgrading GitLab without allowing background migrations to finish before moving on to the next step in the upgrade path often results in database corruption and instability.
If there’s any indication that one or more background migrations failed or did not successfully complete, I advise you revert to the version you were running before beginning the upgrade process, restore your GitLab data from backup, and proceed to do the same upgrade path making sure that background migrations have successfully completed before moving on to the next step in the upgrade path.
I already tried looking at failed migrations as I posted above. 1 resolved itself, 1 keeps failing, 1 gives a 500 error when I try to run it, and 1 is stuck at 99%.
I do not have an actual backup. It’s something I should have done but this is a personal instance and I’m currently in the process of setting up backup plans for all of my data, not just Gitlab.
Is there a way to just remove the repository completely from the project and then re-push the repository? As if I was creating a new project? I’d like to avoid having to recreate all of my issues, releases, etc.
If you remove the project you lose the issues. Pushing code cloned locally doesn’t include the issues, releases, etc. You either need to go back to the previous version that was mentioned and redo the upgrade, or you need to find a way to manually force through the migration that isn’t working.
Since you didn’t take a backup before doing the upgrade, your options are somewhat limited if not difficult.
Otherwise, new project, push code, lose issues, releases, etc.
Things like upgrades can be destructive when they go wrong, and a backup is a must if you wish to recover from something that has failed. This could have happened to me once, but rather than jump in doing an upgrade that would have failed, I read the documentation first relating to upgrading, and the upgrade paths required as well as waiting for background migrations to finish. That meant I never had to endure a destructive situation. Had I done it, I would have also had no way forward other than restore backup. Luckily in my situation though, I had backups. Although since then, I’ve done restores on new machines every six months to ensure I can recover.
@iwalker I think you misunderstood my question. I understand that recreating the project causes me to lose issues, etc, and stated that I want to avoid having to recreate everything. Is there a way to clear out the repository data that is tied to the project so I do not have have to recreate the project? In such a way that the project thinks that there is no repository as if I was creating a new project. This way I can push up my copy of the repository while keeping all of my issues, releases, etc, while repairing the repository.
If you don’t want to delete the repository, you have to do what I mentioned - find the appropriate migrate and what it actually does and apply those changes to the database. Change the status of the migrate as being complete so it no longer shows on the list. There is risk in that that may bite you in the future if it’s not done correctly. And even then it’s not guaranteed that your project still won’t be affected. Sadly I cannot help you with that, and if I had such a situation with my install, I would have reverted to a previous version and backup.
Suggest, reverting to previous version and backup - except that you don’t have one…something which should have been done before you attempted the upgrade. Unfortunate thing is you went through multiple upgrades without waiting for background migrations to finish (had you read the documentation first you wouldn’t have had this problem). Therefore it’s near impossible for you to downgrade back to the original version to repeat your upgrades and wait for migrations to finish this time around. You put yourself in a pretty difficult situation.
Use:
gitlab-rake db:migrate:status
it should show you the migrate that hasn’t finished. You then need to find the file on disk under /opt/gitlab that relates to that migrate and see what the contents of that file is. This is simple enough, take the migration number and do:
find /opt/gitlab -iname *{migrate_number}*
replace {migrate_number} with the one from the list of migrations.
Then manually apply the changes to the database if they haven’t already been done. That will require you to use gitlab-psql to connect to the database and do that. If they have, then it could just be a case of changing the status of the migrate in the database as complete so that it disappears from the list.
I can try those commands tomorrow as I’m busy today. But as an alternate solution I would like to delete the REPOSITORY and not the PROJECT. Those are two different things. I could create a Gitlab project without the Repository. The repository doesn’t get added until I create one or push one to the project. I want to remove the repository that’s in the project so I don’t have to start over.
