Pushing from local and creating new branches on GitLab web application result in 401 error after server migration and backup restore

After experiencing issues with a server that had been running the CE GitLab suite for quite a long time, I migrated to a new server. I am experiencing a problem after going through what I believe are all the steps in order to use GitLab on this new server. The observations/issues are as follows:

• I am unable to create new branches from my ‘Maintainer’ user on the GitLab web application. This returns 401 - Unauthorized
• I am unable to push any branch from my local machine. This fails with

remote: GitLab: 401 Unauthorized
To gitlab.example.com:example/sample.git
 ! [remote rejected] test-branch -> test-branch (pre-receive hook declined)
error: failed to push some refs to 'git@gitlab.example.com:example/sample.git'

• I am able to do git fetch without any problems.
• I am able to do git clone via ssh and https without any problems.

These are the steps I followed to create the backup and restore it on my new server.

1. On the original (old) server (Ubuntu 18.04) I created a backup tar file using sudo gitlab-backup create.
2. I used scp to copy the tar backup, plus the gitlab.rb and gitlab-secrets.json files to my local environment.
3. On the new server (Ubuntu 20.04), I followed GitLab’s instructions to install the latest version of GitLab.
4. I used scp to copy my backup plus config and secrets file to the new machine.
5. I adjusted the permissions:

chown git:git 1636409817_2021_11_08_14.4.2_gitlab_backup.tar
chown root:root gitlab.rb
chown root:root gitlab-secrets.json

6. The files were moved to the correct locations (/etc/gitlab for the .rb and .json files, and /var/opt/gitlab/backups for the .tar file)
7. I ran sudo gitlab-backup restore to restore the backup (there was only one file in /var/opt/gitlab/backups)
8. I ran sudo gitlab-ctl reconfigure

Once I finished that, I accessed my website and was able to sign in with my old user account and view that all of the expected projects with all of the expected source code were present.

I tried to do a git fetch for one of the repositories on my local machine and needed to adjust the known_hosts in order to access it. Once I did that, I was able to fetch all of the updates from the remote server. As mentioned above, I am also able to clone any repositories.

The only things it seems I cannot do are create branches on the GitLab web application and push branches from my local machine.

The troubleshooting has a section for commits, pushes, and clones returning 401 that states:

You need to sync your gitlab-secrets.json file with your GitLab application nodes.

I have no idea what that means, and it also only seems to be pushes that are failing for me.

What’s going wrong here. What have I missed? How can I sync my gitlab-secrets.json file with my GitLab application nodes?

You are not running a cluster. You are running a single server. So that link you found is not relevant to your 401 issue.

Check your log files under /var/log/gitlab and make sure all gitlab services are running. Try rebooting the server since sometimes the systemctl service gitlab-runsvdir needs restarting. You can either restart that as well as gitlab-ctl restart or reboot and see if it is any better.

@iwalker

Thank you so much, rebooting the server seems to have resolved the issue.

Happened to me too, after a server migration I had the same issue and restarting the docker container solved the issue.