After GitLab CE migration, Git operations via SSH don’t work, and personal access tokens are not accepted by APIs

First reported in this comment (and the following replies to that comment) in gitlab-org/gitlab#344876. I won’t copy-paste all the information here.


I migrated GitLab CE from one server to another. On the old server, GitLab was installed directly on the host (using Omnibus YUM/DNF package), while we utilise Docker on the new server.

Most things seem to work as expected, however, the major blockers are:

  1. new SSH keys not working at all;
  2. old SSH keys are working only for repositories created before the migration;
  3. personal access tokens are not accepted by neither REST API v4 nor GraphQL API.

The SSH keys and PATs are persisted in the database created/used/managed by GitLab.

Is there a way to fix these issues?

Thanks in advance! :pray:

Try using the commands from here: Maintenance Rake tasks | GitLab

Especially:

gitlab-rake gitlab:check

or with the additional sanitize parameter:

gitlab-rake gitlab:check SANITIZE=true

can usually give a hint at issues. There is also the commands for rebuilding the authorized_keys file, which probably is more relevant in this situation: Maintenance Rake tasks | GitLab

gitlab-rake gitlab:shell:setup
2 Likes

Hi @tukusejssirs,

In addition to iwalker’s troubleshooting steps above, can you also verify that GitLab secrets were restored on the new server?

You can use sudo gitlab-rake gitlab:doctor:secrets to check if there are any values which can’t be decrypted by the current secrets, which is usually indicative of not properly copying/restoring secrets from the old server to the new server during a GitLab instance migration.

2 Likes

Thanks, @iwalker and @gitlab-greg, for trying to help me out! :pray:

In the end, the issues were actually caused by the forwarding rules in iptables for the GitLab SSH port. When I migrated GitLab, some rules were not updated as required, thus it somehow reached something else or whatever. Note that these rules are not managed by me, therefore I have simply trusted the guy who manages them, however, he does not really know/understand how GitLab works.

Therefore, now it works as expected.

Thanks again for your time and help! :pray:

At least I learnt some inner GitLab stuff. :wink:

2 Likes