After GitLab CE migration, Git operations via SSH don’t work, and personal access tokens are not accepted by APIs

General
1

First reported in this comment (and the following replies to that comment) in gitlab-org/gitlab#344876. I won’t copy-paste all the information here.

I migrated GitLab CE from one server to another. On the old server, GitLab was installed directly on the host (using Omnibus YUM/DNF package), while we utilise Docker on the new server.

Most things seem to work as expected, however, the major blockers are:

  1. new SSH keys not working at all;
  2. old SSH keys are working only for repositories created before the migration;
  3. personal access tokens are not accepted by neither REST API v4 nor GraphQL API.

The SSH keys and PATs are persisted in the database created/used/managed by GitLab.

Is there a way to fix these issues?

Thanks in advance! :pray:

2

Try using the commands from here: Maintenance Rake tasks | GitLab

Especially:

gitlab-rake gitlab:check

or with the additional sanitize parameter:

gitlab-rake gitlab:check SANITIZE=true

can usually give a hint at issues. There is also the commands for rebuilding the authorized_keys file, which probably is more relevant in this situation: Maintenance Rake tasks | GitLab

gitlab-rake gitlab:shell:setup
3 Likes
3

Hi @tukusejssirs,

In addition to iwalker’s troubleshooting steps above, can you also verify that GitLab secrets were restored on the new server?

You can use sudo gitlab-rake gitlab:doctor:secrets to check if there are any values which can’t be decrypted by the current secrets, which is usually indicative of not properly copying/restoring secrets from the old server to the new server during a GitLab instance migration.

2 Likes
4

Thanks, @iwalker and @gitlab-greg, for trying to help me out! :pray:

In the end, the issues were actually caused by the forwarding rules in iptables for the GitLab SSH port. When I migrated GitLab, some rules were not updated as required, thus it somehow reached something else or whatever. Note that these rules are not managed by me, therefore I have simply trusted the guy who manages them, however, he does not really know/understand how GitLab works.

Therefore, now it works as expected.

Thanks again for your time and help! :pray:

At least I learnt some inner GitLab stuff. :wink:

2 Likes