Moving repositories into new gitaly-cluster fails

Hi,

I hope this is the correct forum for this kind of question.

Gitlab Version is 13.12.15-ce

We are currently in the process of migrating from a glusterfs based git storage directly on the app nodes to a gitaly cluster setup. The gitaly cluster was installed as described in the official gitlab documentation.

The setup looks like this (The gitlab part consists of 3 app nodes, a psql cluster similar to the praefect one and 4 ci runners):

The loadbalancer only has an IP in the gitlab network so it has source NAT configured for the gitlab related services. Therefor the LB also does SSL offloading and provides the client IP via X-Forwarded-For header where necessary. Gitlab included nginx does the rest with the real_ip module.

I have configured the new gitlay cluster as additional git storage and set the weight on the cluster to “100” and the on the default storage to “0”. New projects are now successfully created onto the gitaly cluster. Cloning, pushing and so on is working as expected.

Now I’ve tried to move existing repositories from the default storage to the gitlay cluster storage via an api call.

curl --request POST --header "Private-Token: TOKEN" \
     --header "Content-Type: application/json" \
     --data '{"source_storage_name":"default","destination_storage_name":"gitaly-cluster"}' \
     "https://gitlab.host/api/v4/project_repository_storage_moves"

This fails for all repositories. On the praefect nodes I can find errors like this:

{
  "correlation_id": "01FPF42AKQ605PMD59P2WPVJN0",
  "error": "rpc error: code = Internal desc = could not create repository from snapshot: new client: could not dial source: rpc error: code = Unauthenticated desc = authentication required",
  "grpc.meta.auth_version": "v2",
  "grpc.meta.client_name": "gitlab-sidekiq",
  "grpc.meta.deadline_type": "unknown",
  "grpc.method": "ReplicateRepository",
  "grpc.request.deadline": "2021-12-09T15:13:38.480Z",
  "grpc.request.fullMethod": "/gitaly.RepositoryService/ReplicateRepository",
  "grpc.service": "gitaly.RepositoryService",
  "grpc.start_time": "2021-12-09T09:13:38.480Z",
  "level": "error",
  "msg": "proxying to secondary failed",
  "peer.address": "X.X.X.250:24144",
  "pid": 5401,
  "relative_path": "@hashed/72/ba/72ba187b05e705de2dced5824d716a71872dedccf21f0c179bd2d5f2c7c974b1.git",
  "remote_ip": "X.X.X.101",
  "span.kind": "server",
  "system": "grpc",
  "time": "2021-12-09T09:13:38.518Z",
  "username": "my_user",
  "virtual_storage": "gitaly-cluster"
}

I’ve double checked gitlab-secrets.json files and praefect tokens and everything seems right. I also don’t quite get the error message, which client is needing authentication for which service?

Has anyone seen this before and can point me in the right direction? If any more information is needed I will gladly provide it.

Best regards!