Failed repository storage migration to new Gitaly servers

Hi!

I’m trying to move gitaly storage out from all-in-one omnibus server to two dedicated gitaly server. Adding two new storages work perfectly, but gitaly cannot find source storage.
How should I configure storages to make migration work?

gitlab omnibus version gitlab-ce-15.1.0

Step-by-step:

  1. pick repository from default storage:
curl -s --header "Private-Token: $GL_TOKEN" --header "Content-Type: application/json" \
         "$GL_API/projects?repository_storage=default" | jq '.[] | select(.name=="linux") | [.id, .repository_storage]'
[
  2,
  "default"
]

Run migration to storage0:

curl -s --request POST --header "PRIVATE-TOKEN: $GL_TOKEN" --header "Content-Type: application/json" \
              --data '{"destination_storage_name":"storage0"}' \
              "$GL_API/projects/2/repository_storage_moves" | jq
{
  "id": 7,
  "created_at": "2022-07-08T05:51:08.075Z",
  "state": "scheduled",
  "source_storage_name": "default",
  "destination_storage_name": "storage0",
  "project": {
    "id": 2,
    "description": "",
    "name": "linux",
    "name_with_namespace": "demo / linux",
    "path": "linux",
    "path_with_namespace": "demo/linux",
    "created_at": "2022-07-06T08:08:28.260Z"
  }
}

Migration failed:

curl -s --header "PRIVATE-TOKEN: $GL_TOKEN" "$GL_API/project_repository_storage_moves/7" | jq

{
  "id": 7,
  "created_at": "2022-07-08T05:51:08.075Z",
  "state": "failed",
  "source_storage_name": "default",
  "destination_storage_name": "storage0",
  "project": {
    "id": 2,
    "description": "",
    "name": "linux",
    "name_with_namespace": "demo / linux",
    "path": "linux",
    "path_with_namespace": "demo/linux",
    "created_at": "2022-07-06T08:08:28.260Z"
  }
}

there is errors in gitaly logs on storage-0 node:

could not create repository from snapshot: creating repository: extracting snapshot: first snapshot read: rpc error: code = InvalidArgument desc = GetStorageByName: no such storage: \"default\"

full log from /var/log/gitlab/gitaly/current

{"correlation_id":"01G7E5S95SYWQRYRSAR25KRQNW","error":"rpc error: code = InvalidArgument desc = GetStorageByName: no such storage: \"default\"","grpc.code":"InvalidArgument","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"server_stream","grpc.method":"GetSnapshot","grpc.request.deadline":"2022-07-08T14:51:08.137","grpc.request.fullMethod":"/gitaly.RepositoryService/GetSnapshot","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.payload_bytes":118,"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T08:51:08.138","grpc.time_ms":0.062,"level":"info","msg":"finished streaming call with code InvalidArgument","peer.address":"@","pid":7579,"span.kind":"server","system":"grpc","time":"2022-07-08T05:51:08.138Z"}
{"correlation_id":"01G7E5S95SYWQRYRSAR25KRQNW","diskcache":"5028214d-be92-48ce-b7a0-0629c45e12fa","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"unary","grpc.method":"ReplicateRepository","grpc.request.deadline":"2022-07-08T14:51:08.136","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"storage0","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T08:51:08.136","level":"info","msg":"diskcache state change","peer.address":"192.168.16.99:51414","pid":7579,"remote_ip":"192.168.16.11","span.kind":"server","system":"grpc","time":"2022-07-08T05:51:08.148Z","username":"root"}
{"command.count":1,"command.cpu_time_ms":0,"command.inblock":0,"command.majflt":0,"command.maxrss":191508,"command.minflt":136,"command.oublock":32,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":0,"correlation_id":"01G7E5S95SYWQRYRSAR25KRQNW","error":"could not create repository from snapshot: creating repository: extracting snapshot: first snapshot read: rpc error: code = InvalidArgument desc = GetStorageByName: no such storage: \"default\"","grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"unary","grpc.method":"ReplicateRepository","grpc.request.deadline":"2022-07-08T14:51:08.136","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.payload_bytes":237,"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"storage0","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T08:51:08.136","grpc.time_ms":11.55,"level":"error","msg":"finished unary call with code Internal","peer.address":"192.168.16.99:51414","pid":7579,"remote_ip":"192.168.16.11","span.kind":"server","system":"grpc","time":"2022-07-08T05:51:08.148Z","username":"root"}


Main server gitlab.rb configuration:

```ruby
...skip...
git_data_dirs({
  'default' => { 'gitaly_address' => 'unix:/var/opt/gitlab/gitaly/gitaly.socket'},
  'storage0' => { 'gitaly_address' => 'tcp://192.168.16.30:8075'},
  'storage1' => { 'gitaly_address' => 'tcp://192.168.16.31:8075'}
})
...skip...

both additional gitaly servers have similar configuration for storage:

storage0:

..skip...
# storage0 server config
git_data_dirs({
  'storage0' => {
    'path' => '/opt/git-storage0'
  },
})
..skip...

storage1:

..skip...
git_data_dirs({
  'storage1' => {
    'path' => '/opt/git-storage1'
  },
})
..skip...

Default storage weight is 0 as we want to move everything out:

irb(main):030:0> ApplicationSetting.current.repository_storages_weighted
=> {"default"=>0, "storage0"=>50, "storage1"=>50}

Thanks!

I try to add tcp listener for default storage:

gitaly['socket_path'] = "/var/opt/gitlab/gitaly/gitaly.socket"
gitaly['listen_addr'] = "0.0.0.0:8075"

and append default storage configuration to storage-0 node:

git_data_dirs({
  'default' => { 'gitaly_address' => 'unix:/var/opt/gitlab/gitaly/gitaly.socket'},
  'storage0' => {
    'path' => '/opt/git-storage0'
  },
})

now gitaly can find ‘default’ storage, however migration still fails:
rpc error: code = NotFound desc = GetRepoPath: not a git repository: \"/var/opt/gitlab/git-data/repositories/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git\"

{"correlation_id":"01G7E8PK4VNC3VQ83XKVVZW3WH","error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository: \"/var/opt/gitlab/git-data/repositories/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git\"","grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"server_stream","grpc.method":"GetSnapshot","grpc.request.deadline":"2022-07-08T15:42:05.730","grpc.request.fullMethod":"/gitaly.RepositoryService/GetSnapshot","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.payload_bytes":118,"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T09:42:05.732","grpc.time_ms":0.082,"level":"info","msg":"finished streaming call with code NotFound","peer.address":"@","pid":18189,"span.kind":"server","system":"grpc","time":"2022-07-08T06:42:05.733Z"}
{"correlation_id":"01G7E8PK4VNC3VQ83XKVVZW3WH","diskcache":"72923bfa-be92-4ab1-871e-b35dabb88bab","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"unary","grpc.method":"ReplicateRepository","grpc.request.deadline":"2022-07-08T15:42:05.730","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"storage0","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T09:42:05.730","level":"info","msg":"diskcache state change","peer.address":"192.168.16.99:60678","pid":18189,"remote_ip":"192.168.16.11","span.kind":"server","system":"grpc","time":"2022-07-08T06:42:05.736Z","username":"root"}
{"command.count":1,"command.cpu_time_ms":1,"command.inblock":0,"command.majflt":0,"command.maxrss":167512,"command.minflt":136,"command.oublock":32,"command.real_time_ms":1,"command.system_time_ms":1,"command.user_time_ms":0,"correlation_id":"01G7E8PK4VNC3VQ83XKVVZW3WH","error":"rpc error: code = NotFound desc = invalid source repository","grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"unary","grpc.method":"ReplicateRepository","grpc.request.deadline":"2022-07-08T15:42:05.730","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.payload_bytes":237,"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"storage0","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T09:42:05.730","grpc.time_ms":5.939,"level":"info","msg":"finished unary call with code NotFound","peer.address":"192.168.16.99:60678","pid":18189,"remote_ip":"192.168.16.11","span.kind":"server","system":"grpc","time":"2022-07-08T06:42:05.736Z","username":"root"}