Hi!
I’m trying to move gitaly storage out from all-in-one omnibus server to two dedicated gitaly server. Adding two new storages work perfectly, but gitaly cannot find source storage.
How should I configure storages to make migration work?
gitlab omnibus version gitlab-ce-15.1.0
Step-by-step:
- pick repository from default storage:
curl -s --header "Private-Token: $GL_TOKEN" --header "Content-Type: application/json" \
"$GL_API/projects?repository_storage=default" | jq '.[] | select(.name=="linux") | [.id, .repository_storage]'
[
2,
"default"
]
Run migration to storage0:
curl -s --request POST --header "PRIVATE-TOKEN: $GL_TOKEN" --header "Content-Type: application/json" \
--data '{"destination_storage_name":"storage0"}' \
"$GL_API/projects/2/repository_storage_moves" | jq
{
"id": 7,
"created_at": "2022-07-08T05:51:08.075Z",
"state": "scheduled",
"source_storage_name": "default",
"destination_storage_name": "storage0",
"project": {
"id": 2,
"description": "",
"name": "linux",
"name_with_namespace": "demo / linux",
"path": "linux",
"path_with_namespace": "demo/linux",
"created_at": "2022-07-06T08:08:28.260Z"
}
}
Migration failed:
curl -s --header "PRIVATE-TOKEN: $GL_TOKEN" "$GL_API/project_repository_storage_moves/7" | jq
{
"id": 7,
"created_at": "2022-07-08T05:51:08.075Z",
"state": "failed",
"source_storage_name": "default",
"destination_storage_name": "storage0",
"project": {
"id": 2,
"description": "",
"name": "linux",
"name_with_namespace": "demo / linux",
"path": "linux",
"path_with_namespace": "demo/linux",
"created_at": "2022-07-06T08:08:28.260Z"
}
}
there is errors in gitaly logs on storage-0 node:
could not create repository from snapshot: creating repository: extracting snapshot: first snapshot read: rpc error: code = InvalidArgument desc = GetStorageByName: no such storage: \"default\"
full log from /var/log/gitlab/gitaly/current
{"correlation_id":"01G7E5S95SYWQRYRSAR25KRQNW","error":"rpc error: code = InvalidArgument desc = GetStorageByName: no such storage: \"default\"","grpc.code":"InvalidArgument","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"server_stream","grpc.method":"GetSnapshot","grpc.request.deadline":"2022-07-08T14:51:08.137","grpc.request.fullMethod":"/gitaly.RepositoryService/GetSnapshot","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.payload_bytes":118,"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T08:51:08.138","grpc.time_ms":0.062,"level":"info","msg":"finished streaming call with code InvalidArgument","peer.address":"@","pid":7579,"span.kind":"server","system":"grpc","time":"2022-07-08T05:51:08.138Z"}
{"correlation_id":"01G7E5S95SYWQRYRSAR25KRQNW","diskcache":"5028214d-be92-48ce-b7a0-0629c45e12fa","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"unary","grpc.method":"ReplicateRepository","grpc.request.deadline":"2022-07-08T14:51:08.136","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"storage0","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T08:51:08.136","level":"info","msg":"diskcache state change","peer.address":"192.168.16.99:51414","pid":7579,"remote_ip":"192.168.16.11","span.kind":"server","system":"grpc","time":"2022-07-08T05:51:08.148Z","username":"root"}
{"command.count":1,"command.cpu_time_ms":0,"command.inblock":0,"command.majflt":0,"command.maxrss":191508,"command.minflt":136,"command.oublock":32,"command.real_time_ms":1,"command.system_time_ms":0,"command.user_time_ms":0,"correlation_id":"01G7E5S95SYWQRYRSAR25KRQNW","error":"could not create repository from snapshot: creating repository: extracting snapshot: first snapshot read: rpc error: code = InvalidArgument desc = GetStorageByName: no such storage: \"default\"","grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.meta.deadline_type":"unknown","grpc.meta.method_type":"unary","grpc.method":"ReplicateRepository","grpc.request.deadline":"2022-07-08T14:51:08.136","grpc.request.fullMethod":"/gitaly.RepositoryService/ReplicateRepository","grpc.request.glProjectPath":"demo/linux","grpc.request.glRepository":"project-2","grpc.request.payload_bytes":237,"grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"storage0","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2022-07-08T08:51:08.136","grpc.time_ms":11.55,"level":"error","msg":"finished unary call with code Internal","peer.address":"192.168.16.99:51414","pid":7579,"remote_ip":"192.168.16.11","span.kind":"server","system":"grpc","time":"2022-07-08T05:51:08.148Z","username":"root"}
Main server gitlab.rb configuration:
```ruby
...skip...
git_data_dirs({
'default' => { 'gitaly_address' => 'unix:/var/opt/gitlab/gitaly/gitaly.socket'},
'storage0' => { 'gitaly_address' => 'tcp://192.168.16.30:8075'},
'storage1' => { 'gitaly_address' => 'tcp://192.168.16.31:8075'}
})
...skip...
both additional gitaly servers have similar configuration for storage:
storage0:
..skip...
# storage0 server config
git_data_dirs({
'storage0' => {
'path' => '/opt/git-storage0'
},
})
..skip...
storage1:
..skip...
git_data_dirs({
'storage1' => {
'path' => '/opt/git-storage1'
},
})
..skip...
Default storage weight is 0 as we want to move everything out:
irb(main):030:0> ApplicationSetting.current.repository_storages_weighted
=> {"default"=>0, "storage0"=>50, "storage1"=>50}
Thanks!