Hello dear GitLab community,
We are about to migrate from a standalone GitLab Omnibus installation to a highly available Gitaly cluster structure. During the last tests we noticed the following problem, for which we have unfortunately not been able to find any help online.
We version a specific software in a repo which is then obtained from GitLab via created releases. Obtaining the release works without problems on the productive standalone system, but with the new cluster system we get the following error messages in the Workhorse log.
We want the download to be around 3.6 GB with the release, but the subsequent curl command from the cluster always breaks off at around 2.6 GB. This happens at different times depending on the network load, but usually between 52 - 85 seconds. This makes a timeout problem rather unlikely.
If the source code is obtained via a clone command instead of via the release, the application works on the target system without any problems. This means that I can rule out any problems when migrating the data from the standalone to the cluster.
Get Releases from Cluster and Standolne System with the Filesize from the Output.
curl -vvv --header ‘Authorization: Bearer TOKEN’ -o cluster_group_project ‘https://cluster.example.com/group/project/-/archive/2023.12.102/project-2023.12.102.tar’
Filesize cluster_group_project: 2707845120
curl -vvv --header ‘Authorization: Bearer TOKEN’ -o standalone_group_project ‘https://example.com/group/project/-/archive/2023.12.102/project-2023.12.102.tar’
Filesize standalone_group_project: 3776573440
Workhorse-Log:
{
"correlation_id":"01J5R1533WA5W0X8T910DG0B1S",
"error":"SendArchive: copy 'git archive' output: rpc error: code = Unavailable desc = error reading from server: unexpected EOF",
"level":"error",
"method":"GET",
"msg":"",
"time":"2024-08-20T15:57:14+02:00",
"uri":"/group/project/-/archive/2023.12.102/project-2023.12.102.tar"
}
Gitaly-htt2debug Log:
2024-08-20 16:01:08.515888 I | http2: Framer 0xc004bb70a0: read DATA flags=END_STREAM stream=31453 len=5 data="\x00\x00\x00\x00\x00"
2024-08-20 16:01:08.515907 I | http2: Framer 0xc004bb70a0: wrote WINDOW_UPDATE len=4 (conn) incr=5
2024-08-20 16:01:08.515918 I | http2: Framer 0xc004bb70a0: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2024-08-20 16:01:08.516075 I | http2: Framer 0xc004bb70a0: wrote HEADERS flags=END_HEADERS stream=31453 len=2
2024-08-20 16:01:08.516094 I | http2: Framer 0xc004bb70a0: wrote DATA stream=31453 len=7 data="\x00\x00\x00\x00\x02\b\x01"
2024-08-20 16:01:08.516106 I | http2: Framer 0xc004bb70a0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=31453 len=2
2024-08-20 16:01:08.516668 I | http2: Framer 0xc004bb70a0: read PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2024-08-20 16:01:08.516704 I | http2: Framer 0xc004bb70a0: read WINDOW_UPDATE len=4 (conn) incr=7
2024-08-20 16:01:08.516727 I | http2: Framer 0xc004bb70a0: read PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2024-08-20 16:01:08.516743 I | http2: Framer 0xc004bb70a0: wrote PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2024-08-20 16:01:09.203164 I | http2: Framer 0xc004bb70a0: read RST_STREAM stream=30819 len=4 ErrCode=CANCEL
Gitaly-Log:
{
"catfile.duration_ms":2,
"catfile.flush_count":1,
"catfile.flush_ms":0,
"catfile.read_info_count":1,
"catfile.read_info_ms":2,
"catfile.request_info_count":1,
"catfile.request_info_ms":0,
"command.count":3,
"command.cpu_time_ms":5,
"command.inblock":0,
"command.majflt":0,
"command.maxrss":386540,
"command.minflt":655,
"command.oublock":0,
"command.real_time_ms":10,
"command.spawn_token_fork_ms":0,
"command.spawn_token_wait_ms":0,
"command.system_time_ms":2,
"command.user_time_ms":3,
"component":"gitaly.StreamServerInterceptor",
"correlation_id":"01J5R1533WA5W0X8T910DG0B1S",
"error":"context canceled",
"feature_flags":"gpg_signing mailmap_options use_resizable_semaphore_in_concurrency_limiter",
"grpc.code":"Canceled",
"grpc.meta.auth_version":"v2",
"grpc.meta.client_name":"gitlab-workhorse",
"grpc.meta.deadline_type":"none",
"grpc.meta.method_operation":"accessor",
"grpc.meta.method_scope":"repository",
"grpc.meta.method_type":"server_stream",
"grpc.method":"GetArchive",
"grpc.request.fullMethod":"/gitaly.RepositoryService/GetArchive",
"grpc.request.glProjectPath":"group/project",
"grpc.request.glRepository":"project-236",
"grpc.request.payload_bytes":169,
"grpc.request.repoPath":"@cluster/repositories/2c/0f/2702",
"grpc.request.repoStorage":"Gitaly1",
"grpc.response.payload_bytes":2708786356,
"grpc.service":"gitaly.RepositoryService",
"grpc.start_time":"2024-08-20T15:55:50.534",
"grpc.time_ms":318669,
"level":"info",
"msg":"finished streaming call with code Canceled",
"pid":932,
"span.kind":"server",
"system":"grpc",
"time":"2024-08-20T14:01:09.203Z"
}
If you would like the configuration files for the respective servers, I can provide them. Do you have any idea why this might be? We would be grateful for any ideas.
If you have any further questions, please do not hesitate to contact me.
Kind regards,
Arthur