An error when tried to download large file using gitlab API

I try to download a file that is larger than 1 GB via gitlab API

I’m using command bellow

curl --insecure --request GET --output file.tar.xz --header “PRIVATE-TOKEN: {token}”
https://git.example.com/api/v4/projects/2165/repository/files/file.tar.xz/raw?ref=master

But I get the following error from curl command. This error does not occur when the file is small, for example, a file with a size of several MiB can be downloaded successfully.

500

We're sorry. Something went wrong on our end.


Request ID: 01HB5KGP4DWBF3QFY83PG3WS64

Try refreshing the page, or going back and attempting the action again.

Please contact your GitLab administrator if this problem persists.

Go back

Error from gitlab log

sudo gitlab-ctl tail | grep error

GRPC::DeadlineExceeded (4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer {created_time:“2023-09-27T10:21:23.009484945+02:00”, grpc_status:4, grpc_message:“Deadline Exceeded”}}):
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/grape-1.7.1/lib/grape/middleware/error.rb:39:in block in call!' /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/grape-1.7.1/lib/grape/middleware/error.rb:38:in catch’
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/grape-1.7.1/lib/grape/middleware/error.rb:38:in call!' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’
{“time”:“2023-09-27T08:21:23.018Z”,“severity”:“INFO”,“duration_s”:9.07565,“db_duration_s”:0.00102,“view_duration_s”:9.07463,“status”:500,“method”:“GET”,“path”:“/api/v4/projects/2165/repository/files/zephyr-sdk-0.16.1_linux-x86_64.tar.xz/raw”,“params”:[{“key”:“ref”,“value”:“master”}],“host”:“git1.mindmade.local”,“remote_ip”:“10.9.7.102, 127.0.0.1”,“ua”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.41”,“route”:“/api/:version/projects/:id/repository/files/:file_path/raw”,“user_id”:128,“username”:“p.konkel”,“exception.class”:“GRPC::DeadlineExceeded”,“exception.message”:“4:Deadline Exceeded. debug_error_string:{UNKNOWN:Error received from peer {created_time:"2023-09-27T10:21:23.009484945+02:00", grpc_status:4, grpc_message:"Deadline Exceeded"}}”,“exception.backtrace”:[“config/initializers/grpc_patch.rb:21:in receive_and_check_status'","config/initializers/enumerator_next_patch.rb:9:in block (2 levels) in module:EnumeratorNextPatch’”,“lib/gitlab/gitaly_client/call.rb:45:in block (3 levels) in instrument_stream'","lib/gitlab/gitaly_client/call.rb:60:in recording_request’”,“lib/gitlab/gitaly_client/call.rb:45:in block (2 levels) in instrument_stream'","lib/gitlab/gitaly_client/call.rb:44:in loop’”,“lib/gitlab/gitaly_client/call.rb:44:in block in instrument_stream'","lib/gitlab/gitaly_client/blob_service.rb:198:in each’”,“lib/gitlab/gitaly_client/blob_service.rb:198:in each'","lib/gitlab/gitaly_client/blob_service.rb:198:in consume_blob_response’”,“lib/gitlab/gitaly_client/blob_service.rb:22:in get_blob'","lib/gitlab/git/blob.rb:157:in load_all_data!'”,“app/models/blob.rb:129:in block in load_all_data!'","lib/gitlab/gitaly_client.rb:336:in allow_n_plus_1_calls’”,“app/models/blob.rb:128:in load_all_data!'","lib/api/files.rb:60:in block in content_sha’”,“lib/gitlab/cache/metrics.rb:35:in observe_cache_generation'","lib/gitlab/cache/client.rb:33:in block in fetch’”,“lib/gitlab/cache/client.rb:32:in fetch'","lib/api/files.rb:52:in content_sha’”,“lib/api/files.rb:88:in blob_data'","lib/api/files.rb:189:in block (2 levels) in class:Files’”,“lib/api/api_guard.rb:219:in call'","lib/api/api_guard.rb:219:in call’”,“lib/gitlab/middleware/memory_report.rb:13:in call'","lib/gitlab/middleware/speedscope.rb:13:in call’”,“lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call'","lib/gitlab/jira/middleware.rb:19:in call’”,“lib/gitlab/middleware/go.rb:20:in call'","lib/gitlab/etag_caching/middleware.rb:21:in call’”,“lib/gitlab/middleware/query_analyzer.rb:11:in block in call'","lib/gitlab/database/query_analyzer.rb:37:in within’”,“lib/gitlab/middleware/query_analyzer.rb:11:in call'","lib/gitlab/middleware/multipart.rb:173:in call’”,“lib/gitlab/middleware/read_only/controller.rb:50:in call'","lib/gitlab/middleware/read_only.rb:18:in call’”,“lib/gitlab/middleware/same_site_cookies.rb:27:in call'","lib/gitlab/middleware/basic_health_check.rb:25:in call’”,“lib/gitlab/middleware/handle_malformed_strings.rb:21:in call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’”,“lib/gitlab/middleware/request_context.rb:15:in call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in call’”,“config/initializers/fix_local_cache_middleware.rb:11:in call'","lib/gitlab/middleware/compressed_json.rb:44:in call’”,“lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in call’”,“lib/gitlab/metrics/requests_rack_middleware.rb:79:in call'","lib/gitlab/middleware/release_env.rb:13:in call’”],“exception.gitaly”:“{:storage=>"default", :address=>"unix:/var/opt/gitlab/gitaly/gitaly.socket", :service=>:blob_service, :rpc=>:get_blob}”,“gitaly_calls”:3,“gitaly_duration_s”:6.913759,“redis_calls”:10,“redis_allowed_cross_slot_calls”:1,“redis_duration_s”:0.00417,“redis_read_bytes”:480,“redis_write_bytes”:1028,“redis_cache_calls”:1,“redis_cache_duration_s”:0.000253,“redis_cache_write_bytes”:101,“redis_feature_flag_calls”:1,“redis_feature_flag_duration_s”:9.9e-05,“redis_feature_flag_read_bytes”:199,“redis_feature_flag_write_bytes”:62,“redis_repository_cache_calls”:1,“redis_repository_cache_duration_s”:9.5e-05,“redis_repository_cache_read_bytes”:100,“redis_repository_cache_write_bytes”:53,“redis_sessions_calls”:3,“redis_sessions_allowed_cross_slot_calls”:1,“redis_sessions_duration_s”:0.003101,“redis_sessions_read_bytes”:181,“redis_sessions_write_bytes”:594,“redis_shared_state_calls”:4,“redis_shared_state_duration_s”:0.000622,“redis_shared_state_write_bytes”:218,“db_count”:6,“db_write_count”:0,“db_cached_count”:1,“db_replica_count”:0,“db_primary_count”:6,“db_main_count”:6,“db_ci_count”:0,“db_main_replica_count”:0,“db_ci_replica_count”:0,“db_replica_cached_count”:0,“db_primary_cached_count”:1,“db_main_cached_count”:1,“db_ci_cached_count”:0,“db_main_replica_cached_count”:0,“db_ci_replica_cached_count”:0,“db_replica_wal_count”:0,“db_primary_wal_count”:0,“db_main_wal_count”:0,“db_ci_wal_count”:0,“db_main_replica_wal_count”:0,“db_ci_replica_wal_count”:0,“db_replica_wal_cached_count”:0,“db_primary_wal_cached_count”:0,“db_main_wal_cached_count”:0,“db_ci_wal_cached_count”:0,“db_main_replica_wal_cached_count”:0,“db_ci_replica_wal_cached_count”:0,“db_replica_duration_s”:0.0,“db_primary_duration_s”:0.002,“db_main_duration_s”:0.002,“db_ci_duration_s”:0.0,“db_main_replica_duration_s”:0.0,“db_ci_replica_duration_s”:0.0,“cpu_s”:8.510621,“mem_objects”:427424,“mem_bytes”:2823940064,“mem_mallocs”:279466,“mem_total_bytes”:2841037024,“pid”:116822,“worker_id”:“puma_4”,“rate_limiting_gates”:,“correlation_id”:“01HBAVMP24Y2XS8V2PGJQMAV41”,“meta.caller_id”:“GET /api/:version/projects/:id/repository/files/:file_path/raw”,“meta.remote_ip”:“10.9.7.102”,“meta.feature_category”:“source_code_management”,“meta.user”:“p.konkel”,“meta.user_id”:128,“meta.project”:“zephyr_os/mirrors/sdk-ng”,“meta.root_namespace”:“zephyr_os”,“meta.client_id”:“user/128”,“request_urgency”:“low”,“target_duration_s”:5}
{“severity”:“WARN”,“time”:“2023-09-27T08:22:06.589Z”,“message”:“Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in\nSidekiq 7.0. See Best Practices · sidekiq/sidekiq Wiki · GitHub or raise an error today\nby calling Sidekiq.strict_args! during Sidekiq initialization.\n”}
{“args”:[“/var/opt/gitlab/gitaly/run/gitaly-116415/git-exec-3327008812.d/git”,“–git-dir”,“/var/opt/gitlab/git-data/repositories/@hashed/65/32/6532ddd66812255b74142e01ea098af02a79c842687cc289381c0f3031373c7f.wiki.git”,“-c”,“gc.auto=0”,“-c”,“maintenance.auto=0”,“-c”,“core.autocrlf=input”,“-c”,“core.useReplaceRefs=false”,“-c”,“core.fsync=objects,derived-metadata,reference”,“-c”,“core.fsyncMethod=fsync”,“-c”,“core.packedRefsTimeout=10000”,“-c”,“core.filesRefLockTimeout=1000”,“-c”,“core.bigFileThreshold=50m”,“ls-tree”,“–full-tree”,“–name-status”,“-r”,“-z”,“–end-of-options”,“master”,“–”,“.”],“command.cpu_time_ms”:1,“command.exitCode”:128,“command.inblock”:0,“command.maxrss”:373076,“command.oublock”:0,“command.real_time_ms”:1,“command.system_time_ms”:0,“command.user_time_ms”:1,“component”:“gitaly.StreamServerInterceptor”,“correlation_id”:“01HBAVQ42ZPHFG5TQZTZHTX7YM”,“grpc.meta.auth_version”:“v2”,“grpc.meta.client_name”:“gitlab-web”,“grpc.meta.deadline_type”:“regular”,“grpc.meta.method_operation”:“accessor”,“grpc.meta.method_scope”:“repository”,“grpc.meta.method_type”:“server_stream”,“grpc.method”:“SearchFilesByName”,“grpc.request.deadline”:“2023-09-27T10:22:44.101”,“grpc.request.fullMethod”:“/gitaly.RepositoryService/SearchFilesByName”,“grpc.request.glProjectPath”:“zephyr_os/mirrors/zephyr.wiki”,“grpc.request.glRepository”:“wiki-2162”,“grpc.request.repoPath”:“@hashed/65/32/6532ddd66812255b74142e01ea098af02a79c842687cc289381c0f3031373c7f.wiki.git”,“grpc.request.repoStorage”:“default”,“grpc.service”:“gitaly.RepositoryService”,“grpc.start_time”:“2023-09-27T10:22:34.101”,“level”:“error”,“msg”:“fatal: Not a valid object name master\n”,“path”:“/var/opt/gitlab/gitaly/run/gitaly-116415/git-exec-3327008812.d/git”,“peer.address”:“@”,“pid”:345938,“remote_ip”:“192.168.245.183”,“span.kind”:“server”,“system”:“grpc”,“time”:“2023-09-27T08:22:34.105Z”,“user_id”:“192”,“username”:“m.pilch”}
{“severity”:“INFO”,“time”:“2023-09-27T08:22:35.128Z”,“correlation_id”:“01HBAVQ5B0FT6BXRVJA6A7997E”,“meta.caller_id”:“graphql:dismissUserCallout”,“meta.remote_ip”:“192.168.245.160”,“meta.feature_category”:“code_review_workflow”,“meta.user”:“m.mikunda”,“meta.user_id”:47,“meta.client_id”:“user/47”,“trace_type”:“execute_query”,“query_fingerprint”:“dismissUserCallout/BuQ0pIs2rNlJfARYl95dxUsqLruKyIKPlY27qmfSbE8=/1/PJEMMqYwEBf5x2o5vOAD2oGKCnTbsdHQfzNC8AGhYRo=”,“duration_s”:0.005241486011072993,“operation_name”:“dismissUserCallout”,“operation_fingerprint”:“dismissUserCallout/BuQ0pIs2rNlJfARYl95dxUsqLruKyIKPlY27qmfSbE8=”,“is_mutation”:true,“variables”:“{"input"=\u003e{"featureName"=\u003e"rich_text_editor"}}”,“query_string”:“mutation dismissUserCallout($input: UserCalloutCreateInput!) {\n userCalloutCreate(input: $input) {\n errors\n userCallout {\n dismissedAt\n featureName\n __typename\n }\n __typename\n }\n}\n”,“query_analysis.duration_s”:0.00048291985876858234,“query_analysis.depth”:3,“query_analysis.complexity”:7,“query_analysis.used_fields”:[“UserCalloutCreatePayload.errors”,“UserCallout.dismissedAt”,“UserCallout.featureName”,“UserCallout.__typename”,“UserCalloutCreatePayload.userCallout”,“UserCalloutCreatePayload.__typename”,“Mutation.userCalloutCreate”],“query_analysis.used_deprecated_fields”:}
{“method”:“POST”,“path”:“/api/graphql”,“format”:“/”,“controller”:“GraphqlController”,“action”:“execute”,“status”:200,“time”:“2023-09-27T08:22:35.129Z”,“params”:[{“key”:“operationName”,“value”:“dismissUserCallout”},{“key”:“variables”,“value”:“[FILTERED]”},{“key”:“query”,“value”:“mutation dismissUserCallout($input: UserCalloutCreateInput!) {\n userCalloutCreate(input: $input) {\n errors\n userCallout {\n dismissedAt\n featureName\n __typename\n }\n __typename\n }\n}\n”},{“key”:“graphql”,“value”:{“operationName”:“dismissUserCallout”,“variables”:“[FILTERED]”,“query”:“mutation dismissUserCallout($input: UserCalloutCreateInput!) {\n userCalloutCreate(input: $input) {\n errors\n userCallout {\n dismissedAt\n featureName\n __typename\n }\n __typename\n }\n}\n”}}],“correlation_id”:“01HBAVQ5B0FT6BXRVJA6A7997E”,“meta.caller_id”:“GraphqlController#execute”,“meta.remote_ip”:“192.168.245.160”,“meta.feature_category”:“code_review_workflow”,“meta.user”:“m.mikunda”,“meta.user_id”:47,“meta.client_id”:“user/47”,“graphql”:[{“depth”:3,“complexity”:7,“used_fields”:[“UserCalloutCreatePayload.errors”,“UserCallout.dismissedAt”,“UserCallout.featureName”,“UserCallout.__typename”,“UserCalloutCreatePayload.userCallout”,“UserCalloutCreatePayload.__typename”,“Mutation.userCalloutCreate”],“used_deprecated_fields”:,“variables”:“{"input"=>{"featureName"=>"rich_text_editor"}}”,“operation_name”:“dismissUserCallout”}],“remote_ip”:“192.168.245.160”,“user_id”:47,“username”:“m.mikunda”,“ua”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36”,“request_urgency”:“low”,“target_duration_s”:5,“redis_calls”:5,“redis_allowed_cross_slot_calls”:1,“redis_duration_s”:0.000809,“redis_read_bytes”:181,“redis_write_bytes”:705,“redis_sessions_calls”:3,“redis_sessions_allowed_cross_slot_calls”:1,“redis_sessions_duration_s”:0.000579,“redis_sessions_read_bytes”:181,“redis_sessions_write_bytes”:599,“redis_shared_state_calls”:2,“redis_shared_state_duration_s”:0.00023,“redis_shared_state_write_bytes”:106,“db_count”:6,“db_write_count”:1,“db_cached_count”:1,“db_replica_count”:0,“db_primary_count”:6,“db_main_count”:6,“db_ci_count”:0,“db_main_replica_count”:0,“db_ci_replica_count”:0,“db_replica_cached_count”:0,“db_primary_cached_count”:1,“db_main_cached_count”:1,“db_ci_cached_count”:0,“db_main_replica_cached_count”:0,“db_ci_replica_cached_count”:0,“db_replica_wal_count”:0,“db_primary_wal_count”:0,“db_main_wal_count”:0,“db_ci_wal_count”:0,“db_main_replica_wal_count”:0,“db_ci_replica_wal_count”:0,“db_replica_wal_cached_count”:0,“db_primary_wal_cached_count”:0,“db_main_wal_cached_count”:0,“db_ci_wal_cached_count”:0,“db_main_replica_wal_cached_count”:0,“db_ci_replica_wal_cached_count”:0,“db_replica_duration_s”:0.0,“db_primary_duration_s”:0.003,“db_main_duration_s”:0.003,“db_ci_duration_s”:0.0,“db_main_replica_duration_s”:0.0,“db_ci_replica_duration_s”:0.0,“cpu_s”:0.021357,“mem_objects”:11568,“mem_bytes”:724064,“mem_mallocs”:3255,“mem_total_bytes”:1186784,“pid”:116822,“worker_id”:“puma_4”,“rate_limiting_gates”:,“db_duration_s”:0.00243,“view_duration_s”:0.00012,“duration_s”:0.01057}

System version: Ubuntu 20.04

Gitlab version:

GitLabv16.4.0
GitLab Shell14.28.0
GitLab Workhorsev16.4.0
GitLab APIv4
GitLab Pages16.4.0
GitLab KASv16.4.0
Ruby3.0.6p216
Rails7.0.6
PostgreSQL (main)13.11
PostgreSQL (ci)13.11

Please help me solve the problem
Thanks