LFS Performance problem

Hi, I am new to the GitLab forums and GitLab administration in general and currently I am trying to solve an LFS performance problem.

The development site I work at is currently in the progress of switching to Git from another version control system. We tried Gitea first (classic single server installation with Ubuntu 24.04, a separate local disk for Gitea storage, 4 vCPUs, 8 GB of RAM) and want to switch to GitLab now.

As this system is supposed to be the primary vcs for this site availability is extremely important. We don’t want to deal with downtime for patch updates or because a VM crashed. That is why I modelled this setup after the 60 RPS or 3,000 users GitLab reference architecture with High Availibility (including modifications because or user count will be <100). I will give an architectural overview below.

Our problem: After initial setup of the GitLab instance (and some tests by myself) I gave a few developers access to it for testing. They quickly noticed that LFS (which we heavily use) is much slower than the single server Gitea instance (like ~2-4x slower). That is a problem for clones/pulls from LFS heavy repos (mostly ~2x penalty) but especially for pushs to the GitLab instance (~4x penalty).

One test repo for this case is about 23.5 GB of data (only ~42MB of that are code, the rest is in 23311 LFS objects), another about 11 GB (with ~17MB of code and 16247 LFS objects).

For example:
Pushing the larger repo to the Gitea server takes about 4 minutes. Pushing it to the GitLab server takes 14 minutes. All tests are run from the same VM that runs on the same Proxmox cluster with an IP in the same subnet as both Git servers.

Cloning the smaller repo takes about 1,5-2 minutes from Gitea and 4,5-5 minutes from GitLab.
Most of that time is needed for LFS downloads (cloning the repo from GitLab with GIT_LFS_SKIP_SMUDGE takes only 15s).

My question: Can anyone here give me guidance on how to properly debug why the requests take so long and on how to improve the performance? I would also be thankful for feedback about the architecture mentioned below.

What I tried so far:

I have tried debugging this already, first manually, then with AI help. What I found out so far (and how):

• more concurrent tansfers improve performance (16 instead of 8 reduces the larger repo push time from 14 to ~11 minutes, still far too slow)
• neither the VMs nor their hosts have any resource pressure

I temporarily enabled enable_puma_gvl_metrics. All 23,311 uploads succeeded. Per-request Rails logs showed:

Example: one 27,498-byte upload on app01. Storage-upload log:

{"correlation_id":"01M6GSMRHPP116DZPPR5W3H95B","duration_s":0.008172229,"level":"info","msg":"object storage upload completed","strategy":"s3v2","time":"2026-11-03T14:30:12+01:00","uploaded_bytes":27498}

Relevant fields from the matching Workhorse access record:

"correlation_id":"01M6GSMRHPP116DZPPR5W3H95B"
"duration_ms":372,"method":"PUT","status":200

Verbatim timing suffix from its Rails upload_finalize record:

"uploaded_file_upload_duration_s":0.008327316,"uploaded_file_size_bytes":27498,"rate_limiting_gates":[],"gvl_thread_wait_s":0.092467537,"db_duration_s":0.11432,"view_duration_s":0.0,"duration_s":0.18946}

These timings overlap and are not additive. Instrumentation is now disabled.

Current finding: significant per-object application overhead and directly measured GVL waiting, despite little connection queueing. I have not been able to find out, why contention differs so much between nodes.

Architecture:

The GitLab instance and the VM(s) that were used to test this as well as the Gitea server all run on the same hybrid HCI Proxmox 9 cluster. The relevant VMs all run on nodes with 2x Xeon Gold 6354 CPUs, 1.5TB of RAM and 2x100G bonds to redundant 100G switches. Storage is provided by 7 different PVE cluster nodes which run an all NVMe Ceph Cluster (5-6x 15.36TB each) that is connected to the same 100G physical network with different VLANs for VM traffic, ceph private/public and so on.
Neither the VM hosts nor the Ceph hosts are even nearly saturated in any way (not CPU, RAM, disk or network).

The GitLab objects are stored directly on the Proxmox Ceph cluster (in dedicated pools with quotas and without access to the actual Proxmox pools) via the radosgw VMs mentioned below.

The GitLab instance itself consists of 18 VMs:

flowchart TB
    U[Nutzer / CI-Runner<br/>HTTPS :443, Registry :5050]

    subgraph VPublic["public VM VLAN"]
        VIP["Public Virtual IP"]
    end

    subgraph VGitLab["internal GitLab VLAN - MTU 9000, isolated"]
        PX["HAProxy + keepalived<br/>px01–02<br/>internal VIP .20"]
        APP["Rails + Sidekiq + Registry<br/>app01–03"]
        PG["PostgreSQL 17 / Patroni / etcd<br/>pg01–03<br/>VIP .25 = Leader"]
        RD["Redis + Sentinel<br/>rd01–03"]
        PF["Praefect<br/>pf01–03"]
        GY["Gitaly<br/>gy01–03<br/>3 TB local storage"]
        RGW["Ceph RGW (S3)<br/>pf01–03"]
        MON["Prometheus + Backup-Mover<br/>mon01"]
    end

    subgraph VCeph["Proxmox ceph public VLAN"]
        CEPH[("PVE Ceph-Cluster<br/>Pools gitlab01.rgw.*<br/>10 TiB Quota")]
    end

    PBS[("Proxmox Backup Server<br/>VM-Backups + gitlab01-objects")]
    CMK["Checkmk"]

    U --> VIP --> PX
    PX -->|80 Web / 5050 Registry| APP
    APP -->|5432| PG
    APP -->|6379 / 26379| RD
    APP -->|2305 via VIP .20| PF
    PF -->|8075| GY
    PF -->|5432| PG
    APP -->|S3 :9000 via VIP .20| RGW
    GY -->|Repo-Backups S3| RGW
    RGW --> CEPH

    APP -.->|02:00 gitlab-backup| RGW
    PG -.->|pgBackRest, WAL| RGW
    RGW -.->|04:00 Objekt-Export| MON
    MON -.->|proxmox-backup-client| PBS
    VGitLab -.->|nightly VM-Backup| PBS

    MON -->|PromQL| CMK

I just setup a single server GitLab instance for testing purposes on the same hardware (8 vCPUs, 32GB RAM, 1TB disk) and tested pushs and clones for the same repository.
It produced nearly the same results as the original HA instance which leads me to rule out the architecture itself.

I’d compare the change in ruby_process_cpu_seconds_total and ruby_gvl_wait_seconds_total for each Puma process on app01 and app03 during the same upload run, along with each node’s LFS request count. GitLab documents both as per-process measurements. That would help show whether app01 is handling more requests or a Puma worker is busy despite the low VM-wide CPU average.

The single-server result is useful: it makes storage and LFS object handling the likely suspects rather than the HA layout. Since you ruled out architecture, the Puma metrics question above still matters for finding where time goes. One more comparison worth running on the single server: check whether LFS objects land on the local disk versus object storage, and compare that with how Gitea stored them. A 4x push penalty with 23k small objects often points at per-object overhead, not bandwidth. Also confirm whether LFS batch API calls are routed through the same Puma workers as regular Git traffic.