Disk read 99%

Hi. After updating GitLab to version 13.6, from time to time, we began to see a lot of process activity like this:
210407 be/4 git 57.62 M/s 0.00 B/s 0.00 % 99.99 % git pack-objects --revs --thin --stdout --progress --delta-base-offset
224556 be/4 git 60.71 M/s 0.00 B/s 0.00 % 99.99 % git pack-objects --revs --thin --stdout --progress --delta-base-offset --include-tag
211154 be/4 git 42.41 M/s 0.00 B/s 0.00 % 99.99 % git pack-objects --revs --thin --stdout --progress --delta-base-offset
226986 be/4 git 42.89 M/s 0.00 B/s 0.00 % 99.99 % git pack-objects --revs --thin --stdout --progress --delta-base-offset --include-tag
219522 be/4 git 77.60 M/s 0.00 B/s 0.00 % 99.99 % git pack-objects --revs --thin --stdout --progress --delta-base-offset --include-tag
215000 be/4 git 50.24 M/s 0.00 B/s 0.00 % 99.99 % git pack-objects --revs --thin --stdout --progress --delta-base-offset
is it possible to determine what kind of tasks make this loads? After gitlab-ctl restart gitaly its ok, but after about 24 hours (usually after backup process) it makes loads again. Thanks.

1 Like

Hi, @dmitrych61. I believe this 2-year old issue has relevant information about pack-objects: git-pack-objects: excessive cpu and mem usage (#1132) · Issues · GitLab.org / gitaly · GitLab

Be sure to check the most recent comments and linked issues/epics as it might give you a clue on what triggers it and how you can limit its activity. Unfortunately I couldn’t find an easy answer for you.

I don’t see any changes in 13.6.0 that could explain this behaviour being introduced. Is it possible that the load was present before and it’s now exposed?

Hi. Thank you for reply. There is a settings:
If you want to change the number of threads used by pack-objects I think you need this omnibus setting: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template#L1057

and

# postgresql['group'] = "gitlab-psql"

please tell me which setting currently corresponds to the comment from the issue?
Thanks.

we are trying to make settings like this

gitaly['concurrency'] = [
  {
    'rpc' => "/gitaly.SmartHTTPService/PostReceivePack",
    'max_per_repo' => 2
  }, {
    'rpc' => "/gitaly.SSHService/SSHUploadPack",
    'max_per_repo' => 2
  }
]

but i think that this will reduce the speed of cloning repositories. Is it?

Thanks.

Yes, it might reduce the speed as requests will be queued once it hits the limit. You can experiment with it to see how you go.

Also, a Gitaly developer had this to say about the problem you’re experiencing:

Its either GitLab repacking to maintain a decent state for each repository on disk, or it’s like the user mentions, a git-clone(1) or git-fetch(1) . Likely it’s a repository with a not ideal shape. Not much what we can do without the details we got right now

Hi. Thank you for the reply. We found that it may be a Housekeeping, but we disable this feature in Admin area → Settings → Repository.

We also noticed that the git pack-objects --revs --thin --stdout --progress --delta-base-offset command is run for a large number of repositories after the backup process is completed. Is it possible to determine from the logs that it was the Housekeeping command that was run?

Thank you.