Git push fails with gitaly fsync error

How to Use GitLab Self-managed
, ,
1

Hi all,

I have a gitlab-ce installation on my self-hosted bare metal kubernetes cluster, installed through helm. I’m using gitaly for repository access, and gitaly has access to a PV through NFS. Whenever I try to push to a repository, it gives the following error: remote: fatal: fsync error on '/home/git/repositories/@hashed/d5/9e/d59eced1ded07f84c145592f65bdf854358e009c5cd705f5215bf18697fed103.git/./objects/tmp_objdir-incoming-02FNsD/2d/tmp_obj_0A1M4e': Permission denied (also through the web IDE interface). Note that it was working before the summer, so it’s possible I botched a helm upgrade.

Gitaly logs show the following:

Gitaly log 
root@gondor2013:~/helm_manifests# kubectl logs -n gitlab gitlab-gitaly-0 | grep 01H92XDD1GEPPVYS6XNTRXERNH
Defaulted container "gitaly" out of: gitaly, certificates (init), configure (init)
{"component": "gitaly","subcomponent":"gitaly","level":"info","command.count":1,"command.cpu_time_ms":2,"command.inblock":0,"command.majflt":0,"command.maxrss":346388,"command.minflt":139,"command.oublock":0,"command.real_time_ms":3,"command.spawn_token_fork_ms":0,"command.spawn_token_wait_ms":0,"command.system_time_ms":0,"command.user_time_ms":2,"component":"gitaly.UnaryServerInterceptor","correlation_id":"01H92XDD1GEPPVYS6XNTRXERNH","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_operation":"accessor","grpc.meta.method_scope":"repository","grpc.meta.method_type":"unary","grpc.method":"HasLocalBranches","grpc.request.deadline":"2023-08-30T09:47:03.045","grpc.request.fullMethod":"/gitaly.RepositoryService/HasLocalBranches","grpc.request.glProjectPath":"peterkroon/test","grpc.request.glRepository":"project-41","grpc.request.payload_bytes":124,"grpc.request.repoPath":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RepositoryService","grpc.start_time":"2023-08-30T09:46:53.416","grpc.time_ms":4.366,"level":"info","msg":"finished unary call with code OK","peer.address":"10.240.122.223:44370","pid":1,"remote_ip":"10.240.166.146","span.kind":"server","system":"grpc","time":"2023-08-30T09:46:53.420Z","user_id":"2","username":"peterkroon"}
{"component": "gitaly","subcomponent":"gitaly","level":"info","command.count":1,"command.cpu_time_ms":2,"command.inblock":0,"command.majflt":0,"command.maxrss":346388,"command.minflt":140,"command.oublock":0,"command.real_time_ms":3,"command.spawn_token_fork_ms":0,"command.spawn_token_wait_ms":0,"command.system_time_ms":0,"command.user_time_ms":2,"component":"gitaly.StreamServerInterceptor","correlation_id":"01H92XDD1GEPPVYS6XNTRXERNH","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_operation":"accessor","grpc.meta.method_scope":"repository","grpc.meta.method_type":"server_stream","grpc.method":"ListRefs","grpc.request.deadline":"2023-08-30T09:47:03.073","grpc.request.fullMethod":"/gitaly.RefService/ListRefs","grpc.request.glProjectPath":"peterkroon/test","grpc.request.glRepository":"project-41","grpc.request.payload_bytes":137,"grpc.request.repoPath":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.RefService","grpc.start_time":"2023-08-30T09:46:53.443","grpc.time_ms":3.645,"level":"info","msg":"finished streaming call with code OK","peer.address":"10.240.122.223:44370","pid":1,"remote_ip":"10.240.166.146","span.kind":"server","system":"grpc","time":"2023-08-30T09:46:53.447Z","user_id":"2","username":"peterkroon"}
{"component": "gitaly","subcomponent":"gitaly","level":"error","catfile.duration_ms":0,"catfile.flush_count":1,"catfile.flush_ms":0,"catfile.read_object_count":1,"catfile.read_object_ms":0,"catfile.request_object_count":1,"catfile.request_object_ms":0,"command.count":2,"command.cpu_time_ms":2,"command.inblock":0,"command.majflt":0,"command.maxrss":346388,"command.minflt":167,"command.oublock":0,"command.real_time_ms":3,"command.spawn_token_fork_ms":0,"command.spawn_token_wait_ms":0,"command.system_time_ms":0,"command.user_time_ms":2,"component":"gitaly.StreamServerInterceptor","correlation_id":"01H92XDD1GEPPVYS6XNTRXERNH","error":"tree entry not found","error_metadata":{"path":".gitattributes"},"grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_operation":"accessor","grpc.meta.method_scope":"repository","grpc.meta.method_type":"server_stream","grpc.method":"TreeEntry","grpc.request.deadline":"2023-08-30T09:47:23.455","grpc.request.fullMethod":"/gitaly.CommitService/TreeEntry","grpc.request.glProjectPath":"peterkroon/test","grpc.request.glRepository":"project-41","grpc.request.payload_bytes":151,"grpc.request.repoPath":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":0,"grpc.service":"gitaly.CommitService","grpc.start_time":"2023-08-30T09:46:53.455","grpc.time_ms":5.239,"level":"info","msg":"finished streaming call with code NotFound","peer.address":"10.240.122.223:44370","pid":1,"remote_ip":"10.240.166.146","span.kind":"server","system":"grpc","time":"2023-08-30T09:46:53.461Z","user_id":"2","username":"peterkroon"}
{"component": "gitaly","subcomponent":"gitaly","level":"info","branch_name":"bWFpbg==","component":"gitaly.StreamServerInterceptor","correlation_id":"01H92XDD1GEPPVYS6XNTRXERNH","diskcache":"21f198d9-0df6-4ee1-a98c-93aaca81489e","force":false,"grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_operation":"mutator","grpc.meta.method_scope":"repository","grpc.meta.method_type":"client_stream","grpc.method":"UserCommitFiles","grpc.request.deadline":"2023-08-30T09:47:48.163","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.service":"gitaly.OperationService","grpc.start_time":"2023-08-30T09:46:53.463","level":"info","msg":"diskcache state change","peer.address":"10.240.122.223:44370","pid":1,"remote_ip":"10.240.166.146","repository_relative_path":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","repository_storage":"default","span.kind":"server","start_branch_name":"bWFpbg==","start_repository_relative_path":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","start_repository_storage":"default","start_sha":"","system":"grpc","time":"2023-08-30T09:46:53.490Z","user_id":"2","username":"peterkroon"}
{"component": "gitaly","subcomponent":"gitaly","level":"fatal","branch_name":"bWFpbg==","command.count":4,"command.cpu_time_ms":9,"command.inblock":0,"command.majflt":0,"command.maxrss":346388,"command.minflt":580,"command.oublock":8,"command.real_time_ms":16,"command.spawn_token_fork_ms":0,"command.spawn_token_wait_ms":0,"command.system_time_ms":2,"command.user_time_ms":7,"component":"gitaly.StreamServerInterceptor","correlation_id":"01H92XDD1GEPPVYS6XNTRXERNH","error":"write created blob: exit status 128, stderr: \"fatal: fsync error on '/home/git/repositories/+gitaly/tmp/quarantine-f9a22eb2239ac60b-4188035681/3a/tmp_obj_nLkEjO': Permission denied\\n\"","force":false,"grpc.code":"Internal","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_operation":"mutator","grpc.meta.method_scope":"repository","grpc.meta.method_type":"client_stream","grpc.method":"UserCommitFiles","grpc.request.deadline":"2023-08-30T09:47:48.163","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.request.payload_bytes":418,"grpc.response.payload_bytes":0,"grpc.service":"gitaly.OperationService","grpc.start_time":"2023-08-30T09:46:53.463","grpc.time_ms":27.491,"level":"error","msg":"finished streaming call with code Internal","peer.address":"10.240.122.223:44370","pid":1,"remote_ip":"10.240.166.146","repository_relative_path":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","repository_storage":"default","span.kind":"server","start_branch_name":"bWFpbg==","start_repository_relative_path":"@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git","start_repository_storage":"default","start_sha":"","system":"grpc","time":"2023-08-30T09:46:53.490Z","user_id":"2","username":"peterkroon"}

I’m sure the owner and group of the files on the NFS share are correct. I would appreciate some guidance on where to look next. helm uninstall and helm install don’t help unfortunately. A second idea was to use gitlab.gitaly.git.config to set core.fsync = none, but that provided a new error (error when closing loose object file: permission denied)
I’d be happy to provide more details.

Installation details 
helm list -n gitlab 
NAME    NAMESPACE       REVISION        UPDATED                                         STATUS          CHART           APP VERSION
gitlab  gitlab          1               2023-08-30 10:31:02.349725021 +0200 CEST        deployed        gitlab-7.3.0    v16.3.0    

root@gondor2013:~/helm_manifests# kubectl describe -n gitlab pvc repo-data-gitlab-gitaly-0 
Name:          repo-data-gitlab-gitaly-0
Namespace:     gitlab
StorageClass:  
Status:        Bound
Volume:        gitaly-volume
Labels:        name=gitaly-claim
Annotations:   pv.kubernetes.io/bind-completed: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      250Gi
Access Modes:  RWO,ROX,RWX
VolumeMode:    Filesystem
Used By:       gitlab-gitaly-0
Events:        <none>
root@gondor2013:~/helm_manifests# kubectl describe pv gitaly-volume 
Name:            gitaly-volume
Labels:          name=gitaly-volume
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Bound
Claim:           gitlab/repo-data-gitlab-gitaly-0
Reclaim Policy:  Retain
Access Modes:    RWO,ROX,RWX
VolumeMode:      Filesystem
Capacity:        250Gi
Node Affinity:   <none>
Message:         
Source:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    gondor2021.bin.bioinf.nl
    Path:      /storage/git-storage
    ReadOnly:  false
Events:        <none>
2

Would you be able to attach some screenshots of the following?
1 - navigate to /home and do ll
2 - navigate to /home/git and do ll
3 - navigate to /home/git/repositories and do ll
4 - navigate to /home/git/repositories/@hashed and do ll
I think / believe changing the ownership of the directory will resolve this permission issue, just need to establish what user gitlab runs as and what user owns directory / folder in question

3

Thanks for the quick reply. Happy to provide the information you requested:

kubectl exec -n gitlab statefulsets/gitlab-gitaly -it -- bash
Defaulted container "gitaly" out of: gitaly, certificates (init), configure (init)
git@gitlab-gitaly-0:/$ id git
uid=1000(git) gid=1000(git) groups=1000(git)
`ls -la /home` 
total 0
drwxr-xr-x 1 root root 25 Aug 21 09:49 .
drwxr-xr-x 1 root root 40 Aug 30 08:31 ..
drwxr-xr-x 1 git  git  34 Aug 21 09:49 git
`ls -la /home/git` 
drwxr-xr-x 1 git  git    34 Aug 21 09:49 .
drwxr-xr-x 1 root root   25 Aug 21 09:49 ..
-rw-r--r-- 1 git  git   220 Aug 21 09:49 .bash_logout
-rw-r--r-- 1 git  git  3526 Aug 21 09:49 .bashrc
-rw-r--r-- 1 git  git   807 Aug 21 09:49 .profile
drwxrwxrwx 5 git  git    94 Aug 29 14:29 repositories
`ls -la /home/git/repositories` 
total 8
drwxr-xr-x  4 git git   42 Jul 24 13:15 +gitaly
drwxrwxrwx  5 git git   94 Aug 29 14:29 .
drwxr-xr-x  1 git git   34 Aug 21 09:49 ..
-rw-------  1 git git   64 Dec  2  2022 .gitaly-metadata
drwxr-x--- 33 git git 4096 Aug 29 14:18 @hashed
drwxr-x---  4 git git   38 Mar 30 08:55 @pools
`ls -la /home/git/repositories/@hashed` 
total 4
drwxr-x--- 33 git git 4096 Aug 29 14:18 .
drwxrwxrwx  5 git git   94 Aug 29 14:29 ..
drwxr-x---  3 git git   24 May 17 16:09 35
drwxr-x---  3 git git   24 Aug 29 14:18 3d
drwxr-x---  3 git git   24 Feb 14  2023 3f
drwxr-x---  3 git git   24 Feb 22  2023 45
drwxr-x---  3 git git   24 Dec  8  2022 4b
drwxr-x---  4 git git   38 Feb 22  2023 4e
drwxr-x---  3 git git   24 Dec 20  2022 4f
drwxr-x---  3 git git   24 Mar 27 11:31 53
drwxr-x---  3 git git   24 May 15 11:45 59
drwxr-x---  3 git git   24 Apr 24 11:46 5f
drwxr-x---  3 git git   24 May 22 09:14 62
drwxr-x---  3 git git   24 Apr 24 12:39 67
drwxr-x---  4 git git   38 Feb  1  2023 6b
drwxr-x---  3 git git   24 Mar 20 15:51 6f
drwxr-x---  3 git git   24 Aug 29 13:58 76
drwxr-x---  3 git git   24 Mar 21 10:57 78
drwxr-x---  3 git git   24 Feb 20  2023 85
drwxr-x---  3 git git   24 Aug 29 11:04 86
drwxr-x---  3 git git   24 Mar  1  2023 94
drwxr-x---  3 git git   24 Aug 29 12:19 9f
drwxr-x---  3 git git   24 Feb 21  2023 b1
drwxr-x---  3 git git   24 Apr  4 08:26 b7
drwxr-x---  3 git git   24 Apr  3 11:50 c2
drwxr-x---  3 git git   24 Jul 13 09:28 c6
drwxr-x---  3 git git   24 Dec  2  2022 d4
drwxr-x---  3 git git   24 Aug 29 14:03 d5
drwxr-x---  3 git git   24 Jun 26 13:08 e2
drwxr-x---  3 git git   24 Feb 21  2023 e6
drwxr-x---  3 git git   24 Dec 20  2022 e7
drwxr-x---  3 git git   24 Jun 19 12:42 eb
drwxr-x---  3 git git   24 Mar  8 10:39 f5
`find /home/git -not -user git`

None found

All files and directories under /home/git are owned by the user git with uid=1000.

1 Like
4

Thank you for the update, apologies for the delay on my part :slight_smile:
can you try
1 - run
sudo chown -R git: /home/git/repositories/@hashed/d5/9e
2 - are you able to push to other repositories?
Thank you

5

Doesn’t help unfortunately, and no. When I make a new repo via the webinterface it will also not create a README.

6

I’m still thinking about this, in the meantime I just noticed that it’s trying to write to /home/git whereas your PV/NFS are here /storage/git-storage, not sure if it’s related to the issue or not

7

Can it be to do with the user you’re pushing with? maybe to do with your key?

8

ok, it doesn’t create readme, but does it create a repo?

9

I don’t see how this could matter. That’s simply how NFS mounts work. To the container it really really should not matter where the data is actually stored.

git@gitlab-gitaly-0:/$ findmnt /home/git/repositories/
TARGET                 SOURCE                                        FSTYPE OPTIONS
/home/git/repositories gondor2021.bin.bioinf.nl:/storage/git-storage nfs4   rw,relatime,vers=4.2,rsize=1048576,wsize

Can it be to do with the user you’re pushing with? maybe to do with your key?

If I SSH to my gitlab installation it correctly identifies my user.
Welcome to GitLab, @peterkroon!

ok, it doesn’t create readme, but does it create a repo?

It does, at least in the webinterface.

10

Thank you for the update
1 - yeah it was just a thought :slight_smile:
2 - so you have full ssh access to the machine, but is your ssh key added to your profile under preferences > ssh keys? maybe re add your id_rsa.pub ? or any other key that you might be using?
3 - thank you for confirmation :+1:

11

2 - so you have full ssh access to the machine, but is your ssh key added to your profile under preferences > ssh keys? maybe re add your id_rsa.pub ? or any other key that you might be using?

I’m not quite sure what you mean here. I have complete access to the machines on which I host my k8s cluster; gitlab is a collection of pods on top of that.
I use a separate ssh key to do my personal git work, which does seem to be working. Also note that the web interface is also not able to write files/commits/whatevers, which bypasses ssh entirely.

12

thank you for confirming, definitely some sorcery going on there! trying to think :slight_smile:

1 Like
13

I ultimately resolved this by 1) backing up my gitlab installation; 2) Nuking my gitlab installation from orbit; 3) reinstall with a different storage class (not NFS, but rather local to the cluster); 4) restoring the backup.

1 Like