Git lfs - Access forbidden. Check your access level

Hello!

New project was created on production environemnt. I forgot to enable lfs and when pushing it didn’t work. But now for some when I do git lfs push for any project when adding some LFS file (it can be small so it isn’t some timeout issue) it keeps pushing, everytime saying “LFS: Access forbidden. Check your access level.” trying again and every time it tries for another time it makes a new folder in cache folder of lfs-objects. I know that project has LFS enabled, because if I disable it it failes at:
trace git-lfs: HTTP: POST …/info/lfs/locks/verify
trace git-lfs: HTTP: POST …/info/lfs/objects/batch

And when LFS is enabled it goes pass this but fails here:

22:48:00.649880 trace git-lfs: HTTP: {“objects”:[{“oid”:“7571a85db75aa74d7902aec1fd52d57d2f61bd48d477d319f05e7cbba46155ce”,“size”:2088566784,“actions”:{“upload”:{“href”:"…/gitlab-lfs/objects/7571a85db75aa74d7902aec1fd52d57d2f61bd48d477d319f05e7cbba46155ce/2088566784",“header”:{“Authorization”:“Basic ZGVuaXNvOiE0S3JhdG5vU3Jhbmpldkl0YWxpamkh”}}}}]}
22:48:00.649880 trace git-lfs: tq: starting transfer adapter “basic”
Git LFS: (0 of 1 files) 0 B / 1.95 GB 22:48:00.650380 trace git-lfs: HTTP: PUT …/gitlab-lfs/objects/7571a85db75aa74d7902aec1fd52d57d2f61bd48d477d319f05e7cbba46155ce/2088566784
Git LFS: (0 of 1 files) 1.95 GB / 1.95 GB 22:50:48.049854 trace git-lfs: HTTP: 403
22:50:48.049854 trace git-lfs: HTTP: {“message”:“Access forbidden. Check your access level.”,“documentation_url”:“https://git.comtrade.com/help”}
22:50:48.050354 trace git-lfs: tq: retrying object 7571a85db75aa74d7902aec1fd52d57d2f61bd48d477d319f05e7cbba46155ce: LFS: Access forbidden. Check your access level.
22:50:48.050354 trace git-lfs: tq: enqueue retry #1 for “7571a85db75aa74d7902aec1fd52d57d2f61bd48d477d319f05e7cbba46155ce” (size: 2088566784)

Any idea what could be the problem?

This still works on test environment. I’m looking for differences but can’t find them.

BR Denis

I am having this same problem on a fresh install of 11.0.3-ee (f25aa33).

Did you ever resolve your issue?

Here is a dump from my git push using:

GIT_TRACE=1 git push origin master

Output (sanitized):

09:54:37.522057 trace git-lfs: Filled credentials for https://gitlab.example.com/group/repo.git
09:54:37.572732 trace git-lfs: HTTP: POST https://gitlab.example.com/group/repo.git/info/lfs/locks/verify
09:54:38.094242 trace git-lfs: HTTP: 200
09:54:38.094334 trace git-lfs: creds: git credential approve ("https", "gitlab.example.com", "")
09:54:38.151779 trace git-lfs: HTTP: {"ours":[],"theirs":[]}
Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://gitlab.example.com/group/repo.git/info/lfs.locksverify true
09:54:38.151934 trace git-lfs: tq: running as batched queue, batch size of 100
09:54:38.152191 trace git-lfs: run_command: git rev-list --stdin --objects --not --remotes=origin --
09:54:38.157853 trace git-lfs: run_command: git cat-file --batch
09:54:38.166370 trace git-lfs: tq: sending batch of size 2
09:54:38.166639 trace git-lfs: api: batch 2 files
09:54:38.166710 trace git-lfs: creds: git credential cache ("https", "gitlab.example.com", "")
09:54:38.166721 trace git-lfs: Filled credentials for https://gitlab.example.com/group/repo.git
09:54:38.166738 trace git-lfs: HTTP: POST https://gitlab.example.com/group/repo.git/info/lfs/objects/batch
09:54:38.275712 trace git-lfs: HTTP: 200
09:54:38.275785 trace git-lfs: HTTP: {"objects":[{"oid":"a55753f12328e18ddde46310166b79066db5ed0b47ba389d38c38702469d5e70","size":348146,"actions":{"upload":{"href":"https://gitlab.example.com/group/repo.git/gitlab-lfs/objects/a55753f12328e18ddde46310166b79066db5ed0b47ba389d38c38702469d5e70/348146","header":{"Authorization":"Basic XnJhbmRvbi5mcmllc3NAaW5tYXIuY29tOjlqRTFNNzZGQzdzQksyTGptc3p0"}}}},{"oid":"c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146","size":30027,"actions":{"upload":{"href":"https://gitlab.
09:54:38.275873 trace git-lfs: HTTP: example.com/group/repo.git/gitlab-lfs/objects/c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146/30027","header":{"Authorization":"Basic XnJhbmRvbi5mcmllc3NAaW5tYXIuY29tOjlqRTFNNzZGQzdzQksyTGptc3p0"}}}}]}
09:54:38.276069 trace git-lfs: tq: starting transfer adapter "basic"
09:54:38.276375 trace git-lfs: HTTP: PUT https://gitlab.example.com/group/repo.git/gitlab-lfs/objects/a55753f12328e18ddde46310166b79066db5ed0b47ba389d38c38702469d5e70/348146
09:54:38.276695 trace git-lfs: HTTP: PUT https://gitlab.example.com/group/repo.git/gitlab-lfs/objects/c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146/30027
09:54:38.816283 trace git-lfs: HTTP: 403B | 0 B/s
09:54:38.816445 trace git-lfs: HTTP: 403
09:54:38.816526 trace git-lfs: HTTP: {"message":"Access forbidden. Check your access level.","documentation_url":"https://gitlab.example.com/help"}
09:54:38.816655 trace git-lfs: HTTP: {"message":"Access forbidden. Check your access level.","documentation_url":"https://gitlab.example.com/help"}
09:54:38.816957 trace git-lfs: tq: retrying object c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146: LFS: Access forbidden. Check your access level.
09:54:38.817043 trace git-lfs: tq: retrying object a55753f12328e18ddde46310166b79066db5ed0b47ba389d38c38702469d5e70: LFS: Access forbidden. Check your access level.
09:54:38.817064 trace git-lfs: tq: enqueue retry #1 for "c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146" (size: 30027)
09:54:38.817082 trace git-lfs: tq: enqueue retry #1 for "a55753f12328e18ddde46310166b79066db5ed0b47ba389d38c38702469d5e70" (size: 348146)
09:54:38.817134 trace git-lfs: tq: sending batch of size 2
09:54:38.817292 trace git-lfs: api: batch 2 files

Output from production.log on GitLab Server:

Started PUT "/group/repo.git/gitlab-lfs/objects/c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146/30027" for 1.2.3.4 at 2018-07-11 09:54:38 -0500

Processing by Projects::LfsStorageController#upload_finalize as HTML

Parameters: {"file.md5"=>"71149e44dbcab7b9fa16daf81aa03f41", "file.name"=>"c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146", "file.path"=>"/gitlab-data/shared/lfs-objects/tmp/uploads/c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146204444403", "file.sha1"=>"f453ed7cfa933818215bafd371d0c44a90bb6595", "file.sha256"=>"c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146", "file.sha512"=>"f894cbd80626a576cf8947aa3a84bb8d3e814013d9d61a281f8d3be4d2fccfbc71eacfb219dc723dd03da19d50f3941c1088b3613d7e6d171e1f2fbd08f2bb76", "file.size"=>"30027", "namespace_id"=>"group", "project_id"=>"repo.git", "oid"=>"c124cd4ddfdc022e0babda6aa1a538e2e51a4ba0be58397b13724bf841221146", "size"=>"30027"}

Completed 403 Forbidden in 22ms (Views: 0.1ms | ActiveRecord: 10.5ms | Elasticsearch: 0.0ms)

I am not sure where the Forbidden is coming from - nginx? github-workhorse? The user pushing is an owner on the repo so I have full permissions to write to the repo.

Any insight would be appreciated.

Brandon

Just want to update that I worked around my issue by deleting and rebuilding my GitLab server.

Having the exact same issue with 11.1.4-ce. Is there somebody who was able to fix it? We really would like to migrate our repositories to LFS but rebuilding the server just for this purpose is absolute no option.

Has anybody found a solution to this problem?

Sorry. No solution. We decided not to use LFS. :frowning: BR Denis

We have the same issue with the gitlab. Does anybody know the solution for this issue?

At our Company we have had the same situation after we had to rebuild gitlab repositories.
The LFS files are stored once in the <git_data>/lfs-object directory. This ensures only one copy of the file is stored even if it is referenced by multiple repos.

The git lfs uses the sha254sum has to determine the file structure and file name.
so if file.zip has a hash of 45dac46c5e34efca58f7d8ce0b389a905bc453de36d6f66de690a1f961752df2
The lfs files are stored two levels deep in with the structure being ‘first byte’/‘second byte’/‘the rest of it’ - ie for object 45dac46c5e34efca58f7d8ce0b389a905bc453de36d6f66de690a1f961752df2 it is /45/da/c46c5e34efca58f7d8ce0b389a905bc453de36d6f66de690a1f961752df2

The root cause of the issue is that the Gitlab database has a refrence for this hash but no corresponding file location. So when the file is pushed to gitlab the server thinks this file exists and when it can not find the file location or file in the lfs-objects directory it gives a 403 forbidden error. the gitlab server does not check if the file exists or not.

A workaround to get the file committed would be to recreate the file with a change. The hash for the resulting file needs to be different than the original file - just renaming is not enough. Since gitlab still has the hash in the databse, this doesn’t solve the original issue.

The actual steps to fix the issue are:
copy of over the files to git
use the sha256sum to get the first two directories
create them if they dont exist and copy the file to the new location
and then rename the file using the remaing part fo the hash value

1 Like