Getting "GitLab: Push operation timed out" pushing a repository to local GitLab

Cross-posted to Stack Overflow (https://stackoverflow.com/questions/67265757)

I’ve created a local GitLab instance (GitLab 13.10 in Ubuntu 20) and want to import but not mirror a project from GitHub. In this instance openwrt. I created the project in the GitLab instance and then tried to push to it.

I ran:

git clone https://git.openwrt.org/openwrt/openwrt.git
git remote rename origin old-origin
git remote add origin http://gitlab.local/group/project.git
git push -u origin --all

And I got:

Counting objects: 533132, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (142241/142241), done.
Writing objects: 100% (533132/533132), 162.16 MiB | 6.35 MiB/s, done.
Total 533132 (delta 369475), reused 530420 (delta 368128)
remote: Resolving deltas: 100% (369475/369475), done.
remote: Checking connectivity: 533132, done.
remote: GitLab: Push operation timed out
remote:
remote: Timing information for debugging purposes:
remote: Running checks for ref: master
remote: Checking if you are allowed to push... (22.92ms)
remote: Checking if default branch is being deleted... (0.77ms)
remote: Checking if you are force pushing to a protected branch... (579.48ms)
remote: Checking if you are allowed to push to the protected branch... (21.38ms)
remote: Scanning repository for blobs stored in LFS and verifying their files have been uploaded to GitLab... (cancelled after 29409.76ms)
To http://gitlab.local/group/project.git
 ! [remote rejected]       master -> master (pre-receive hook declined)
error: failed to push some refs to 'http://gitlab.local/group/project.git'

I’ve done the same with another project and the push worked fine, so I think there’s something it doesn’t like with this project. There are 9210 files in the project, so I’ve assumed it’s something to do with lots of files.

Things I’ve tried:

  • I’ve gone through just about every log I can find, but there are no errors on the server.
  • I’ve increased all the timeouts I could find, but it didn’t seem to help
  • Increased memory to 8192MB
  • Increased the puma worker max memory from 865MB to 4096MB
  • Using git instead of http (tried that first…)
  • setting the lfs endpoint to use http rather than https (git config --add lfs.url "http://gitlab.local/group/project.git/info/lfs" though I’m not sure this did anything, setting it to “wibble:wobble” didn’t make it fail in any spectacular way)
  • Googling the errors
  • Searching the GitLab forums and issues for hits (Unable to import a very large project with LFS objects into Gitlab-CE instance (#326893) · Issues · GitLab.org / GitLab · GitLab looks interesting, but I don’t think mine is that large)

I don’t think there should be any files in there on LFS, the largest file is 4KB. But since the error mentions LFS I thought I’d play with those settings.

I finally found this in the gitaly logs:

{"correlation_id":"01F46PRCPB7Y470Q3HFJNK4S2X","error":"rpc error: code = Canceled desc = signal: terminated","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type
":"regular","grpc.method":"GetNewLFSPointers","grpc.request.deadline":"2021-04-26T09:04:58.013Z","grpc.request.fullMethod":"/gitaly.BlobService/GetNewLFSPointers","grpc.request.glProjectPath":"group/project","grpc.request.glReposit
ory":"project-2","grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.BlobService","
grpc.start_time":"2021-04-26T09:04:28.214Z","grpc.time_ms":30047.092,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":12659,"remote_ip":"127.0.0.1","span.kind":"server","system":"grpc","time":"2021
-04-26T09:04:58.261Z","username":"<me>"}

But it doesn’t point me to something I can debug…

Any help appreciated; I’m tearing my hair out.

Hi, I just did this on my install right now and it worked fine using your steps. First I pushed directly under my username, details below:

ian@elise:/git$ git clone https://git.openwrt.org/openwrt/openwrt.git
Cloning into 'openwrt'...
remote: Enumerating objects: 560304, done.
remote: Counting objects: 100% (560304/560304), done.
remote: Compressing objects: 100% (151363/151363), done.
remote: Total 560304 (delta 390821), reused 553878 (delta 385710)
Receiving objects: 100% (560304/560304), 166.04 MiB | 2.52 MiB/s, done.
Resolving deltas: 100% (390821/390821), done.

ian@elise:/git/openwrt$ git remote rename origin old-origin

ian@elise:/git/openwrt$ git remote add origin https://iwalker:mytoken@gitlab.mydomain.com/iwalker/openwrt

ian@elise:/git/openwrt$ git push -u origin --all
warning: redirecting to https://gitlab.mydomain.com/iwalker/openwrt.git/
Enumerating objects: 533195, done.
Counting objects: 100% (533195/533195), done.
Delta compression using up to 12 threads
Compressing objects: 100% (142258/142258), done.
Writing objects: 100% (533195/533195), 162.17 MiB | 3.92 MiB/s, done.
Total 533195 (delta 369520), reused 530483 (delta 368173)
remote: Resolving deltas: 100% (369520/369520), done.
remote: Checking connectivity: 533195, done.
remote: 
remote: 
remote: The private project iwalker/openwrt was successfully created.
remote: 
remote: To configure the remote, run:
remote:   git remote add origin https://gitlab.mydomain.com/iwalker/openwrt.git
remote: 
remote: To view the project, visit:
remote:   https://gitlab.mydomain.com/iwalker/openwrt
remote: 
remote: 
remote: 
To https://gitlab.mydomain.com/iwalker/openwrt
 * [new branch]            master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

I then decided to create a group on my server called openwrt, and then removed the origin remote, and created again:

ian@elise:/git/openwrt$ git remote add origin https://iwalker:mytoken@gitlab.mydomain.com/openwrt/openwrt

ian@elise:/git/openwrt$ git push -u origin --all
warning: redirecting to https://gitlab.mydomain.com/openwrt/openwrt.git/
Enumerating objects: 533195, done.
Counting objects: 100% (533195/533195), done.
Delta compression using up to 12 threads
Compressing objects: 100% (142258/142258), done.
Writing objects: 100% (533195/533195), 162.19 MiB | 6.47 MiB/s, done.
Total 533195 (delta 369518), reused 530483 (delta 368173)
remote: Resolving deltas: 100% (369518/369518), done.
remote: Checking connectivity: 533195, done.
remote: 
remote: 
remote: The private project openwrt/openwrt was successfully created.
remote: 
remote: To configure the remote, run:
remote:   git remote add origin https://gitlab.mydomain.com/openwrt/openwrt.git
remote: 
remote: To view the project, visit:
remote:   https://gitlab.mydomain.com/openwrt/openwrt
remote: 
remote: 
remote: 
To https://gitlab.mydomain.com/openwrt/openwrt
 * [new branch]            master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

maybe yours isn’t working because the group doesn’t exist on your server prior to making the push? Mine failed if I didn’t create the group first, but then I got a different error:

ian@elise:/git/openwrt$ git push -u origin --all
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab.mydomain.com/openwrt/openwrt/' not found

done on a VM with 4cpu and 8gb ram.

The only main difference is I put the username/token in the remote add origin URL, as your URL didn’t include this. The access token was created with API access rights only which effectively gives full access.

Hi @andreasmartens
there are several issues for the LFS cancelled issue. See this and this or just search for it.
There is a feature flag you can try to disable.

Feature.disable(:lfs_check)

Thanks, I didn’t know what to search for before.

To make it explicit to others who are new like me. The feature flag lfs_check needs to be disabled using the rails application (as root):

# gitlab-rails console
--------------------------------------------------------------------------------
 Ruby:         ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
 GitLab:       13.10.0 (5eafdaf7b07) FOSS
 GitLab Shell: 13.17.0
 PostgreSQL:   12.5
--------------------------------------------------------------------------------
Feature.enable(:lfs_check)
Loading production environment (Rails 6.0.3.4)
irb(main):002:0> Feature.disable(:lfs_check)
=> true

According to Developing with feature flags | GitLab this can also be done on a per-project basis.

I tried first disabling it using the feature flags functionality in the WebUI, but it didn’t seem to have any effect.

thanks again, my push has now succeeded.

That’s great to hear. Don’t forget to push LFS as well so you don’t have just links and no files.
You can also try to enable it again since it shouldn’t break anything when you don’t do git push --all.

1 Like