When trying to fetch a repository using SSH my whole team gets the following error:
(Private repository shopiq/core)
15:38 $ git fetch
remote:
remote: ========================================================================
remote:
remote: ERROR: maximum time in concurrency queue reached
remote:
remote: ========================================================================
remote:
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
The issue started today at around 14.00 CET.
Other repositories in the same group work fine.
I tried doing the same thing from an ec2 server but with identical results.
EDIT:
Deleting and re-adding my ssh key did not change the result
EDIT2:
Fetching via HTTP works fine, only ssh does not work since today 14.00 CET
EDIT3:
After about 6 hours it works again.
We are a small team with only 5 persons and I would be very surprised if we consistently hit those rate limits which in turn would block ssh access to our repository from anywhere for any user for hours on end and therefore even blocking all of our CI pipelines.
(The only thing polling the git repository is our old jenkins with no more than 5 git fetches per minute everything else is done via gitlab pipelines.)
Is the only solution to switch to github.com or is there a way to make gitlab ssh consistently work for developers?
EDIT: I haven’t been able to do a single git fetch since last week thursday on our repository.
EDIT2: My current best guess is that the gitaly node hosting our repository is constantly overloaded which in turn produces these errors. If I find the time deleting and recreating our repository could move it to another node which would very probably fix most if not all of our problems.
But again - this should not be necessary for us to use gitlab.com via ssh.
Limits are IP-based, so if your 5 users, CI/CD and Jenkins are all from the same IP, then depending on how often your CI/CD is running, there is the potential it could exceed it.
I doubt deleting your repository will help. I expect that geo location via IP will choose which server you end up connecting to, so eg: your nearest Gitaly instance. In which case you’ll just end up re-creating it on the same server.
I can only guess that this is the situation, since it works, and stops working that the only explanation is your IP being rate limited or something else. Unless you have the ability to report this directly to Gitlab with opening a support ticket with them, then I guess we’ll just have to wait and see if it improves in the future. Unfortunately, free users cannot open tickets with Gitlab. Perhaps a Gitlab employee will appear here sometime soon and be able to help with this.
You could run your own installation of Gitlab it just depends on whether you want to do that or not. That way, you don’t have rate limits at all.
As said in the first post as soon as the error occurs acessing the repository via SSH from any ip-address with any user account produces the error message. Our jenkins server has its own ip-adress separate from the office and even starting a new ec2 server with a new ip-address does not allow one to clone the repository.
To further test this I just created an ec2 server in us-east-1 (our office and all our servers are normally based in germany) and that server produces the exact same error message after about 5 seconds of git clone.
EDIT: It just started working again for my teammates and me
Unfortunately, only the Gitlab team/employees can help you with this - obviously they have an infrastructure issue or something that is just way too sensitive if it is blocking after 5 seconds of activity.