Gitlab runers take too much to pick a CI job


I have a hosted gitlab installation with 3 shared runners.
the problem is it takes like 10 minutes for any runner to pick the CI job and it is completely random. on the same job, first step picked immediately and second step ~8 minutes, final step 2 minutes. completely random.

P.S: we are using shell executor on all runners

anybody have any idea what is the problem and how we could manage to run our jobs as fast as possible?