SSH key per project solution

Hello, i have a problem that i don’t know how to handle. I have 2 Repos in Gitlab , RepoA and RepoB. In my gitlab user i have 3 ssh keys, one from my laptop and 2 from the 2 VM (VMA and VMB) that work with the 2 repos. In this way i can pull and push easy from the VMs and from my laptop. On RepoA i have a colaborator that also works. He has RepoA shared to his Gitlab account and he has access to VMA to push and pull code. He sould never have access to RepoB, so from here comes my problem. If he can guess the RepoB url , he can pull and see it on the VMA where he has access already just becouse Gitlab does not have a granular way o adding ssh keys per project. I probably am missing something very logical here but i just can’t see what. I don’t even know how to google search this. Could anybody give me some hints?