Keep the package registry in my project only visible to that project, not parent groups

Hi, I created a private project to host my npm package registry in Gitlab. Everything is working fine there and I can see my test packages that I published, but I’m also able to see them in the parent groups all the way to the root group of my company.
I want to keep this registry private only to the group it’s in. I’m using a group access token that was created in the group where I want the registry to be visible. The problem is when I go up to the parents group and click on Deploy > Package Registry I see my packages in there as well which isn’t good because it could be accessed by other users that aren’t part of this project. Does anyone know why this is happening and how I can keep the registry private to my group? Thanks!