Restrict OAuth OpenID access to members of specific GitLab group

When setting up GitLab as OAuth/OpenID identity provider, it seems like every user on that GitLab server can login to the other system (Polarion).

Is it possible to require the users to be member of certain group/subgroup in order to be able to login.
I’ve setup a group owned application since I thought this would already restrict access to group members, but it does not.
ref: GitLab as OAuth2 authentication service provider | GitLab

We are currently running GitLab Enterprise Edition 13.12.3-ee in our own cloud.