Self Hosted Gitlab access to IIS Application Pool Users

Hi, I have a application hosted on IIS and that application is using application pools with Network Service User. From that application when I do test connection to my selfhosted Gitlab Repositories, its successful without any authentication and even the repos are private. When I change application pools user to any other administrator user and do the test connection, it starts failing.
How can I troubleshoot this and stop access to my private repo without auth.
This is from one machine only, and GitLab is hosted on any other server. When I try Git clone from this machine, it asks for credentials and even in browser it asks for LDAP credentials. Can someone help me in finding the reason.