Distributed s3 cache is using dualstack domain[gitlab-runner on kubernetes]

I have enabled VPC endpoint gateway for S3, and I found that its supports ipv4. I have added prefix list in route tables as well. But still my traffic is going through NAT gateway instead of internally. I doubt that is it because of https://bucket_name.s3.dualstack.ap-southeast-1.amazonaws.com/runner/project dualstack domain. I have provided s3.ap-southeast-1.amazonaws.com in its server address. Is there a way I can disable dualstack? Can anyone point me if I have configured something wrong.