ArgoCD CLI commands take a long time to process in GitLab pipelines

Environment

GitLab v16.5.1-ee
Rancher RKE2 v2.7.6
Kubernetes v1.26.10
ArgoCD v2.7.4

We are running production clusters with GitLab runners on the K8s cluster alongside with ArgoCD for deployment of applications to multiple environment/namespaces. When the GitLab pipeline stage begins the application deployment, we are seeing long latency between the ArgoCD CLI calls.

For example, our job includes four agrocd commands to login, add repo, app create, and finally app sync. The job takes about 15 - 20 mins to complete those commands, where we can see the same activity usually complete within a few minutes. Each command is taking almost 5 minutes to process, and sometimes the command just fails intermittently.

We can manually deploy a utility pod and execute the same commands and don’t see the latency issues. We only see the long running latency issues when the commands are getting executed from a running GitLab CI job.

Looking for any suggestions or troubleshooting help.