CI/CD : Error CONNECT tunnel failed, response 503

GitLab CI/CD
, ,
1

Describe your question in as much detail as possible:

“I’m facing a weird problem with my GitLab Runner. Although the runner is able to pick up the job, it encounters an issue when attempting to clone the repository of the project.”

Troubleshooting Situations:
I can make git clone from my runner container
No error log on gitlab container
I added clone_url in runner config, and still the same error

Environment Description:

Gitlab on podman 4.6.1
Runner on podman 4.6.1

Versions :
GitLab CE Version:** v16.3.6-ee
GitLab Runner Version:** 16.3.0

Error Observed: The GitLab Runner fails with the following error : CONNECT tunnel failed, response 503

Full Error Message:

Running with gitlab-runner 16.3.0 (8ec04662)
  on Runner_Name 
Preparing the "docker" executor
00:02
Using Docker executor with image podman/stable ...
Using helper image:  gitlab/gitlab-runner-helper:x86_64-v16.3.0  (overridden, default would be  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-8ec04662 )
Pulling docker image gitlab/gitlab-runner-helper:x86_64-v16.3.0 ...
Using docker image sha256:44f59320761bccd4e41e5f2f576d862e5d8a00879a49bfa7e27a6b475c8d7813 for gitlab/gitlab-runner-helper:x86_64-v16.3.0 with digest docker.io/gitlab/gitlab-runner-helper@sha256:2289b40140d0153f3e2d8e03daee473b2d5d40fdf3b3b6757f2608ce6e841238 ...
Using helper image:  gitlab/gitlab-runner-helper:x86_64-v16.3.0  (overridden, default would be  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-8ec04662 )
Using docker image sha256:44f59320761bccd4e41e5f2f576d862e5d8a00879a49bfa7e27a6b475c8d7813 for gitlab/gitlab-runner-helper:x86_64-v16.3.0 with digest docker.io/gitlab/gitlab-runner-helper@sha256:2289b40140d0153f3e2d8e03daee473b2d5d40fdf3b3b6757f2608ce6e841238 ...
Pulling docker image podman/stable ...
Using docker image sha256:409e36421941f1e712ea8b51cf3c79873eb22f02fecf59ebad5212727e981c59 for podman/stable with digest docker.io/podman/stable@sha256:45b32794452befbc8cea1d9c2470103bdef7bf4869cf0abb1da8e806c142c2ac ...
Preparing environment
00:01
Using helper image:  gitlab/gitlab-runner-helper:x86_64-v16.3.0  (overridden, default would be  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-8ec04662 )
Using docker image sha256:44f59320761bccd4e41e5f2f576d862e5d8a00879a49bfa7e27a6b475c8d7813 for gitlab/gitlab-runner-helper:x86_64-v16.3.0 with digest docker.io/gitlab/gitlab-runner-helper@sha256:2289b40140d0153f3e2d8e03daee473b2d5d40fdf3b3b6757f2608ce6e841238 ...
Running on runner-7zyh5asx7-project-4-concurrent-0 via 3782d4c3cfa6...
Getting source from Git repository
01:15
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/s2e_tls/maven-basic/.git/
fatal: unable to access 'https://git*****.com/project_group/maven-basic.git/': CONNECT tunnel failed, response 503
2

I am seeing the exact same thing, seemingly on random versions of the pipeline.
Just now, I added some variables and proxy build args to Docker, and it started happening again (trying, and failing miserably, to build a custom Kaniko container):

stages:
- build

variables:
  DOCKER_BUILDKIT: 1

build-kaniko:
  stage: build
  services:
    - name: redacted.com/gitlab/dind-custom
      alias: docker
  variables:
    HTTP_PROXY: redacted
    HTTPS_PROXY: redacted
    NO_PROXY: redacted
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build
      --build-arg http_proxy=$HTTP_PROXY
      --build-arg https_proxy=$HTTPS_PROXY
      --build-arg no_proxy=$NO_PROXY
      -t ${CI_REGISTRY_IMAGE}/kaniko-custom:latest
      -t ${CI_REGISTRY_IMAGE}/kaniko-custom:${CI_COMMIT_SHORT_SHA} .
    - docker push ${CI_REGISTRY_IMAGE}/kaniko-custom:latest
    - docker push ${CI_REGISTRY_IMAGE}/kaniko-custom:${CI_COMMIT_SHORT_SHA}
    - echo "Pushed kaniko-custom:${CI_COMMIT_SHORT_SHA} image"