GitLab Pipeline Push to GitLab Registry ERROR: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value:

GitLab Pipeline Push to GitLab Registry ERROR: error parsing HTTP 404 response body: invalid character ‘<’ looking for beginning of value:

Hi, I deployed GitLab and GitLab Runner on VM inside an Intranet environment behind HTTP Proxy and Custom CA Cert.

Everything works well but the CI pipeline sometimes able to push to GitLab Registry, mostly cannot with error:

#9 writing image sha256:c6e9dd2b86c6b96b5febce2e07ff90eb600e4534422111d22982445c4ef5c4ab done
#9 naming to registry.intranet.domain-name/gitlab-ci-example/test-ssl-registry/master:441e5771392f91334220e23bf657632ea13c1f12 done
#9 naming to registry.intranet.domain-name/gitlab-ci-example/test-ssl-registry/master:latest done
#9 DONE 0.0s
#10 pushing registry.intranet.domain-name/gitlab-ci-example/test-ssl-registry/master:441e5771392f91334220e23bf657632ea13c1f12 with docker
#10 pushing layer 9d830bd22c50
#10 pushing layer 208e467e5f77
#10 pushing layer 36ffdceb4c77
#10 pushing layer 9d830bd22c50 0.2s done
#10 pushing layer 208e467e5f77 0.2s done
#10 pushing layer 36ffdceb4c77 0.2s done
#10 ERROR: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n  <title>The page you're looking for could not be found (404)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: 400;\n
  • What are you seeing, and how does that differ from what you expect to see?

It should be run normally without error

  • Consider including screenshots, error messages, and/or other helpful visuals

  • What version are you on? Are you using self-managed or GitLab.com?

    • GitLab (Hint: /help): 14.9.3-ee
    • Runner (Hint: /admin/runners): 14.10.0
  • Add the CI configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml)

config.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner-1"
  url = "https://git.intranet.domain-name"
  token = "redacted"
  tls-cert-file = "/etc/gitlab-runner/certs/sancertall.crt"
  tls-key-file = "/etc/gitlab-runner/certs/sancertall.key"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

[[runners]]
  name = "runner-2"
  url = "https://git.intranet.domain-name"
  token = "redacted"
  tls-cert-file = "/etc/gitlab-runner/certs/sancertall.crt"
  tls-key-file = "/etc/gitlab-runner/certs/sancertall.key"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0

Files in /etc/gitlab-runner/ directory

root@server-runner:~# ls -la /etc/gitlab-runner/
total 16
drwx------.  3 root root   38 Apr 20 03:53 .
drwxr-xr-x. 91 root root 8192 Apr 20 03:49 ..
drwxr-xr-x.  2 root root  127 Apr 20 22:29 certs
-rw-------.  1 root root 1340 Apr 20 22:07 config.toml
root@server-runner:~# ls -la /etc/gitlab-runner/certs/
total 24
drwxr-xr-x. 2 root root  127 Apr 20 22:29 .
drwx------. 3 root root   38 Apr 20 03:53 ..
-rw-r--r--. 1 root root 3938 Apr 20 04:16 git.intranet.domain-name.crt
-rw-r--r--. 1 root root 3938 Apr 20 22:29 registry.intranet.domain-name.crt
-rw-r--r--. 1 root root 8560 Apr 20 04:28 sancertall.crt
-rw-r--r--. 1 root root 1704 Apr 20 04:28 sancertall.key

.gitlab-ci.yml

# You can override the included template(s) by including variable overrides
# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Note that environment variables can be set in several places
# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
image: php:8.1

stages:
  - build

variables:
  POSTGRES_USER: db_user
  POSTGRES_PASSWORD: db_pass
  POSTGRES_DB: db_name
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""
  HTTP_PROXY: "http://proxy-dc:8080/"
  HTTPS_PROXY: "http://proxy-dc:8080/"
  http_proxy: "http://proxy-dc:8080/"
  https_proxy: "http://proxy-dc:8080/"
  NO_PROXY: "docker"
  no_proxy: "docker"

test_build:
  image: docker:dind
  stage: build
  variables:
    IMAGE_TAG_BACKEND: $CI_REGISTRY_IMAGE/backend:$CI_COMMIT_REF_SLUG
    IMAGE_TAG_PHP: $CI_REGISTRY_IMAGE/php:$CI_COMMIT_REF_SLUG
    DOCKER_TLS_CERTDIR: ''
    HTTP_PROXY: "http://proxy-dc:8080/"
    HTTPS_PROXY: "http://proxy-dc:8080/"
    http_proxy: "http://proxy-dc:8080/"
    https_proxy: "http://proxy-dc:8080/"
    NO_PROXY: "docker"
    no_proxy: "docker"
  services:
    - name: 'docker:20.10.6-dind'
      command: [ '--tls=false', '--host=tcp://0.0.0.0:2375' ]
      alias: docker
  script:
    - export http_proxy="http://proxy-dc:8080/"
    - export https_proxy="http://proxy-dc:8080/"
    - export ftp_proxy="http://proxy-dc:8080/"
    - export HTTP_PROXY="http://proxy-dc:8080/"
    - export HTTPS_PROXY="http://proxy-dc:8080/"
    - export FTP_PROXY="http://proxy-dc:8080/"
    - export no_proxy="127.0.0.1,localhost,10.245.192.52,docker"
    - export NO_PROXY="127.0.0.1,localhost,10.245.192.52,docker"
    - echo $CI_REGISTRY_USER
    - echo $CI_REGISTRY
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

include:
  - template: Jobs/Build.gitlab-ci.yml

GitLab.rb file

external_url 'https://git.intranet.domain-name'
nginx['ssl_certificate'] = "/data/gitlab/cert/sancertall.crt"
nginx['ssl_certificate_key'] = "/data/gitlab/cert/sancertall.key"
# registry_external_url 'https://registry.example.com'
# registry_external_url 'https://git.intranet.domain-name:5050'
registry_external_url 'https://registry.intranet.domain-name'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
# gitlab_rails['registry_host'] = "registry.gitlab.example.com"
# gitlab_rails['registry_port'] = "5050"
# gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
gitlab_rails['registry_path'] = "/data/gitlab-storage/registry"
registry['env'] = {
  'http_proxy' => "http://proxy-dc:8080",
  'https_proxy' => "http://proxy-dc:8080",
  'no_proxy' => ".intranet.domain-name,127.0.0.1,localhost,10.245.192.52,proxy-dc",
  'HTTP_PROXY' => "http://proxy-dc:8080",
  'HTTPS_PROXY' => "http://proxy-dc:8080",
  'NO_PROXY' => ".intranet.domain-name,127.0.0.1,localhost,10.245.192.52,proxy-dc"
}

registry_nginx['ssl_certificate'] = "/data/gitlab/cert/sancertall.crt"
registry_nginx['ssl_certificate_key'] = "/data/gitlab/cert/sancertall.key"
  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

I have already tried to import the SSL cert from Self-signed certificates or custom Certification Authorities | GitLab but still doesn’t work

Update:
From the previous configuration, I changed the gitlab.rb file in the following line (make registry on the same domain with the git):

registry_external_url 'https://git.intranet.domain-name:5050'
# registry_external_url 'https://registry.intranet.domain-name'

and it can push successfully.

My question, what should I change so that the registry can run with the external url at https://registry.intranet.domain-name (separate domain with the git)?

Thanks for taking the time to be thorough in your request, it really helps! :blush: