HTTP 403 When K8s Cluster Pulls Image From In-Cluster GitLab Registry

General
, , ,
1

I’m currently running GitLab Omibus 16.3.7-ce.0 Docker image in an Amazon EKS managed Kubernetes cluster and am encountering unexpected HTTP 403 responses when the cluster attempts to pull images from the private, in-cluster GitLab registry.

This instance was migrated from Docker Swarm to Kubernetes within the past few months, and I expect those changes are related to the problem I’m having today: the Kubernetes cluster was able to pull images from the private GitLab registry before GitLab was moved into the cluster.

I have pretty high confidence that this is not a legitimate auth failure: login and image pull work correctly on a local machine when testing the GitLab token used by the cluster. Following these steps, Pull an Image from a Private Registry | Kubernetes, I’ve created my image-pull-secret from the the one that works correctly locally. I’ve also confirmed that the Secret is associated with the Pod as expected.

Pods Log These Events / Errors:

Failed to pull image "registry.k8s.dataforma.com/docker/diff-server:master": failed to pull and unpack image "registry.k8s.dataforma.com/docker/diff-server:master": failed to resolve reference "registry.k8s.dataforma.com/docker/diff-server:master": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

And I believe the following to be the relevant log lines:

2024-08-19 10:32:17.988	==> /var/log/gitlab/registry/current <==
2024-08-19 10:32:17.988	2024-08-19_14:32:17.98807 time="2024-08-19T14:32:17.987Z" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=true correlation_id=07a9c2235d45e78570b289265af2e4f9 go_version=go1.20.5 method=HEAD path=/v2/docker/build/manifests/latest root_repo=docker router=gorilla/mux vars_name=docker/build vars_reference=latest version=v3.79.0-gitlab
2024-08-19 10:32:17.988	2024-08-19_14:32:17.98828 {"content_type":"application/json","correlation_id":"07a9c2235d45e78570b289265af2e4f9","duration_ms":0,"host":"registry.k8s.dataforma.com","level":"info","method":"HEAD","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:44974","remote_ip":"127.0.0.1","status":401,"system":"http","time":"2024-08-19T14:32:17.988Z","ttfb_ms":0,"uri":"/v2/docker/build/manifests/latest","user_agent":"containerd/1.6.19","written_bytes":172}
2024-08-19 10:32:17.988	
2024-08-19 10:32:17.988	==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
2024-08-19 10:32:17.988	10.4.60.76 - - [19/Aug/2024:14:32:17 +0000] "HEAD /v2/docker/build/manifests/latest HTTP/1.1" 401 0 "" "containerd/1.6.19" -
2024-08-19 10:32:18.065	
2024-08-19 10:32:18.065	==> /var/log/gitlab/gitlab-workhorse/current <==
2024-08-19 10:32:18.065	{"content_type":"text/html; charset=utf-8","correlation_id":"01J5NGV5AZ91C068BMJT48NJ2Z","duration_ms":48,"host":"razor.dataforma.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"http://razor.dataforma.com/jwt/auth","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":404,"system":"http","time":"2024-08-19T14:32:18Z","ttfb_ms":48,"uri":"/jwt/auth","user_agent":"containerd/1.6.19","written_bytes":3207}
2024-08-19 10:32:18.067	
2024-08-19 10:32:18.067	==> /var/log/gitlab/nginx/gitlab_access.log <==
2024-08-19 10:32:18.067	10.4.80.135 - - [19/Aug/2024:14:32:18 +0000] "POST /jwt/auth HTTP/1.1" 404 1625 "http://razor.dataforma.com/jwt/auth" "containerd/1.6.19" 1.99
2024-08-19 10:32:18.124	
2024-08-19 10:32:18.124	==> /var/log/gitlab/gitlab-rails/auth.log <==
2024-08-19 10:32:18.124	{"severity":"WARN","time":"2024-08-19T14:32:18.123Z","correlation_id":"01J5NGV5CS55FQYCPBVF1HGJW0","meta.caller_id":"JwtController#auth","meta.remote_ip":"10.4.80.135","meta.feature_category":"system_access","meta.client_id":"ip/10.4.80.135","message":"Denied container registry permissions","scope_type":"repository","requested_project_path":"docker/build","requested_actions":["pull"],"authorized_actions":[]}
2024-08-19 10:32:18.127	
2024-08-19 10:32:18.127	==> /var/log/gitlab/gitlab-rails/production_json.log <==
2024-08-19 10:32:18.127	{"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":403,"time":"2024-08-19T14:32:18.126Z","params":[{"key":"scope","value":"repository:docker/build:pull"},{"key":"service","value":"container_registry"}],"correlation_id":"01J5NGV5CS55FQYCPBVF1HGJW0","meta.caller_id":"JwtController#auth","meta.remote_ip":"10.4.80.135","meta.feature_category":"system_access","meta.client_id":"ip/10.4.80.135","remote_ip":"10.4.80.135","ua":"containerd/1.6.19","request_urgency":"low","target_duration_s":5,"redis_calls":11,"redis_duration_s":0.003529,"redis_read_bytes":2178,"redis_write_bytes":616,"redis_feature_flag_calls":11,"redis_feature_flag_duration_s":0.003529,"redis_feature_flag_read_bytes":2178,"redis_feature_flag_write_bytes":616,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"db_main_duration_s":0.003,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.038649,"mem_objects":12960,"mem_bytes":2519312,"mem_mallocs":4535,"mem_total_bytes":3037712,"pid":694564,"worker_id":"puma_2","rate_limiting_gates":[],"db_duration_s":0.00273,"view_duration_s":0.00075,"duration_s":0.03647}
2024-08-19 10:32:18.131	
2024-08-19 10:32:18.131	==> /var/log/gitlab/gitlab-workhorse/current <==
2024-08-19 10:32:18.131	{"content_type":"application/json; charset=utf-8","correlation_id":"01J5NGV5CS55FQYCPBVF1HGJW0","duration_ms":56,"host":"razor.dataforma.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"http://razor.dataforma.com/jwt/auth?scope=repository%3Adocker%2Fbuild%3Apull\u0026service=container_registry","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":403,"system":"http","time":"2024-08-19T14:32:18Z","ttfb_ms":56,"uri":"/jwt/auth?scope=repository%3Adocker%2Fbuild%3Apull\u0026service=container_registry","user_agent":"containerd/1.6.19","written_bytes":77}
2024-08-19 10:32:18.131	
2024-08-19 10:32:18.131	==> /var/log/gitlab/nginx/gitlab_access.log <==
2024-08-19 10:32:18.131	10.4.80.135 - - [19/Aug/2024:14:32:18 +0000] "GET /jwt/auth?scope=repository%3Adocker%2Fbuild%3Apull&service=container_registry HTTP/1.1" 403 77 "http://razor.dataforma.com/jwt/auth" "containerd/1.6.19" -

Following is what I believe to be the relevant entries in my GitLab Omnibus config:

external_url 'http://razor.dataforma.com'
registry_external_url 'http://registry.k8s.dataforma.com:4567'
registry['env']['REGISTRY_HTTP_RELATIVEURLS'] = true

I expect that my problem is related to the configuration above or what is missing from the configuration above. More specifically, I’m thinking my problem may lie with:

  1. My External URLs specifying HTTP and relying on Ingress and cert-manager to secure
  2. The explicit port in the Registry’s external URL
  3. The relative URL option. Was applied to resolve GitLab container registry administration | GitLab

I know this is a lot of content, but I have the feeling that this has a simple solution if I actually understood the problem. I’m hoping someone with a deeper understanding of the configuration options can offer me some guidance.

I don’t think this is very relevant, but since I mentioned it earlier, following is the Ingress spec:

spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - razor.dataforma.com
        - registry.k8s.dataforma.com
      secretName: gitlab
  rules:
    - host: razor.dataforma.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: gitlab
                port:
                  name: http
    - host: registry.k8s.dataforma.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: gitlab
                port:
                  name: docker-registry

This issue Containerd can't can not pull image from gitlab-registry involving containerd may be related, but there isn’t much detail.

This GitLab issue registry: impossible to login - /jwt/auth/ returns http 403 error (#14564) · Issues · GitLab.org / GitLab · GitLab referring specifically to auth issues on the jwt/auth endpoint also looks related. It’s old, stale, and I think unanswered though.

2

After troubleshooting in a test environment, I was able to resolve my problems by adjusting my GitLab configuration. This article on configuring GitLab for reverse proxies was specifically relevant. I made the following configuration changes:

  • Changed external_url 'http://razor.dataforma.com' to external_url 'https://razor.dataforma.com'
    • Change to https
  • Changed registry_external_url 'http://registry.k8s.dataforma.com:4567' to registry_external_url 'https://registry.k8s.dataforma.com'
    • Change to https
    • Dropped port

and added the following additional configuration to accomplish what I had been trying to express through the URL config:

nginx['listen_https'] = false
nginx['listen_port'] = 80
registry_nginx['listen_https'] = false
registry_nginx['listen_port'] = 4567
2 Likes