CI Runner can use Dependency Proxy, but Kaniko cannot

kohenkatz · August 31, 2021, 7:46pm

I am now running enough CI jobs that I have started to hit the Docker Hub limits, so I decided it was time to start using the Dependency Proxy.

I have a project which has a build stage and then a packaging stage. The build stage runs the build in Docker, then the package stage uses Kaniko to build containers using the artifacts of the build stage.

Here are the relevant parts of the .gitlab-ci.yml file

stages:
  - build
  - test
  - package

build:
  image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/node:lts-alpine
  stage: build
  script:
    - npm set progress=false
    - npm config set depth 0
    - npm ci --cache .npm --prefer-offline
    - npm run build
  artifacts:
    paths:
      - dist/

.package:
  stage: package
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  needs: ['build']
  before_script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"username\":\"$CI_DEPENDENCY_PROXY_USER\",\"password\":\"$CI_DEPENDENCY_PROXY_PASSWORD\"}}}" > /kaniko/.docker/config.json
  script:
    - /kaniko/executor --build-arg THEME=$VUE_APP_THEME --cache=true --context $CI_PROJECT_DIR --skip-unused-stages --dockerfile $CI_PROJECT_DIR/Dockerfile $REGISTRY_DESTINATIONS

package development:
  extends: .package
  parallel:
    matrix:
      - VUE_APP_THEME: theme1
        APP_ABBR: t1
        PLATFORM: ['', '-aarch64']
      - VUE_APP_THEME: theme2
        APP_ABBR: t2
        PLATFORM: ['', '-aarch64']
  variables:
    REGISTRY_DESTINATIONS: --destination $CI_REGISTRY_IMAGE/$APP_ABBR-snapshots$PLATFORM:$CI_COMMIT_REF_SLUG --destination $CI_REGISTRY_IMAGE/$APP_ABBR-snapshots$PLATFORM:$CI_COMMIT_SHORT_SHA --destination $CI_REGISTRY_IMAGE/$APP_ABBR-snapshots$PLATFORM:latest
  tags:
    - docker${PLATFORM}
  rules:
    - if: '$CI_COMMIT_TAG && $CI_COMMIT_REF_PROTECTED == "true"'
      when: never
    - when: on_success

package production:
  extends: .package
  parallel:
    matrix:
      - VUE_APP_THEME: theme1
        APP_ABBR: t1
        PLATFORM: ['', '-aarch64']
      - VUE_APP_THEME: theme2
        APP_ABBR: t2
        PLATFORM: ['', '-aarch64']
  variables:
    REGISTRY_DESTINATIONS: --destination $CI_REGISTRY_IMAGE/$APP_ABBR$PLATFORM:$CI_COMMIT_TAG --destination $CI_REGISTRY_IMAGE/$APP_ABBR$PLATFORM:latest
  tags:
    - docker${PLATFORM}
  rules:
    - if: '$CI_COMMIT_TAG && $CI_COMMIT_REF_PROTECTED == "true"'

My Dockerfile starts with:

FROM gitlab.example.com/mygroup/dependency_proxy/containers/alpine

The build image being loaded through the Dependency Proxy works perfectly. The problem I have is with Kaniko in the package tasks:

$ /kaniko/executor --build-arg THEME=$VUE_APP_THEME --cache=true --context $CI_PROJECT_DIR --skip-unused-stages --dockerfile $CI_PROJECT_DIR/Dockerfile $REGISTRY_DESTINATIONS
INFO[0000] Using dockerignore file: /builds/mygroup/myproject/.dockerignore 
INFO[0000] Retrieving image manifest gitlab.example.com/mygroup/dependency_proxy/containers/alpine 
INFO[0000] Retrieving image gitlab.example.com/mygroup/dependency_proxy/containers/alpine from registry gitlab.example.com 
ERRO[0000] Error while retrieving image from cache: gitlab.example.com/mygroup/dependency_proxy/containers/alpine GET https://gitlab.example.com/jwt/auth?scope=repository%3Amygroup%2Fdependency_proxy%2Fcontainers%2Falpine%3Apull&service=dependency_proxy: unexpected status code 403 Forbidden: {"message":"access forbidden","status":"error","http_status":403} 
INFO[0000] Retrieving image manifest gitlab.example.com/mygroup/dependency_proxy/containers/alpine 
INFO[0000] Retrieving image gitlab.example.com/mygroup/dependency_proxy/containers/alpine from registry gitlab.example.com 
error building image: GET https://gitlab.example.com/jwt/auth?scope=repository%3Amygroup%2Fdependency_proxy%2Fcontainers%2Falpine%3Apull&service=dependency_proxy: unexpected status code 403 Forbidden: {"message":"access forbidden","status":"error","http_status":403}

As you can see, the credentials are not working.

However, if I manually set these environment variables and run this script in the Kaniko image on my local machine, it works fine and downloads the image from the dependency proxy:

docker run --rm -it --entrypoint="" -v `pwd`:/builds/mygroup/myproject gcr.io/kaniko-project/executor:debug sh

export CI_PROJECT_DIR=/builds/mygroup/myproject
export CI_REGISTRY=registry.example.com
export CI_REGISTRY_USER=myusername
export CI_REGISTRY_PASSWORD=myaccesstoken
export CI_DEPENDENCY_PROXY_SERVER=gitlab.example.com
export CI_DEPENDENCY_PROXY_USER=myusername
export CI_DEPENDENCY_PROXY_PASSWORD=myaccesstoken
mkdir -p /kaniko/.docker
echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"username\":\"$CI_DEPENDENCY_PROXY_USER\",\"password\":\"$CI_DEPENDENCY_PROXY_PASSWORD\"}}}" > /kaniko/.docker/config.json
/kaniko/executor --cache=true --context /builds/mygroup/myproject --skip-unused-stages --dockerfile /builds/mygroup/myproject/Dockerfile --destination registry.example.com/mygroup/myproject/t1-snapshots:latest

Why can Kaniko not use the CI job’s credentials for the Dependency Proxy? It uses the CI job’s credentials for the Container Registry, and that has been working fine for months.

GitLab version: 14.2.1
Runner version: 14.2.0 and 13.11.0

kohenkatz · September 1, 2021, 1:17am

Turns out the reason for this is mentioned in the docs:

CI_DEPENDENCY_PROXY_SERVER and CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX include the server port. If you explicitly include the Dependency Proxy path, the port must be included, unless you have logged into the Dependency Proxy manually without including the port

Since I don’t want to change my Dockerfile to add the port there, I changed the line that writes config.json to remove the port number using awk:

echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD | base64)\"},\"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\":{\"auth\":\"$(echo -n $CI_DEPENDENCY_PROXY_USER:$CI_DEPENDENCY_PROXY_PASSWORD | base64)\"}}}" > /kaniko/.docker/config.json

discordier · May 2, 2023, 3:01pm

Life safer. I kept struggling for days to find the reason for this. Thank you so much!