Kaniko can not push to GitLab registry as it omits port

GitLab CI/CD
1

Kaniko fails to push to GitLab Container Registry as it omits the port though it is included in tag

We are running gitlab/gitlab-ee:13.1.2-ee with GitLab Container Registry enabled.

We’ve set up a pipeline which uses Kaniko for building images.

The image is built successfully but not pushed to the registry as the port is missing as it seems.

As GitLab is running behind an NGINX reverse proxy we see status code 405.

We are really puzzled about this issue, as the Docker documentation clearly says:

To do this, tag the image with the host name or IP address, and the port of the registry

See:

This is what the job log shows:

$ export
...
export CI_PROJECT_DIR='/builds/MYGROUP/MYPROJECT'
...
export CI_REGISTRY_IMAGE='service.MYDOMAIN.de:5050/MYGROUP/MYPROJECT'
...
$ /kaniko/executor --context $CI_PROJECT_DIR/MYDIRECTORY --dockerfile $CI_PROJECT_DIR/MYDIRECTORY/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
INFO[0000] Resolved base name mcr.microsoft.com/dotnet/core/aspnet:3.1 to runtime
...
INFO[0010] ENTRYPOINT ["dotnet", "MYSERVICE.dll"]
error pushing image: failed to push to destination service.MYDOMAIN.de:5050/MYGROUP/MYPROJECT:: PATCH https://service.MYDOMAIN.de/v2/MYGROUP/MYPROJECT/blobs/uploads/ffffffff-ffff-ffff-ffff-ffffffffffff?_state=REDACTED: unsupported status code 405
405; body: <html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>
ERROR: Job failed: exit code 1

As one can see the port is even shown in the error message, but missing in the PATCH request itself.

Does anybody know how to solve this?

Thanks in advance! :slight_smile:

2

Hi,

is your registry configured to use TLS or plain HTTP? If the latter is the case, can you try to add --insecure --skip-tls-verify?

Greets Nico

3

Thanks for the response! I finally found the time to continue working on a solution and, for the moment, decided to workaround the issue by setting up a dedicated sub domain for the registry which can work on the default SSL port.

While I was implementing that I noticed though that the Kaniko runner does not use the configured registry_external_url but the host defined in registry_nginx['proxy_set_headers'].

Without having it tested, my assumption is that setting X-Forwarded-Port there, may actually solve the issue.