Docker push to self-hosted gitlab registry results in a 403 invalid character

Pushing a large container image layer to our self-hosted gitlab instance, fails with the following error message:

error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"
  • What are you seeing, and how does that differ from what you expect to see?
    I expected the docker push to complete successfully. Smaller images (for instance alpine:latest) can be pushed without any problem.

  • Consider including screenshots, error messages, and/or other helpful visuals

  • What version are you on? Are you using self-managed or GitLab.com?

    • GitLab (Hint: /help):
      13.1.3-ee self-managed

    • Runner (Hint: /admin/runners):
      n/a

you can reproduce this from the command line. The docker version is 19.03.12.

  • Add the CI configuration from .gitlab-ci.yml and other configuration if relevant (e.g. docker-compose.yml)
    N/A

  • What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

https://docs.gitlab.com/charts/installation/deployment.html
We are using the helm charts to deploy gitlab. We checked the nginx settings and both
BodySize (0) and proxyBuffering (off) are set to the expected values.

Other sources, suggest that chunked_transfer_encoding should be set to on, but that is not an option in the helm charts.

We also tried the push with max-concurrent-uploads in the docker daemon set to 1. The error is the same.

Hi,

the invalid character error sources from parsing the HTML error into a JSON response. That should not happen, and needs a fix on its own.

This sounds like a problem/bug with large files and needs a reproducible case as well as thoughts from our engineers. Please go ahead and create a new issue and link this forum topic then :slight_smile:

Cheers,
Michael

@dnsmichi I have created the issue https://gitlab.com/gitlab-org/gitlab/-/issues/229778

1 Like

and moved to https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2273