Our organization is having an issue with the docker registry. What appears to be intermittently and only in our HQ building we are unable to pull from our registry. I can have it work fine and then five minutes later I will get this error:
Error response from daemon: Head "https://registry.gitlab.com/v2/<repo-path>": error parsing HTTP 403 response body: unexpected end of JSON input: ""
Users outside the building and our EC2 instances have no issues.
We also appear to be having issues with a couple people being able to pull the main branch with similar issues. I’ve logged out and generated a new docker token and I now I can’t login into the registry inside the building, but I can login on a machine outside the building and pull just fine. I.T made sure to whitelist gitlab.com on the firewall. and we’re now trying to use wireshark to see if we can identify an issue. Our working theory is that our building’s IP has somehow been blocked.
unexpected end of JSON input: "" combined with
docker pull in a Google search leads to topics which say that the CA certificate chain may not be not correct. This leads me to another guess: I’ve seen network hardware which inspects TLS encrypted traffic to optimize the packets and decrease traffic, something like Riverbed appliances. They open the TLS packets, and sign it again later, sometimes breaking the packet header and then the size of bytes to read is off, or otherwise broken.
If you happen to have such a firewall in place, ensure to disable TLS traffic optimizations for anything coming from gitlab.com. Some more discussion and examples are in Document examples or best practices for configuring network proxies with ssl interception (self-hosted runners communicating to GitLab SaaS) (#297648) · Issues · GitLab.org / GitLab · GitLab Hope it helps