Dependency proxy returns HTTP 503 for cached images

So there is this major AWS outage today and Docker Hub is also affected. To protect us from issues with external registries, we implemented the dependency proxy. So far, this has worked great.

Today I am getting HTTP 503 errors for pulling both cached and uncached images. At least for the latter, this is expected. But why can’t I download images that (should be) already cached? To make sure I am using a cached image, I accessed the dependency proxy via webui and copy the full image path. I tried looking at logs (e.g. gitlab-workhorse/current), but found no explanation.

We are running CE via Omnibus on Ubuntu 24.04, our current version is 17.11.6. I am happy to provide logs and other information, but any configuration changes or restarts need to be discussed with my team.

This was all yesterday of course, but my post was not released until now.

Also, the issue disappeared as soon as AWS/Docker Hub was working again. This makes it really hard to debug, but I would still like to have input how this can be prevented during future outages.

This behaviour is expected.

Pull through docker proxies still check (at least for any tags that are not sha256 qualified) to see if there are new manifests to pull so an upstream error will bubble down to the client.

To properly isolate yourself from upstream problems you need to run a docker registry that, well, just runs as a registry that is manually updated.