Container registry: unknown blob

GitLab CI/CD
1

I setup GitLab CE on my server using https://about.gitlab.com/installation/#ubuntu.

Currently when trying to push an image to the container registry of a project (both manual and CI) the following error occurs:

docker tag myvck/myvck-frontend registry.vck.tv/myvck/myvck-frontend
docker push registry.vck.tv/myvck/myvck-frontend
The push refers to a repository [registry.vck.tv/myvck/myvck-frontend]
565f5c4aa666: Pushing [==================================================>]  9.193MB
6fd18951c87e: Pushing [=====>                                             ]  16.44MB/157.7MB
9e737f65a324: Pushing [==>                                                ]  13.35MB/290.9MB
49c9cb1a8328: Pushing [==================================================>]  227.3kB
3452fee044b5: Pushing [==================================================>]  6.656kB
c1b8e8020161: Pushing  2.048kB
757844ae961b: Pushing [==================================================>]  4.184MB
78ce99ee35de: Waiting 
5bef08742407: Waiting 
unknown blob

In the logs (gitlab-ctl tail registry):

2017-10-24_17:28:24.59540 127.0.0.1 - - [24/Oct/2017:19:28:24 +0200] "HEAD /v2/myvck/myvck-frontend/blobs/sha256:55aa60eb5f5ecbf8d977d0e6a6e8829d4b023b7d68a1a0893f499bf7766c2d1e HTTP/1.1" 404 157 "" "docker/17.10.0-ce go/go1.8.3 git-commit/f4ffd25 kernel/4.10.0-37-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \\(linux\\))"

2017-10-24_17:28:24.60184 time="2017-10-24T19:28:24.601712223+02:00" level=info msg="response completed" environment=production go.version=go1.8.1 http.request.host=registry.vck.tv http.request.id=17ff3a57-50f1-4057-842b-fd7588b4fdc1 http.request.method=GET http.request.referer="http://registry.vck.tv/v2/myvck/myvck-frontend/blobs/uploads/b3e8a1e4-764a-4670-91c6-234780c4a7ae?_state=cA0WliL3FF2Zp1kSXTbqw3a8HdJM3tY2HgJwnWekeVh7Ik5hbWUiOiJteXZjay9teXZjay1mcm9udGVuZCIsIlVVSUQiOiJiM2U4YTFlNC03NjRhLTQ2NzAtOTFjNi0yMzQ3ODBjNGE3YWUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMTAtMjRUMTc6Mjg6MjRaIn0%3D&digest=sha256%3Af768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84" http.request.remoteaddr=84.245.13.12 http.request.uri="/v2/myvck/myvck-frontend/blobs/uploads/b3e8a1e4-764a-4670-91c6-234780c4a7ae?_state=cA0WliL3FF2Zp1kSXTbqw3a8HdJM3tY2HgJwnWekeVh7Ik5hbWUiOiJteXZjay9teXZjay1mcm9udGVuZCIsIlVVSUQiOiJiM2U4YTFlNC03NjRhLTQ2NzAtOTFjNi0yMzQ3ODBjNGE3YWUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMTAtMjRUMTc6Mjg6MjRaIn0%3D&digest=sha256%3Af768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84" http.request.useragent="docker/17.10.0-ce go/go1.8.3 git-commit/f4ffd25 kernel/4.10.0-37-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \\(linux\\))" http.response.duration=81.519627ms http.response.status=204 http.response.written=0 instance.id=b8567c27-f316-4793-9a51-60f1be6bce1b service=registry version=v2.6.1-1-gdd544a8

2017-10-24_17:28:24.60190 127.0.0.1 - - [24/Oct/2017:19:28:24 +0200] "GET /v2/myvck/myvck-frontend/blobs/uploads/b3e8a1e4-764a-4670-91c6-234780c4a7ae?_state=cA0WliL3FF2Zp1kSXTbqw3a8HdJM3tY2HgJwnWekeVh7Ik5hbWUiOiJteXZjay9teXZjay1mcm9udGVuZCIsIlVVSUQiOiJiM2U4YTFlNC03NjRhLTQ2NzAtOTFjNi0yMzQ3ODBjNGE3YWUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMTAtMjRUMTc6Mjg6MjRaIn0%3D&digest=sha256%3Af768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84 HTTP/1.1" 204 0 "http://registry.vck.tv/v2/myvck/myvck-frontend/blobs/uploads/b3e8a1e4-764a-4670-91c6-234780c4a7ae?_state=cA0WliL3FF2Zp1kSXTbqw3a8HdJM3tY2HgJwnWekeVh7Ik5hbWUiOiJteXZjay9teXZjay1mcm9udGVuZCIsIlVVSUQiOiJiM2U4YTFlNC03NjRhLTQ2NzAtOTFjNi0yMzQ3ODBjNGE3YWUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMTAtMjRUMTc6Mjg6MjRaIn0%3D&digest=sha256%3Af768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84" "docker/17.10.0-ce go/go1.8.3 git-commit/f4ffd25 kernel/4.10.0-37-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \\(linux\\))"

2017-10-24_17:28:24.64686 time="2017-10-24T19:28:24.646201796+02:00" level=error msg="response completed with error" auth.user.name=daniel environment=production err.code="blob unknown" err.detail=sha256:f768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84 err.message="blob unknown to registry" go.version=go1.8.1 http.request.host=registry.vck.tv http.request.id=ccb1752c-a889-48fb-853c-2e5d9f5a01cf http.request.method=HEAD http.request.remoteaddr=84.245.13.12 http.request.uri="/v2/myvck/myvck-frontend/blobs/sha256:f768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84" http.request.useragent="docker/17.10.0-ce go/go1.8.3 git-commit/f4ffd25 kernel/4.10.0-37-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=3.273325ms http.response.status=404 http.response.written=157 instance.id=b8567c27-f316-4793-9a51-60f1be6bce1b service=registry vars.digest="sha256:f768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84" vars.name="myvck/myvck-frontend" version=v2.6.1-1-gdd544a8

2017-10-24_17:28:24.64692 127.0.0.1 - - [24/Oct/2017:19:28:24 +0200] "HEAD /v2/myvck/myvck-frontend/blobs/sha256:f768b3ba268cffb832e064f8d09aa02f9645064fc8fdd228253b5b468ad28f84 HTTP/1.1" 404 157 "" "docker/17.10.0-ce go/go1.8.3 git-commit/f4ffd25 kernel/4.10.0-37-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.10.0-ce \\(linux\\))"

I tried to reproduce this bug on gitlab.com, but no success. It appears to be a configuration issue with my GitLab Registry. Does somebody have an idea on how to solve this?

1 Like
2

I solved it using this Reddit thread: https://www.reddit.com/r/gitlab/comments/5zd329/container_registry_behind_reverse_proxy/

3

Hi @DanielHuisman, can you please describe what the actual problem was and how exactly you solved it? I am hitting the exact same problem but am running the registry behind HAProxy.

4

In Apache I configured a basic reverse HTTP proxy from https://registry.vck.tv to http://localhost:20081.
This is my final gitlab.rb config:

################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ce/administration/container_registry.html
################################################################################

registry_external_url 'https://registry.vck.tv'

### Settings used by GitLab application
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.vck.tv"
gitlab_rails['registry_port'] = "5005"
gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

###! **Do not change the following 3 settings unless you know what you are
###!   doing**
# gitlab_rails['registry_api_url'] = "http://localhost:5000"
# gitlab_rails['registry_key_path'] = "/var/opt/gitlab/gitlab-rails/certificate.key"
# gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"

### Settings used by Registry application
registry['enable'] = true
registry['username'] = "registry"
registry['group'] = "registry"
registry['uid'] = nil
registry['gid'] = nil
registry['dir'] = "/var/opt/gitlab/registry"
registry['registry_http_addr'] = "0.0.0.0:5000"
registry['debug_addr'] = "localhost:5001"
registry['log_directory'] = "/var/log/gitlab/registry"
registry['log_level'] = "info"
registry['rootcertbundle'] = "/var/opt/gitlab/registry/gitlab-registry.crt"
registry['storage_delete_enabled'] = true

################################################################################
## Registry NGINX
################################################################################

# All the settings defined in the "GitLab NGINX" section are also available in this "Registry NGINX" section
# You just have to change the key "nginx['some_settings']" with "registry_nginx['some_settings']"

# Below you can find settings that are exclusive to "Registry NGINX"
registry_nginx['enable'] = true

registry_nginx['listen_port'] = 20081
registry_nginx['listen_https'] = false

registry_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

At first I didn’t have the proxy_set_headers, so I uncommented those and I fixed a typo in the port number somewhere else. I think this configuration should work with HAProxy as well.
Hope it helps.

1 Like
5

Hi Daniel, thank you very much for your prompt reply. I found a solution and the key seems to be that the protocol is forced to https. From what I can tell you are doing the same thing. Here is a reference on how to solve this for HAProxy https://github.com/docker/distribution/issues/2225

6

We had the same issue actually on GitLab.com this morning.

Irritatingly, our pipeline doesn’t consistently fail (currently only in a Scheduled Pipeline).

Since we don’t have any influence on their infrastructure I’ll bring this up with GitLab.com via a support ticket.

Simple All-GitLab Setup

Our CI setup is both simplistic and generic, all with GitLab variables:

docker-image:
  image: docker:latest
  stage: build
  services:
  - docker:dind
  before_script:
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
  - docker build --pull -t "$CI_REGISTRY_IMAGE" .
  - docker tag "$CI_REGISTRY_IMAGE" "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
  - docker push "$CI_REGISTRY_IMAGE"
  only:
  - master