Error "blob upload unknown" pushing to gitlab 8.10 docker registry

How to Use GitLab
1

After upgrading gitlab from 8.9.6 to 8.10 I cannot push images to my gitlab project’s docker registry.

After several attempts to push each layer with retries the entire “docker push” operation fails with “blob upload unknown”.

I’m using docker for mac 1.12-beta20 and was able to push images to my gitlab project’s docker registry before the upgrade…

2

After struggling with this I came here to post the same issue. This is my first time setting up GitLab so I’ve never seen it work before and can’t speak to prior releases. I did replace my GitLab’s instance with a new certificate that I created from a CA I set up in my lab. I had some issues with that, but I think the cert is okay now. The cert looks good when I access the site, I’m able to docker login with no issue and GitLab CI is working (after I added the entire cert chain to GitLab). I set up another container registry as described here: https://docs.docker.com/registry/ and it works.

I used the instructions at http://docs.gitlab.com/ce/administration/container_registry.html. My gitlab.rb file is pretty basic. The only things I modified are:

external_url 'https://gitlab.vmware.local'
registry_external_url 'https://gitlab.vmware.local:4567'
nginx['redirect_http_to_https'] = true

I created a small scratch container as described here: https://docs.docker.com/engine/userguide/eng-image/baseimages/#/creating-a-simple-base-image-using-scratch. It also just says retrying and fails with the error blog upload unknown

I looked in /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/chrisgreene/phoenix-ci-test/_uploads and I can see several folders with UUIDs for names, but not sure how to troubleshoot this.

I ran gitlab-ctl tail, pushed an image and saw:

192.168.3.219 - - [23/Jul/2016:13:47:47 -0600] "PATCH /v2/chrisgreene/phoenix-ci-test/blobs/uploads/726a2db5-dd58-4585-8915-ecfbb385ad6b?_state=E24FPdoio6htOH16MpwXHTYQf9CcxTbNDlSA2cUauGh7Ik5hbWUiOiJjaHJpc2dyZWVuZS9waG9lbml4LWNpLXRlc3QiLCJVVUlEIjoiNzI2YTJkYjUtZGQ1OC00NTg1LTg5MTUtZWNmYmIzODVhZDZiIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE2LTA3LTIzVDE5OjQ3OjQxLjk1MDAxMzI5NVoifQ%3D%3D HTTP/1.1" 404 2275 "-" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \x5C(linux\x5C))"

==> /var/log/gitlab/registry/current <==
2016-07-23_19:47:47.16113 time="2016-07-23T13:47:47.161072611-06:00" level=info msg="response completed" environment=production go.version=go1.5.4 http.request.host=gitlab.vmware.local http.request.id=4124ed60-6e7a-43a0-87e7-5728e146bba7 http.request.method=POST http.request.remoteaddr=192.168.3.219 http.request.uri="/v2/chrisgreene/phoenix-ci-test/blobs/uploads/" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))" http.response.duration=14.566508ms http.response.status=202 http.response.written=0 instance.id=20d6762c-e375-4260-ab18-27deb8741c97 service=registry version=v2.4.1 
2016-07-23_19:47:47.16115 127.0.0.1 - - [23/Jul/2016:13:47:47 -0600] "POST /v2/chrisgreene/phoenix-ci-test/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))"

==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
192.168.3.219 - - [23/Jul/2016:13:47:47 -0600] "POST /v2/chrisgreene/phoenix-ci-test/blobs/uploads/ HTTP/1.1" 202 0 "-" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \x5C(linux\x5C))"

==> /var/log/gitlab/gitlab-rails/production.log <==
Started PATCH "/v2/chrisgreene/phoenix-ci-test/blobs/uploads/f489d8f5-860c-4dd8-8903-8c62d8f0591f?_state=u9EV7s8GZxFDb4l25lrEO5WaV0dIBG3VD_kbCdjvX9B7Ik5hbWUiOiJjaHJpc2dyZWVuZS9waG9lbml4LWNpLXRlc3QiLCJVVUlEIjoiZjQ4OWQ4ZjUtODYwYy00ZGQ4LTg5MDMtOGM2MmQ4ZjA1OTFmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE2LTA3LTIzVDE5OjQ3OjQ3LjE0OTc5NTQ0NloifQ%3D%3D" for 192.168.3.219 at 2016-07-23 13:47:47 -0600
** [Raven] User excluded error: #<ActionController::RoutingError: No route matches [PATCH] "/v2/chrisgreene/phoenix-ci-test/blobs/uploads/f489d8f5-860c-4dd8-8903-8c62d8f0591f">

ActionController::RoutingError (No route matches [PATCH] "/v2/chrisgreene/phoenix-ci-test/blobs/uploads/f489d8f5-860c-4dd8-8903-8c62d8f0591f"):
  actionpack (4.2.7) lib/action_dispatch/middleware/debug_exceptions.rb:21:in `call'
  actionpack (4.2.7) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  railties (4.2.7) lib/rails/rack/logger.rb:38:in `call_app'
  railties (4.2.7) lib/rails/rack/logger.rb:20:in `block in call'
  activesupport (4.2.7) lib/active_support/tagged_logging.rb:68:in `block in tagged'
  activesupport (4.2.7) lib/active_support/tagged_logging.rb:26:in `tagged'
  activesupport (4.2.7) lib/active_support/tagged_logging.rb:68:in `tagged'
  railties (4.2.7) lib/rails/rack/logger.rb:20:in `call'
  request_store (1.3.0) lib/request_store/middleware.rb:9:in `call'
  actionpack (4.2.7) lib/action_dispatch/middleware/request_id.rb:21:in `call'
  rack (1.6.4) lib/rack/methodoverride.rb:22:in `call'
  rack (1.6.4) lib/rack/runtime.rb:18:in `call'
  rack (1.6.4) lib/rack/lock.rb:17:in `call'
  rack (1.6.4) lib/rack/sendfile.rb:113:in `call'
  sentry-raven (1.1.0) lib/raven/integrations/rack.rb:53:in `call'
  railties (4.2.7) lib/rails/engine.rb:518:in `call'
  railties (4.2.7) lib/rails/application.rb:165:in `call'
  railties (4.2.7) lib/rails/railtie.rb:194:in `public_send'
  railties (4.2.7) lib/rails/railtie.rb:194:in `method_missing'
  rack (1.6.4) lib/rack/urlmap.rb:66:in `block in call'
  rack (1.6.4) lib/rack/urlmap.rb:50:in `each'
  rack (1.6.4) lib/rack/urlmap.rb:50:in `call'
  unicorn (4.9.0) lib/unicorn/http_server.rb:580:in `process_client'
  unicorn-worker-killer (0.4.4) lib/unicorn/worker_killer.rb:52:in `process_client'
  unicorn (4.9.0) lib/unicorn/http_server.rb:674:in `worker_loop'
  unicorn (4.9.0) lib/unicorn/http_server.rb:529:in `spawn_missing_workers'
  unicorn (4.9.0) lib/unicorn/http_server.rb:140:in `start'
  unicorn (4.9.0) bin/unicorn:126:in `<top (required)>'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load'
  /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>'

I’m using Docker version 1.11.2, build b9f10c9 and GitLab 8.10.0 (9dec995) on CentOS 7.

Thanks.

3

I ended up changing:

registry_external_url 'https://gitlab.vmware.local:4567'
to
registry_external_url 'https://registry.gitlab.vmware.local'

After making the new DNS entry for this I was able to push.

4

Same issue here. Replacing GitLab Container Registry by standard Docker Registry is not an option, neither is switching from port-based to subdomain-based registry (unless this is a backward incompatible change)

Runners also cannot push images failing with “blob upload unknown”

Edit: Fixed by applying this change: https://gitlab.com/gitlab-org/gitlab-ce/issues/20164#note_13270027

5

Just upgraded to 8.10.1 and I can confirm that this has been fixed :relieved: