How to fix "Something went wrong while fetching the repository list"?

How to Use GitLab Self-managed
,
1

One of my projects recently started showing “Something went wrong while fetching the repository list.” errors when I go to the Container Registry page and click the Image Repository.

In the Chrome DevTools, I see that the request looks like this (copied using right-click then Copy as cURL (Bash)):

curl 'https://gitlab.example.com/api/graphql' \
  -H 'authority: gitlab.example.com' \
  -H 'pragma: no-cache' \
  -H 'cache-control: no-cache' \
  -H 'accept: */*' \
  -H 'x-csrf-token: [REDACTED]' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.27 Safari/537.36' \
  -H 'content-type: application/json' \
  -H 'origin: https://gitlab.example.com' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'referer: https://gitlab.example.com/mygroup/myproject/container_registry/19' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cookie: new_nav=true; diff_view=inline; sidebar_collapsed=false; explore_groups_landing_dismissed=true; frequently_used_emojis=thumbsup; experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqY3pZamc1TlRWbUxUTmtNMkV0TkdFMk5pMWlOR013TFdZeU4yWTJaVEUwTnpreVlpST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--b000b308b6e50bdbc2d44f8b917a3193b9810cea; _ga=GA1.2.1478498910.1502137268; _fbp=fb.1.1595255074787.1600510118; visitor_id=6f5d0f24-fb9a-4537-b10a-2616718d640b; event_filter=all; diff_whitespace=0; _gitlab_session=[REDACTED]' \
  --data-raw $'[{"operationName":"getContainerRepositoryDetails","variables":{"id":"gid://gitlab/ContainerRepository/19","first":10},"query":"query getContainerRepositoryDetails($id: ID\u0021, $first: Int, $last: Int, $after: String, $before: String) {\\n  containerRepository(id: $id) {\\n    id\\n    name\\n    path\\n    status\\n    location\\n    canDelete\\n    createdAt\\n    updatedAt\\n    tagsCount\\n    expirationPolicyStartedAt\\n    tags(after: $after, before: $before, first: $first, last: $last) {\\n      nodes {\\n        digest\\n        location\\n        path\\n        name\\n        revision\\n        shortRevision\\n        createdAt\\n        totalSize\\n        canDelete\\n        __typename\\n      }\\n      pageInfo {\\n        ...PageInfo\\n        __typename\\n      }\\n      __typename\\n    }\\n    project {\\n      visibility\\n      __typename\\n    }\\n    __typename\\n  }\\n}\\n\\nfragment PageInfo on PageInfo {\\n  hasNextPage\\n  hasPreviousPage\\n  startCursor\\n  endCursor\\n  __typename\\n}\\n"}]' \
  --compressed

Here is the (formatted) response:

[{
	"data": {
		"containerRepository": {
			"id": "gid://gitlab/ContainerRepository/19",
			"name": "",
			"path": "mygroup/myproject",
			"status": null,
			"location": "registry.example.com/mygroup/myproject",
			"canDelete": true,
			"createdAt": "2020-12-29T13:16:58Z",
			"updatedAt": "2020-12-29T13:16:58Z",
			"tagsCount": 21,
			"expirationPolicyStartedAt": null,
			"tags": {
				"nodes": [null, null, null, null, null, null, null, null, null, null],
				"pageInfo": {
					"__typename": "PageInfo",
					"hasNextPage": true,
					"hasPreviousPage": false,
					"startCursor": "MQ",
					"endCursor": "MTA"
				},
				"__typename": "ContainerRepositoryTagConnection"
			},
			"project": {
				"visibility": "private",
				"__typename": "Project"
			},
			"__typename": "ContainerRepositoryDetails"
		}
	},
	"errors": [{
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}, {
		"message": "Cannot return null for non-nullable field ContainerRepositoryTag.digest"
	}]
}]

When I look at the GitLab logs, I see this:

==> /var/log/gitlab/registry/current <==
2020-12-29_19:35:51.82064 time="2020-12-29T19:35:51Z" level=info msg="authorized request" auth_user_name= correlation_id=krelSBSAaR4 go_version=go1.14.7 vars_name=mygroup/myproject
2020-12-29_19:35:51.84794 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:51 +0000] "GET /v2/mygroup/myproject/tags/list HTTP/1.1" 200 311 "" "GitLab/13.7.1" 29
2020-12-29_19:35:51.85862 time="2020-12-29T19:35:51Z" level=info msg="authorized request" auth_user_name= correlation_id=WlYnqSIqb54 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=375c3fd0
2020-12-29_19:35:51.92448 time="2020-12-29T19:35:51Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=WlYnqSIqb54 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:51.92457 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:51 +0000] "HEAD /v2/mygroup/myproject/manifests/375c3fd0 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 68
2020-12-29_19:35:51.92745 time="2020-12-29T19:35:51Z" level=info msg="authorized request" auth_user_name= correlation_id=g3Uej4Qtaz6 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=43d5bd04
2020-12-29_19:35:51.99596 time="2020-12-29T19:35:51Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=g3Uej4Qtaz6 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:51.99605 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:51 +0000] "HEAD /v2/mygroup/myproject/manifests/43d5bd04 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 70
2020-12-29_19:35:51.99918 time="2020-12-29T19:35:51Z" level=info msg="authorized request" auth_user_name= correlation_id=H5oiIKwosL6 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=9bf761b3
2020-12-29_19:35:52.06105 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=H5oiIKwosL6 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.06108 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/9bf761b3 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 63
2020-12-29_19:35:52.06396 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=4TpoiP1GlH5 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=9c970366
2020-12-29_19:35:52.12313 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=4TpoiP1GlH5 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.12319 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/9c970366 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 60
2020-12-29_19:35:52.12608 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=fXCfLdAvFB9 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=b4acd36c
2020-12-29_19:35:52.19410 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=fXCfLdAvFB9 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.19413 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/b4acd36c HTTP/1.1" 404 193 "" "GitLab/13.7.1" 69
2020-12-29_19:35:52.19867 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=026xnJOGpj8 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=b4aed944
2020-12-29_19:35:52.25148 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=026xnJOGpj8 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.25151 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/b4aed944 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 55
2020-12-29_19:35:52.25439 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=zO5aSKTw9z7 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=b583cb97
2020-12-29_19:35:52.33065 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=zO5aSKTw9z7 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.33068 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/b583cb97 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 77
2020-12-29_19:35:52.33439 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=EmzNkXrQ7T4 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=c426f28b
2020-12-29_19:35:52.38693 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=EmzNkXrQ7T4 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.38696 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/c426f28b HTTP/1.1" 404 193 "" "GitLab/13.7.1" 54
2020-12-29_19:35:52.39068 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=0asCRyYACo9 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=c8f76684
2020-12-29_19:35:52.45004 time="2020-12-29T19:35:52Z" level=error msg="manifest unknown" auth_user_name= code=MANIFEST_UNKNOWN correlation_id=0asCRyYACo9 detail="unknown manifest name=mygroup/myproject revisi
2020-12-29_19:35:52.45007 localhost:5000 127.0.0.1 - - [2020/12/29:19:35:52 +0000] "HEAD /v2/mygroup/myproject/manifests/c8f76684 HTTP/1.1" 404 193 "" "GitLab/13.7.1" 61
2020-12-29_19:35:52.45335 time="2020-12-29T19:35:52Z" level=info msg="authorized request" auth_user_name= correlation_id=Uvpd9KeKXs5 go_version=go1.14.7 vars_name=mygroup/myproject vars_reference=efaeecc1

I know that these tags were created some weeks ago and then manually deleted using the delete button in the list in GitLab, so I’m not sure why they only got partially deleted and they are causing this trouble now. (Partially deleted seems to mean that the scheduled garbage collection removed the actual manifests and layers, but the links to them remained.)

I can fix the issue by manually deleting the folders that I know should no longer be there, but I’d still like to know why that did not happen automatically.

2

I’m unsure if this matches exactly with your scenario but a common (bad) development practice is to retag an image without actually deleting it in between. While docker accepts a new build being tagged with some existing name (forced), it ends up orphaning the previous reference, which may still be in use somewhere else if deployed.

These forms of leftovers are not cleaned up when the regular garbage collection runs, as they could be considered re-required by some running containers using them (when they ran on the previous tag’s pointer). To actually wipe them out (if this is no concern), follow GitLab Container Registry administration | GitLab

3

@hchouraria But how to handle for example the lastet tag? this is replaced all the time and I did not find a docker command to delete images from the registry from within my CI scripts

4

@fritz-net - The registry’s delete functions are available on its API but the docker client program does not implement any calls to it. It also requires privileged identity tokens to function. You can try to modify the script presented here that exercises the API directly: How to delete a Docker image tag from Docker Hub registry

Or alternatively, use GitLab’s Container Registry APIs if it is simply tags you are looking to delete: Container Registry API | GitLab

The latest floating tag is best used as a secondary tag. For example, new or temporary images can be published with their commit reference ID as the tag, and the latest as a second tag. When the latest tag moves to another image build later, the commit reference ID tag is still holding a reference to the image instead of letting it dangle, and this tag name can then be cleaned up normally through the regular cleanup policy schedule: GitLab Container Registry | GitLab

1 Like