Gitlab runner Docker image error

Hi,

I using the Gitlab runner in k8s. With previous versions all was fine but wiht the 3 last i have problems.

I have one runner with Docker compose and all works as expected but the same Docker image not is possible run it in k8s.

I have this error… ImagePullBackOff (Back-off pulling image “gitlab/gitlab-runner:alpine-v16.0.0”)

I checked that i can pull this image in my computer with docker pull but checking the manifest i have this error (tested in 2 different pc’s)

docker manifest inspect gitlab/gitlab-runner:alpine-v16.0.0
no such manifest: docker.io/gitlab/gitlab-runner:alpine-v16.0.0

And the same error for the 16.0.1…

Is possible that the image uploaded contains some errors?

I have this response for the 15.10.1

docker manifest inspect gitlab/gitlab-runner:alpine-v15.10.1
{
“schemaVersion”: 2,
“mediaType”: “application/vnd.docker.distribution.manifest.list.v2+json”,
“manifests”: [
{
“mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,
“size”: 896,
“digest”: “sha256:b6c61ec57d0157425cc095a87362f2704df8a9bb93f3c7695232ad39466657b5”,
“platform”: {
“architecture”: “amd64”,
“os”: “linux”
}
},
{
“mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,
“size”: 896,
“digest”: “sha256:3e6adb9c3735cbc89394b609f0d9668b025950ec5dc394c59ac80710ae4dd972”,
“platform”: {
“architecture”: “arm64”,
“os”: “linux”
}
},
{
“mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,
“size”: 896,
“digest”: “sha256:9a472e3d75b3f605fe859983c38dd050b408da3890fbcb9634b3eb75efaa2134”,
“platform”: {
“architecture”: “ppc64le”,
“os”: “linux”
}
},
{
“mediaType”: “application/vnd.docker.distribution.manifest.v2+json”,
“size”: 896,
“digest”: “sha256:b3fa959c625efd7562326f588c1f35cc51912ef76134ac6718946e5b36fc87a1”,
“platform”: {
“architecture”: “s390x”,
“os”: “linux”
}
}
]
}

Have you changed the image for GitLab runner in your k8s? I am asking, because by default the image is downloaded from GitLab registry registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v16.0.0 not from Docker Hub.

I’m using, like in previous versions, the Docker Hub repository but I tested with the image from the Gitlab registry and the error is the same

image912×318 11.7 KB

image1029×59 13.2 KB

I cannot replicate your issue on 10+ k8s and docker gitlab runners which are running v16. All of them can pull the image just fine.

Trying to pull registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v16.0.0...
Getting image source signatures
Copying blob d677f481755c done  
Copying blob 0ce1dd7918a4 done  
Copying blob f0a9735d68e4 done  
Copying config 00eae5bda2 done  
Writing manifest to image destination
Storing signatures
00eae5bda206bf6fa1e124169a89716c87e936314124d745018377b1cbff7868

Do you have some proxy for egress by any chance?

My k8s is corporate and yes, I behind of one proxy but all the versions before of 15.10.1 works correctly. I tested the version 16 in 2 different clusters and i have the same error.

On some forums you may find that this error is related to some wrong field in the manifest and for this reason I tried on my home computer as well and I get the same error for the manifest that I mentioned before. For version 15.10.1 I can see it.

Can you check if you can see the manifest for the version 16.0.0 and 15.10.1?

Works ok on my local workstation with Podman

$ podman manifest inspect registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v16.0.0
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 862,
            "digest": "sha256:62e4ba9f593df48f10fda475e8c92e9fd26ff851fa49540afd9f290bf3c7ca50",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 862,
            "digest": "sha256:da7ad1ea769e0c84a798da9e61e6ba11e9471563a56ff58ca9c8d77a60eddf1d",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 862,
            "digest": "sha256:1d55207347a34c32d384b3dfa49f8b132ebd71dca0f6f202de38c7ced94a1cbb",
            "platform": {
                "architecture": "ppc64le",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 862,
            "digest": "sha256:293dd500c94c0abb7ca29617855a98385b67815fed12152ca57341a07062e6f5",
            "platform": {
                "architecture": "s390x",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 566,
            "digest": "sha256:8b0e8b17c41c86be23ebb0d3ac2ddd8cdf6084ad5a4c426e7f2ae4e7f48eee29",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 566,
            "digest": "sha256:04287ffb77de23d469b142644f5dd067924a28368fa63c4b99c40204ed556f22",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 566,
            "digest": "sha256:6ad6bc6e465da3096f45fa3f45ab4a0a0557e23ce64f489387f9e33d163a7a5b",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 566,
            "digest": "sha256:6a4f76b6fd75ee3576449d15e1897ca2ff98bf8c6ce5b601a8247309ec40ff17",
            "platform": {
                "architecture": "unknown",
                "os": "unknown"
            }
        }
    ]
}

but I see couple of entries with

"platform": {
                "architecture": "unknown",
                "os": "unknown"
            }

which are not in the 15.x manifests.

I suggest to raise an Issue in the issue tracker

Thanks!!. Issue created Docker Image Error :: Manifest unknown fields (#31104) · Issues · GitLab.org / gitlab-runner · GitLab