Gitlab CI images not finding entrypoint file in path

Whenever i try to to run a Gitlab CI job using an docker image i made or from other sources they all fail with the error message:
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown (exec.go:78:0s)
where “sh” can be any entrypoint i define, both in the Gitlab CI image:entrypoint or in the CMD/ENTRYPOINT in the Dockerfile. So far the only container that still seems to work for me is the Docker in Docker and the ubuntu container.

I stripped everything to bare bone and still i have the problem happen, so reproduction should be a breeze.

gitlab-ci.yml:

build:
  image: "$CI_REGISTRY_IMAGE:main"
  script:
    - echo "Hello World"

Dockerfile:

#syntax=docker/dockerfile:1
FROM ubuntu:22.04


CMD [ "/bin/bash" ]

The Response:

Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on blue-1.shared.runners-manager.gitlab.com/default j1aLDqxS, system ID: s_b437a71a38f9
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Preparing the "docker+machine" executor
00:05
Using Docker executor with image registry.gitlab.com/tverloop/cmpp:main ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/tverloop/cmpp:main ...
Using docker image sha256:b9ee81beea1c7287380cb77f83dbb742789d18afb05a628c8c900b48ca685724 for registry.gitlab.com/tverloop/cmpp:main with digest registry.gitlab.com/tverloop/cmpp@sha256:5ddf1f2f9832d6c651a6b750bee1f66af2ad9a3040427c111e6b9af70491ad4b ...
Preparing environment
00:01
Running on runner-j1aldqxs-project-46858809-concurrent-0 via runner-j1aldqxs-shared-1687430868-a2d0a55d...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/TVerloop/cmpp/.git/
Created fresh repository.
Checking out 3ec9adbe as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:00
Using docker image sha256:b9ee81beea1c7287380cb77f83dbb742789d18afb05a628c8c900b48ca685724 for registry.gitlab.com/tverloop/cmpp:main with digest registry.gitlab.com/tverloop/cmpp@sha256:5ddf1f2f9832d6c651a6b750bee1f66af2ad9a3040427c111e6b9af70491ad4b ...
Cleaning up project directory and file based variables
00:01
ERROR: Job failed (system failure): Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown (exec.go:78:0s)

But when i change the image i made for the ubuntu:22.04 version, it suddenly works.

Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on blue-5.shared.runners-manager.gitlab.com/default -AzERasQ, system ID: s_8a38c517a741
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Preparing the "docker+machine" executor
00:05
Using Docker executor with image ubuntu:22.04 ...
Pulling docker image ubuntu:22.04 ...
Using docker image sha256:99284ca6cea039c7784d1414608c6e846dd56830c2a13e1341be681c3ffcc8ac for ubuntu:22.04 with digest ubuntu@sha256:6120be6a2b7ce665d0cbddc3ce6eae60fe94637c6a66985312d1f02f63cc0bcd ...
Preparing environment
00:01
Running on runner--azerasq-project-46858809-concurrent-0 via runner-azerasq-shared-1687431216-5771a9ae...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/TVerloop/cmpp/.git/
Created fresh repository.
Checking out 87eb826e as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:01
Using docker image sha256:99284ca6cea039c7784d1414608c6e846dd56830c2a13e1341be681c3ffcc8ac for ubuntu:22.04 with digest ubuntu@sha256:6120be6a2b7ce665d0cbddc3ce6eae60fe94637c6a66985312d1f02f63cc0bcd ...
$ echo "Hello World"
Hello World
Cleaning up project directory and file based variables
00:00
Job succeeded

Anyone have any ideas? I already made an issue but so far no response

I am sorry, but I cannot reproduce this issue. Used exactly same Dockerfile and pushed to repository.

Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on blue-2.shared.runners-manager.gitlab.com/default XxUrkriX, system ID: s_90897a2659b5
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Resolving secrets
00:00
Preparing the "docker+machine" executor
00:06
Using Docker executor with image public.ecr.aws/_redacted_/ubuntu:bad ...
Pulling docker image public.ecr.aws_redacted_/ubuntu:bad ...
Using docker image sha256:c33f313a366178bb88ea7a1c6eb2db91e25ab74f1a3e99ae74b1a0b97e35d5f2 for public.ecr.aws/_redacted_/ubuntu:bad with digest public.ecr.aws/_redacted_/ubuntu@sha256:d6e5a246522a6bf1e878df96e6a09d1f83e982cc3f640ddd5f295a87f2678607 ...
Preparing environment
00:01
Running on runner-xxurkrix-project-38655043-concurrent-0 via runner-xxurkrix-shared-1687961296-cfaf11d5...
Getting source from Git repository
00:02
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/_redacted_/.git/
Created fresh repository.
Checking out 5071b6b2 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:00
Using docker image sha256:c33f313a366178bb88ea7a1c6eb2db91e25ab74f1a3e99ae74b1a0b97e35d5f2 for public.ecr.aws/_redacted_/ubuntu:bad with digest public.ecr.aws/_redacted_/ubuntu@sha256:d6e5a246522a6bf1e878df96e6a09d1f83e982cc3f640ddd5f295a87f2678607 ...
$ echo hello
hello
Cleaning up project directory and file based variables
00:01
Job succeeded

Make sure you have correct image in your registry.

Im not sure what changed. But it seems to work now.

It seems that maybe

gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)

is the problem.
Some exact same code was running well on gitlab-runner 15.6.0~beta.186.ga889181a (a889181a) but not on gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823).