Gitlab custom executor makes podman look into the wrong network namespace directory

Using Podman 3.4 on RHEL 8

I have a shell script (let’s call it prepare.sh) with just the following one line in it, and it works fine when I run it from the shell:

podman run -d docker.io/library/alpine:latest

But, when the same shell script is called by a Gitlab runner, in order to be used as a custom executor, and then when the Gitlab runner does execute that prepare.sh script, the following happens:

ERRO[0000] error joining network namespace for container f62f7b061205dce066156cf4ea90f3a90ab417b45cf8b3a5f745ffd6fca13fe7: error retrieving network namespace at /tmp/custom-executor886608394/podman-run-1001/netns/cni-e606bcb7-7156-feeb-0866-15415ebab091: failed to Statfs "/tmp/custom-executor886608394/podman-run-1001/netns/cni-e606bcb7-7156-feeb-0866-15415ebab091": no such file or directory

The problem is, for some reason when the same script is called by the Gitlab runner, podman is looking for a network namespace inside a different folder.

So, in the above case, it’s looking for a network namespace inside /tmp/custom-executor886608394/podman-run-1001/netns instead of just/tmp/podman-run-1001/netns which is the default location for that user.

This gitlab runner is being run as a non-root user. Question is why is the Gitlab runner, as part of the execution of prepare.sh script, forcing podman to create a new user namespace when trying to bring up a new container?

Also, both the Gitlab runner and manual execution are using the same user.

I know it’s a tricky problem, coz this has one leg in Gitlab and another in podman.

Any ideas on how to resolve this issue?

Of course, I informed the podman team about this issue as well here