Why is my Gitlab shell executor only returning "0 0"?

GitLab CI/CD
1

All the sudden my self-hosted runner set to use the shell executor started returning “0 0” which is causing the pipeline to incorrectly “succeed” at every step without executing the commands in the pipeline.

Running with gitlab-runner 15.9.1 (d540b510)
on runner.my.domain DFy6JND6, system ID: s_e0fd8a656439
Preparing the "shell" executor
00:00
Using Shell (bash) executor...
Preparing environment
00:00
Running on runner.my.domain...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /home/gitlab-runner/builds/DFy6JND6/0/mygitlab/myrepo/.git/
Checking out 99ba9c4f as detached HEAD (ref is develop)...
Removing --version.hwm
Removing --version.pwd
Removing --version.pwi
Removing validate.hwm
Removing validate.pwd
Removing validate.pwi
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
$ packer --version
0 0
$ find . -maxdepth 1 -name '*.pkr.hcl' -print0 | xargs -t0n1 packer validate
packer validate ./rocky8.pkr.hcl
0 0
packer validate ./rocky9.pkr.hcl
0 0
Cleaning up project directory and file based variables
00:00
Job succeeded

If I go in and login as the user gitlab-runner and go to the folder the runner is running and type those same commands (same shell) I get this:

[gitlab-runner@runner]$ packer --version
1.8.6
[gitlab-runner@runner]$ packer validate ./rocky8.pkr.hcl
The configuration is valid.
[gitlab-runner@runner]$ packer validate ./rocky9.pkr.hcl
The configuration is valid.

The pipeline was running fine before I’m not sure what I’ve changed.

In case anyone’s curious here’s the whole pipeline:

before_script:
  - packer --version

stages:
  - validate
  - build
  - test
  - deploy

validate:
  stage: validate
  script:
    - find . -maxdepth 1 -name '*.pkr.hcl' -print0 | xargs -t0n1 packer validate

build:
  stage: deploy
  environment: production
  script:
    - find . -maxdepth 1 -name '*.pkr.hcl' -print0 | xargs -t0n1 packer validate
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: on_success

and here’s the runner config:

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "runner.my.domain"
  url = "https://gitlab.com/"
  id = 21590649
  token = "xxxxxxxxxxxxxxxxxxxxx"
  token_obtained_at = 2023-03-05T02:09:32Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  shell = "bash"
  [runners.cache]
    MaxUploadedArchiveSize = 0

I’m still pretty new to the self-hosted runner and relatively new to Gitlab. Any thoughts?

EDIT: installed the runner on an identical machine and had the same problem. here’s some additional environmental info:

[gitlab-runner@runner ~]$ uname -a
Linux runner.xtal.in 5.14.0-162.18.1.el9_1.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 9 04:28:41 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

[gitlab-runner@runner ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.1 (Plow)

[gitlab-runner@runner ~]$ bash --version
GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

[gitlab-runner@runner ~]$ gitlab-runner --version
Version:      15.9.1
Git revision: d540b510
Git branch:   15-9-stable
GO version:   go1.18.10
Built:        2023-02-20T21:03:05+0000
OS/Arch:      linux/amd64
2

Is it the packer command that shows this specific behavior, or any other command that produces shell output?

My guess would be that packer requires a shell environment configured, which is there in a manual tty login, but not the runner shell executor environment.