[Resolved] Gitlab-runner not executing jobs

How to Use GitLab
,
1

We have a gitlab-runner on a Raspberry Pi which is not executing jobs properly. The jobs are succeeding, but before_script, script & after_script are not executing.
Example job: check-current (#4384050801) · Jobs · SKAO / SKA Mid Sky Simulator Controller · GitLab
In the runner logs, I can see that the job is picked up and it does say that it is pulling the repo and executing the “step_script” stage of the job script. However, there is no output, and the job finishes in a few seconds, despite adding multiple sleeps of 2 minutes.
The job configuration can be found here:
https://gitlab.com/ska-telescope/ska-te-mid-skysimctl/-/blob/at-553/.gitlab-ci.yml?ref_type=heads#L100
The runner output is as follows:

May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Checking for jobs...nothing                         runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Processing runner                                   builds=0 runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Acquiring executor from provider                    builds=0 runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Acquiring job slot                                  builds=0 runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Acquiring request slot                              builds=0 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Checking for jobs... received                       job=4384066441 repo_url=https://gitlab.com/ska-telescope/ska-te-mid-skysimctl.git runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Processing chain                                    chain-leaf=[0x48ff8c0 0x48ffb80 0x4dd3340] context=certificate-chain-build resolve-full-chain=true
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Certificate doesn't provide parent URL: exiting the loop  Issuer=Baltimore CyberTrust Root IssuerCertURL=[] Serial=<snip> Subject=Baltimore CyberTrust Roo>
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Verifying last certificate to find the final root certificate  Issuer=Baltimore CyberTrust Root IssuerCertURL=[] Serial=<snip> Subject=Baltimore CyberTrus>
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Failed to requeue the runner                        builds=1 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Running with gitlab-runner 16.0.1 (79704081)        job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   on mid-itf-sky-sim zh_hSFRBy, system ID: s_33e8ecff1544  job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Resolving secrets                       job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Preparing the "shell" executor          job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Shell configuration: command: su
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: arguments:
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - -s
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - /bin/bash
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - gitlab-runner
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - -c
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - bash -l
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: cmdline: su -s /bin/bash gitlab-runner -c $'bash -l'
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: dockercommand:
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - sh
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - -c
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash -l\nelif [ -x /usr/bin/bash
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   ]; then\n\texec /usr/bin/bash -l\nelif [ -x /bin/bash ]; then\n\texec /bin/bash
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   -l\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh -l\nelif [ -x
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   /usr/bin/sh ]; then\n\texec /usr/bin/sh -l\nelif [ -x /bin/sh ]; then\n\texec /bin/sh
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   -l\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh -l\nelse\n\techo shell not
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   found\n\texit 1\nfi\n\n"
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: passfile: false
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: extension: ""
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]:   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Using Shell (bash) executor...                      job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Waiting for signals...                              job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: No referees configured                              job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=prepare_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Preparing environment                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim su[18809]: (to gitlab-runner) root on none
May 31 12:21:57 mid-itf-sky-sim su[18809]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=get_sources job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Getting source from Git repository      job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim su[18830]: (to gitlab-runner) root on none
May 31 12:21:59 mid-itf-sky-sim su[18830]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:21:59 mid-itf-sky-sim su[18830]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=restore_cache job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Restoring cache                         job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim su[18835]: (to gitlab-runner) root on none
May 31 12:21:59 mid-itf-sky-sim su[18835]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Feeding runners to channel                          builds=1
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Feeding runner to channel                           builds=1 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim su[18835]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=download_artifacts job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Skipping stage (nothing to do)                      build_stage=download_artifacts job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=step_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing "step_script" stage of the job script  job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim su[18840]: (to gitlab-runner) root on none
May 31 12:22:00 mid-itf-sky-sim su[18840]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:00 mid-itf-sky-sim su[18840]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=after_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Running after_script                    job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim su[18845]: (to gitlab-runner) root on none
May 31 12:22:00 mid-itf-sky-sim su[18845]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:01 mid-itf-sky-sim su[18845]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=archive_cache job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Saving cache for successful job         job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim su[18851]: (to gitlab-runner) root on none
May 31 12:22:01 mid-itf-sky-sim su[18851]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:01 mid-itf-sky-sim su[18851]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=upload_artifacts_on_success job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Skipping stage (nothing to do)                      build_stage=upload_artifacts_on_success job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Skipping referees execution                         job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage                               build_stage=cleanup_file_variables job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Cleaning up project directory and file based variables  job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution                   job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim su[18856]: (to gitlab-runner) root on none
May 31 12:22:01 mid-itf-sky-sim su[18856]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Appending trace to coordinator...ok                 code=202 job=4384066441 job-log=0-1049 job-status=running runner=zh_hSFRBy sent-log=0-1048 status=202 Ac>
May 31 12:22:02 mid-itf-sky-sim su[18856]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Job succeeded                                       duration_s=4.291811577 job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Appending trace to coordinator...ok                 code=202 job=4384066441 job-log=0-1430 job-status=running runner=zh_hSFRBy sent-log=1049-1429 status=202>
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Updating job...                                     bytesize=1430 checksum=crc32:a1b07778 job=4384066441 runner=zh_hSFRBy
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Submitting job to coordinator...accepted, but not yet completed  bytesize=1430 checksum=crc32:a1b07778 code=202 job=4384066441 job-status= runner=zh_hSFRBy >
May 31 12:22:03 mid-itf-sky-sim gitlab-runner[18734]: Updating job...                                     bytesize=1430 checksum=crc32:a1b07778 job=4384066441 runner=zh_hSFRBy
May 31 12:22:04 mid-itf-sky-sim gitlab-runner[18734]: Submitting job to coordinator...ok                  bytesize=1430 checksum=crc32:a1b07778 code=200 job=4384066441 job-status= runner=zh_hSFRBy update-interv>

Runner configuration:

 $ sudo cat /etc/gitlab-runner/config.toml
concurrent = 1
log_level = "debug"
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "mid-itf-sky-sim"
  url = "https://gitlab.com"
  id = <snip>
  token = <snip>
  token_obtained_at = <snip>
  token_expires_at = <snip>
  executor = "shell"
  shell = "bash"
  output_limit = 4096

runner version:

p.jordaan@mid-itf-sky-sim:~ $ gitlab-runner --version
Version:      16.0.1
Git revision: 79704081
Git branch:   16-0-stable
GO version:   go1.19.9
Built:        2023-05-23T22:22:28+0000
OS/Arch:      linux/arm

Any ideas why the jobs are not showing any output and don’t appear to execute fully?

2

Do you see any files in /home/gitlab-runner? There is where runner is cloning files.

3

No there is nothing there.
I’ve also tried deleting the user and reinstalling the runner, but this didn’t help.

4

Stupid question, but do you have git installed on the box?

5

yes, the runner was working previously but has recently stopped working.
We are currently trying to determine whether anything changed on our side to cause this.

6

This issue was due to a rogue profile script which ran exit instead of return.
Thanks for your assistance.

1 Like