We have a gitlab-runner on a Raspberry Pi which is not executing jobs properly. The jobs are succeeding, but before_script, script & after_script are not executing.
Example job: check-current (#4384050801) · Jobs · SKAO / SKA Mid Sky Simulator Controller · GitLab
In the runner logs, I can see that the job is picked up and it does say that it is pulling the repo and executing the “step_script” stage of the job script. However, there is no output, and the job finishes in a few seconds, despite adding multiple sleeps of 2 minutes.
The job configuration can be found here:
https://gitlab.com/ska-telescope/ska-te-mid-skysimctl/-/blob/at-553/.gitlab-ci.yml?ref_type=heads#L100
The runner output is as follows:
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Checking for jobs...nothing runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Processing runner builds=0 runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Acquiring executor from provider builds=0 runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Acquiring job slot builds=0 runner=zh_hSFRBy
May 31 12:21:56 mid-itf-sky-sim gitlab-runner[18734]: Acquiring request slot builds=0 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Checking for jobs... received job=4384066441 repo_url=https://gitlab.com/ska-telescope/ska-te-mid-skysimctl.git runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Processing chain chain-leaf=[0x48ff8c0 0x48ffb80 0x4dd3340] context=certificate-chain-build resolve-full-chain=true
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Certificate doesn't provide parent URL: exiting the loop Issuer=Baltimore CyberTrust Root IssuerCertURL=[] Serial=<snip> Subject=Baltimore CyberTrust Roo>
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Verifying last certificate to find the final root certificate Issuer=Baltimore CyberTrust Root IssuerCertURL=[] Serial=<snip> Subject=Baltimore CyberTrus>
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Failed to requeue the runner builds=1 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Running with gitlab-runner 16.0.1 (79704081) job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: on mid-itf-sky-sim zh_hSFRBy, system ID: s_33e8ecff1544 job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Resolving secrets job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Preparing the "shell" executor job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Shell configuration: command: su
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: arguments:
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - -s
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - /bin/bash
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - gitlab-runner
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - -c
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - bash -l
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: cmdline: su -s /bin/bash gitlab-runner -c $'bash -l'
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: dockercommand:
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - sh
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - -c
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: - "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash -l\nelif [ -x /usr/bin/bash
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: ]; then\n\texec /usr/bin/bash -l\nelif [ -x /bin/bash ]; then\n\texec /bin/bash
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: -l\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh -l\nelif [ -x
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: /usr/bin/sh ]; then\n\texec /usr/bin/sh -l\nelif [ -x /bin/sh ]; then\n\texec /bin/sh
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: -l\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh -l\nelse\n\techo shell not
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: found\n\texit 1\nfi\n\n"
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: passfile: false
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: extension: ""
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Using Shell (bash) executor... job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Waiting for signals... job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: No referees configured job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=prepare_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Preparing environment job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:57 mid-itf-sky-sim su[18809]: (to gitlab-runner) root on none
May 31 12:21:57 mid-itf-sky-sim su[18809]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=get_sources job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Getting source from Git repository job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim su[18830]: (to gitlab-runner) root on none
May 31 12:21:59 mid-itf-sky-sim su[18830]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:21:59 mid-itf-sky-sim su[18830]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=restore_cache job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Restoring cache job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:21:59 mid-itf-sky-sim su[18835]: (to gitlab-runner) root on none
May 31 12:21:59 mid-itf-sky-sim su[18835]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Feeding runners to channel builds=1
May 31 12:21:59 mid-itf-sky-sim gitlab-runner[18734]: Feeding runner to channel builds=1 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim su[18835]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=download_artifacts job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Skipping stage (nothing to do) build_stage=download_artifacts job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=step_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing "step_script" stage of the job script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim su[18840]: (to gitlab-runner) root on none
May 31 12:22:00 mid-itf-sky-sim su[18840]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:00 mid-itf-sky-sim su[18840]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=after_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Running after_script job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:00 mid-itf-sky-sim su[18845]: (to gitlab-runner) root on none
May 31 12:22:00 mid-itf-sky-sim su[18845]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:01 mid-itf-sky-sim su[18845]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=archive_cache job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: WARNING: Cache config not defined. Skipping cache operation.
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Saving cache for successful job job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim su[18851]: (to gitlab-runner) root on none
May 31 12:22:01 mid-itf-sky-sim su[18851]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:01 mid-itf-sky-sim su[18851]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=upload_artifacts_on_success job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Skipping stage (nothing to do) build_stage=upload_artifacts_on_success job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Skipping referees execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Executing build stage build_stage=cleanup_file_variables job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Cleaning up project directory and file based variables job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Using new shell command execution job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:01 mid-itf-sky-sim su[18856]: (to gitlab-runner) root on none
May 31 12:22:01 mid-itf-sky-sim su[18856]: pam_unix(su:session): session opened for user gitlab-runner(uid=999) by (uid=0)
May 31 12:22:01 mid-itf-sky-sim gitlab-runner[18734]: Appending trace to coordinator...ok code=202 job=4384066441 job-log=0-1049 job-status=running runner=zh_hSFRBy sent-log=0-1048 status=202 Ac>
May 31 12:22:02 mid-itf-sky-sim su[18856]: pam_unix(su:session): session closed for user gitlab-runner
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Job succeeded duration_s=4.291811577 job=4384066441 project=42553017 runner=zh_hSFRBy
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Appending trace to coordinator...ok code=202 job=4384066441 job-log=0-1430 job-status=running runner=zh_hSFRBy sent-log=1049-1429 status=202>
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Updating job... bytesize=1430 checksum=crc32:a1b07778 job=4384066441 runner=zh_hSFRBy
May 31 12:22:02 mid-itf-sky-sim gitlab-runner[18734]: Submitting job to coordinator...accepted, but not yet completed bytesize=1430 checksum=crc32:a1b07778 code=202 job=4384066441 job-status= runner=zh_hSFRBy >
May 31 12:22:03 mid-itf-sky-sim gitlab-runner[18734]: Updating job... bytesize=1430 checksum=crc32:a1b07778 job=4384066441 runner=zh_hSFRBy
May 31 12:22:04 mid-itf-sky-sim gitlab-runner[18734]: Submitting job to coordinator...ok bytesize=1430 checksum=crc32:a1b07778 code=200 job=4384066441 job-status= runner=zh_hSFRBy update-interv>
Runner configuration:
$ sudo cat /etc/gitlab-runner/config.toml
concurrent = 1
log_level = "debug"
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "mid-itf-sky-sim"
url = "https://gitlab.com"
id = <snip>
token = <snip>
token_obtained_at = <snip>
token_expires_at = <snip>
executor = "shell"
shell = "bash"
output_limit = 4096
runner version:
p.jordaan@mid-itf-sky-sim:~ $ gitlab-runner --version
Version: 16.0.1
Git revision: 79704081
Git branch: 16-0-stable
GO version: go1.19.9
Built: 2023-05-23T22:22:28+0000
OS/Arch: linux/arm
Any ideas why the jobs are not showing any output and don’t appear to execute fully?