Self-hosted gitlab CI runner Shell executer fails immediately with ERROR: Job failed: exit status 1

Hello, I’ve been running a self-hosted gitlab-ee for over a year now and never had any issues with CI. Recently I ran

$ gitlab-ctl registry-garbage-collect -m
$ gitlab-ctl registry-garbage-collect

to free up space on the disk and since around then, all my pipeline steps fail immediately in the same way:

1 Running with gitlab-runner 12.6.0 (ac8e767a)
2 on <description> LEiUnv7K
3 Using Shell executor...
5 Running on Debian-93-stretch-64-minimal...
8 ERROR: Job failed: exit status 1

I’ve updated and reconfigured the gitlab-runner, I’ve updated the registry, but nothing helps.
Here is the gitlab-runner syslog, ther only other thing that I see in there for each pipeline step is:
WARNING: Failed to process runner

Jan  9 15:00:01 Debian-93-stretch-64-minimal CRON[23923]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jan  9 15:00:01 Debian-93-stretch-64-minimal CRON[23924]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Jan  9 15:00:50 Debian-93-stretch-64-minimal gitlab-runner[23054]: Checking for jobs... received                     #033[0;m  job#033[0;m=1086 repo_url#033[0;m=https://gitlab.shiftcrypto.ch/shiftdevices/webshop-saleor.git runner#033[0;m=LEiUnv7K
Jan  9 15:00:50 Debian-93-stretch-64-minimal gitlab-runner[23054]: Checking for jobs... received                     #033[0;m  job#033[0;m=1086 repo_url#033[0;m=https://gitlab.shiftcrypto.ch/shiftdevices/webshop-saleor.git runner#033[0;m=LEiUnv7K
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Created slice User Slice of UID 999.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Starting User Runtime Directory /run/user/999...
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Started User Runtime Directory /run/user/999.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Starting User Manager for UID 999...
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Reached target Paths.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Listening on GnuPG network certificate management daemon.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Listening on GnuPG cryptographic agent and passphrase cache.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Reached target Sockets.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Reached target Timers.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Reached target Basic System.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Reached target Default.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[24267]: Startup finished in 43ms.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Started User Manager for UID 999.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Started Session c43 of user gitlab-runner.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: session-c43.scope: Succeeded.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: Started Session c44 of user gitlab-runner.
Jan  9 15:00:50 Debian-93-stretch-64-minimal systemd[1]: session-c44.scope: Succeeded.
Jan  9 15:00:50 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Job failed: exit status 1                #033[0;m  #033[0;33mduration#033[0;m=501.835123ms #033[0;33mjob#033[0;m=1086 #033[0;33mproject#033[0;m=1 #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:50 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Job failed: exit status 1                #033[0;m  #033[0;33mduration#033[0;m=501.835123ms #033[0;33mjob#033[0;m=1086 #033[0;33mproject#033[0;m=1 #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:51 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Failed to process runner                 #033[0;m  #033[0;33mbuilds#033[0;m=0 #033[0;33merror#033[0;m=exit status 1 #033[0;33mexecutor#033[0;m=shell #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:51 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Failed to process runner                 #033[0;m  #033[0;33mbuilds#033[0;m=0 #033[0;33merror#033[0;m=exit status 1 #033[0;33mexecutor#033[0;m=shell #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:53 Debian-93-stretch-64-minimal gitlab-runner[23054]: Checking for jobs... received                     #033[0;m  job#033[0;m=1087 repo_url#033[0;m=https://gitlab.shiftcrypto.ch/shiftdevices/webshop-saleor.git runner#033[0;m=LEiUnv7K
Jan  9 15:00:53 Debian-93-stretch-64-minimal gitlab-runner[23054]: Checking for jobs... received                     #033[0;m  job#033[0;m=1087 repo_url#033[0;m=https://gitlab.shiftcrypto.ch/shiftdevices/webshop-saleor.git runner#033[0;m=LEiUnv7K
Jan  9 15:00:53 Debian-93-stretch-64-minimal systemd[1]: Started Session c45 of user gitlab-runner.
Jan  9 15:00:53 Debian-93-stretch-64-minimal systemd[1]: session-c45.scope: Succeeded.
Jan  9 15:00:53 Debian-93-stretch-64-minimal systemd[1]: Started Session c46 of user gitlab-runner.
Jan  9 15:00:53 Debian-93-stretch-64-minimal systemd[1]: session-c46.scope: Succeeded.
Jan  9 15:00:53 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Job failed: exit status 1                #033[0;m  #033[0;33mduration#033[0;m=79.142196ms #033[0;33mjob#033[0;m=1087 #033[0;33mproject#033[0;m=1 #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:53 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Job failed: exit status 1                #033[0;m  #033[0;33mduration#033[0;m=79.142196ms #033[0;33mjob#033[0;m=1087 #033[0;33mproject#033[0;m=1 #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:53 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Failed to process runner                 #033[0;m  #033[0;33mbuilds#033[0;m=0 #033[0;33merror#033[0;m=exit status 1 #033[0;33mexecutor#033[0;m=shell #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:53 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Failed to process runner                 #033[0;m  #033[0;33mbuilds#033[0;m=0 #033[0;33merror#033[0;m=exit status 1 #033[0;33mexecutor#033[0;m=shell #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:56 Debian-93-stretch-64-minimal gitlab-runner[23054]: Checking for jobs... received                     #033[0;m  job#033[0;m=1088 repo_url#033[0;m=https://gitlab.shiftcrypto.ch/shiftdevices/webshop-saleor.git runner#033[0;m=LEiUnv7K
Jan  9 15:00:56 Debian-93-stretch-64-minimal gitlab-runner[23054]: Checking for jobs... received                     #033[0;m  job#033[0;m=1088 repo_url#033[0;m=https://gitlab.shiftcrypto.ch/shiftdevices/webshop-saleor.git runner#033[0;m=LEiUnv7K
Jan  9 15:00:56 Debian-93-stretch-64-minimal systemd[1]: Started Session c47 of user gitlab-runner.
Jan  9 15:00:56 Debian-93-stretch-64-minimal systemd[1]: session-c47.scope: Succeeded.
Jan  9 15:00:56 Debian-93-stretch-64-minimal systemd[1]: Started Session c48 of user gitlab-runner.
Jan  9 15:00:56 Debian-93-stretch-64-minimal systemd[1]: session-c48.scope: Succeeded.
Jan  9 15:00:56 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Job failed: exit status 1                #033[0;m  #033[0;33mduration#033[0;m=149.358886ms #033[0;33mjob#033[0;m=1088 #033[0;33mproject#033[0;m=1 #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:56 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Job failed: exit status 1                #033[0;m  #033[0;33mduration#033[0;m=149.358886ms #033[0;33mjob#033[0;m=1088 #033[0;33mproject#033[0;m=1 #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:56 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Failed to process runner                 #033[0;m  #033[0;33mbuilds#033[0;m=0 #033[0;33merror#033[0;m=exit status 1 #033[0;33mexecutor#033[0;m=shell #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:00:56 Debian-93-stretch-64-minimal gitlab-runner[23054]: #033[0;33mWARNING: Failed to process runner                 #033[0;m  #033[0;33mbuilds#033[0;m=0 #033[0;33merror#033[0;m=exit status 1 #033[0;33mexecutor#033[0;m=shell #033[0;33mrunner#033[0;m=LEiUnv7K
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: Stopping User Manager for UID 999...
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Stopped target Default.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Stopped target Basic System.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Stopped target Timers.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Stopped target Paths.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Stopped target Sockets.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: gpg-agent-ssh.socket: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: gpg-agent-extra.socket: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: dirmngr.socket: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Closed GnuPG network certificate management daemon.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: gpg-agent-browser.socket: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: gpg-agent.socket: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Closed GnuPG cryptographic agent and passphrase cache.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Reached target Shutdown.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: systemd-exit.service: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Started Exit the Session.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[24267]: Reached target Exit the Session.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: user@999.service: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: Stopped User Manager for UID 999.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: Stopping User Runtime Directory /run/user/999...
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[2788]: run-user-999.mount: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: run-user-999.mount: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: user-runtime-dir@999.service: Succeeded.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: Stopped User Runtime Directory /run/user/999.
Jan  9 15:01:06 Debian-93-stretch-64-minimal systemd[1]: Removed slice User Slice of UID 999.
Jan  9 15:04:23 Debian-93-stretch-64-minimal gitlab-runner[23054]: Configuration loaded                              #033[0;m  builds#033[0;m=0
Jan  9 15:04:23 Debian-93-stretch-64-minimal gitlab-runner[23054]: Configuration loaded                              #033[0;m  builds#033[0;m=0

I haven’t found any very helpful logs or errors that would point me to what is going wrong. Any help would be greatly appreciated.

Hi,

maybe you could run the GitLab runner in debug mode to get more insights on what’s going on.
https://docs.gitlab.com/runner/faq/#run-in---debug-mode

Also, check the changelog for 12.6.3. whether you see something related and do an upgrade in case.

Cheers,
Michael

Hi @TomasV
On my team we are also facing something really similar.
The runner has been working fine since long time now, and a few days ago we got also the following: “WARNING: Job failed: exit status 1” and 3 lines below: “WARNING: Failed to process runner”.

This 2 “WARNING” logs are triggered and the pipeline is breaking ONLY when we use on the .gitlab-ci.yaml file a cd bash command to 2 specific project folders… We reinstall the runner, we re-register the runner and give all permissions to these 2 folders, but nothing solves the problem.

@TomasV , did you solved the problem? How did you do it?
If not, does anybody have an idea about which can be the reason for this?

Thank you in advance

I surprisingly found the bug

ERROR: Build failed with: exit status 1 (#114) · Issues · GitLab.org / gitlab-runner · GitLab

Apparently the rvm(2.6.0), that I had installed previously, was overriding the cd bash command
So you can either unset cd or, in my case, uninstall the rvm if you don’t need it