Working runner config until suddenly "Failed to process runner"

Hi all!

My pipelines all began failing on the initial step 4 days ago, which is a simple script to auto-increment the build number, with an opaque error. gitlab-runner version was 13.5.0 (ece86343) on the Gitlab SaaS, not self-hosted, and is now 14.8.2, same error.

NOTE: There was a Gitlab deploy on March 2nd, it looks like, and my last successful job was March 3rd at 5 AM MT, in case there was some weird bug introduced.

The runner shows as active, and can connect just fine. I even re-registered a new runner to test that - also connects just fine, picks up the job no problem, just fails in the same way.

I retried a successful job that worked before the failure (5 days ago), and it also now fails with the same output.

It’s a pretty bog-standard registered runner with a stock generated config (name, cache, executor of shell, url, token).

On to the errors! You can see in the “What I used to get” section all that this task is doing - literally, those three commands are it.
The URLs were replaced with urlxxxx because the forum only allows 10 URLs in the post - it’s just the normal gitlab job url.
The commands have ? instead of forward slashes for the same reason.

What I get now (web output):

Skipping Git submodules setup

15 (urlxxxx#L15)Executing “step_script” stage of the job script

16 (urlxxxx#L16)$ VAR=$(curl -s -f --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” | jq -r ‘.value’ )

17 (urlxxxx#L17)Cleaning up file based variables

18 (urlxxxx#L18)ERROR: Job failed: exit status 1

What I used to get:

Skipping Git submodules setup

15 (urlxxxx#L15)Executing “step_script” stage of the job script

16 (urlxxxx#L16)$ VAR=$(curl -s -f --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” | jq -r ‘.value’ )

17 (urlxxxx#L17)$ let VAR=VAR+1

18 (urlxxxx#L18)$ curl -s -f --request PUT --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” --form “value=${VAR}”

19 (urlxxxx#L19){“variable_type”:“env_var”,“key”:“BUILD_NUMBER”,“value”:“1288”,“protected”:false,“masked”:false,“environment_scope”:"*"}

21 (urlxxxx#L21)Cleaning up file based variables

23 (urlxxxx#L23)Job succeeded

Debug log below, in case it helps one of you more than it did me :slight_smile: :

WARNING: Job failed: exit status 1 duration=1m55.724408311s job=xxxx project=zzzz runner=yyyy
Appending trace to coordinator… ok code=202 job=xxxx job-log=0-1366 job-status=running runner=yyyy sent-log=724-1365 status=202 Accepted update-interval=3s
Submitting job to coordinator… accepted, but not yet completed code=202 job=xxxx job-status= runner=yyyy update-interval=1s
Submitting job to coordinator… ok code=200 job=xxxx job-status= runner=yyyy update-interval=0s
WARNING: Failed to process runner builds=0 error=exit status 1 executor=shell runner=yyyy

The relevant section of the

Auto Increment:
stage: Init
- macos
GITLAB_URL: “the expected URL”
- ‘VAR=$(curl -s -f --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” | jq -r ‘’.value’’ ) ’
- let VAR=VAR+1
- 'curl -s -f --request PUT --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” --form “value=${VAR}” ’

Summary of what I’ve tried: restarting stuck job, restarting runner, recreating runner entirely, checking network connectivity, testing formerly working job to confirm that it’s a gitlab runner thing and not a code change somehow. Have also tried the url with and without the trailing / for the runner, although I doubt that matters. Tested updating gitlab-runner to 14.8.2 as well. This is running on a mac, which I have to do to build for iOS.

Thanks for the help! I’m truly stuck.



For future reference - eventually got around this by creating/incrementing build numbers another way (using a date command and the pipeline job id).