Hi all!
My pipelines all began failing on the initial step 4 days ago, which is a simple script to auto-increment the build number, with an opaque error. gitlab-runner version was 13.5.0 (ece86343) on the Gitlab SaaS, not self-hosted, and is now 14.8.2, same error.
NOTE: There was a Gitlab deploy on March 2nd, it looks like, and my last successful job was March 3rd at 5 AM MT, in case there was some weird bug introduced.
The runner shows as active, and can connect just fine. I even re-registered a new runner to test that - also connects just fine, picks up the job no problem, just fails in the same way.
I retried a successful job that worked before the failure (5 days ago), and it also now fails with the same output.
It’s a pretty bog-standard registered runner with a stock generated config (name, cache, executor of shell, url, token).
On to the errors! You can see in the “What I used to get” section all that this task is doing - literally, those three commands are it.
The URLs were replaced with urlxxxx because the forum only allows 10 URLs in the post - it’s just the normal gitlab job url.
The commands have ? instead of forward slashes for the same reason.
What I get now (web output):
…
Skipping Git submodules setup
15 (urlxxxx#L15)Executing “step_script” stage of the job script
16 (urlxxxx#L16)$ VAR=$(curl -s -f --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” | jq -r ‘.value’ )
17 (urlxxxx#L17)Cleaning up file based variables
18 (urlxxxx#L18)ERROR: Job failed: exit status 1
What I used to get:
…
Skipping Git submodules setup
15 (urlxxxx#L15)Executing “step_script” stage of the job script
16 (urlxxxx#L16)$ VAR=$(curl -s -f --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” | jq -r ‘.value’ )
17 (urlxxxx#L17)$ let VAR=VAR+1
18 (urlxxxx#L18)$ curl -s -f --request PUT --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” --form “value=${VAR}”
19 (urlxxxx#L19){“variable_type”:“env_var”,“key”:“BUILD_NUMBER”,“value”:“1288”,“protected”:false,“masked”:false,“environment_scope”:"*"}
21 (urlxxxx#L21)Cleaning up file based variables
23 (urlxxxx#L23)Job succeeded
Debug log below, in case it helps one of you more than it did me 
:
WARNING: Job failed: exit status 1 duration=1m55.724408311s job=xxxx project=zzzz runner=yyyy
Appending trace to coordinator… ok code=202 job=xxxx job-log=0-1366 job-status=running runner=yyyy sent-log=724-1365 status=202 Accepted update-interval=3s
Submitting job to coordinator… accepted, but not yet completed code=202 job=xxxx job-status= runner=yyyy update-interval=1s
Submitting job to coordinator… ok code=200 job=xxxx job-status= runner=yyyy update-interval=0s
WARNING: Failed to process runner builds=0 error=exit status 1 executor=shell runner=yyyy
The relevant section of the .gitlab.ci.yml:
Auto Increment:
stage: Init
tags:
- macos
variables:
VAR_NAME: BUILD_NUMBER
TOKEN: ${CI_PIPELINE_IID_TOKEN}
GITLAB_URL: “the expected URL”
script:
- ‘VAR=$(curl -s -f --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” | jq -r ‘’.value’’ ) ’
- let VAR=VAR+1
- 'curl -s -f --request PUT --header “PRIVATE-TOKEN: ${TOKEN}” “${GITLAB_URL}?api?v4?projects?${CI_PROJECT_ID}?variables?${VAR_NAME}” --form “value=${VAR}” ’
Summary of what I’ve tried: restarting stuck job, restarting runner, recreating runner entirely, checking network connectivity, testing formerly working job to confirm that it’s a gitlab runner thing and not a code change somehow. Have also tried the url with and without the trailing / for the runner, although I doubt that matters. Tested updating gitlab-runner to 14.8.2 as well. This is running on a mac, which I have to do to build for iOS.
Thanks for the help! I’m truly stuck.
Cheers,
Rob