We are seeing a problem with a few of our projects where the pipeline completes all stages successfully, but none of the stages do anything. Indeed, they do not even fetch the repo or artifacts (as the modification time of the gitlab-runner home directory does not change).
Gitlab version 15.6 (Self-managed EE)
Gitlab runner version 15.7.1 (and now 15.7.2)
We have tried pointing the affected projects at different runners that have proven to successfully build other projects. (So, we suspect this is project related and not runner related.)
We have compared the CI/CD settings between affected projects and successful projects.
(May or may not be relevant, but we upgraded about a month ago by creating a new instance and exporting/importing the projects into the new instances.)
The jobs always complete in 0 or 1 seconds without error.
The job output for the build step is always limited to just:
Running with gitlab-runner 15.7.1 (6d480948)
on nodejs runner TKLEXHn3
Resolving secrets 00:00
[MASKED]
The job output for the deploy step is the same except for using a different runner.
Our .gitlab-ci.yml is:
stages:
- build
- deploy
build:
stage: build
tags: [ nodejs ]
script:
- ci/this-is-broken-build.sh
artifacts:
paths:
- bpp-dashboard.*.tgz
expire_in: 1 day
deploy:
stage: deploy
tags: [ static-deploy ]
script:
- ci/deploy.sh
The actual scripts are irrelevant as they have been verified to never execute.
Hopefully someone has seen this before or has fresh ideas as we have run out of ideas for diagnosing this.
Enabling the debug log on the runner would be my first idea to analyse if there is more going on beyond the secrets fetch.
Iād also suggest keeping the GitLab server and runner version the same, to rule out any potential incompatibilities and make troubleshooting more focussed.
Last guess: Maybe the disk is full on the runner hosts.
1 Like
Thank you for these suggestions. Unfortunately, they did not change the behavior.
I implemented all 3 suggestions. The results were the same. However, I now see a panic in the system journal now (it was probably there before but I did not think to look for it there):
Journal messages
systemd[1]: Started GitLab Runner.
gitlab-runner[2084]: Runtime platform arch=amd64 os=linux pid=2084 revision=c081f284 version=15.6.2
gitlab-runner[2084]: Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0
gitlab-runner[2084]: Checking runtime mode GOOS=linux uid=0
gitlab-runner[2084]: Running in system-mode.
gitlab-runner[2084]:
gitlab-runner[2084]: Configuration loaded builds=0
gitlab-runner[2084]: listenaddress: ""
gitlab-runner[2084]: sessionserver:
gitlab-runner[2084]: listenaddress: ""
gitlab-runner[2084]: advertiseaddress: ""
gitlab-runner[2084]: sessiontimeout: 1800
gitlab-runner[2084]: concurrent: 4
gitlab-runner[2084]: checkinterval: 0
gitlab-runner[2084]: loglevel: debug
gitlab-runner[2084]: logformat: null
gitlab-runner[2084]: user: gitlab-runner
gitlab-runner[2084]: runners:
gitlab-runner[2084]: - name: nodejs runner
gitlab-runner[2084]: limit: 0
gitlab-runner[2084]: outputlimit: 0
gitlab-runner[2084]: requestconcurrency: 0
gitlab-runner[2084]: runnercredentials:
gitlab-runner[2084]: url: https://scm.at1.netcentrix.net/
gitlab-runner[2084]: id: 0
gitlab-runner[2084]: token: XXXXXXXXXXXXXXXXXXXX
gitlab-runner[2084]: tokenobtainedat: 0001-01-01T00:00:00Z
gitlab-runner[2084]: tokenexpiresat: 0001-01-01T00:00:00Z
gitlab-runner[2084]: tlscafile: ""
gitlab-runner[2084]: tlscertfile: ""
gitlab-runner[2084]: tlskeyfile: ""
gitlab-runner[2084]: runnersettings:
gitlab-runner[2084]: executor: shell
gitlab-runner[2084]: buildsdir: ""
gitlab-runner[2084]: cachedir: ""
gitlab-runner[2084]: cloneurl: ""
gitlab-runner[2084]: environment: []
gitlab-runner[2084]: preclonescript: ""
gitlab-runner[2084]: postclonescript: ""
gitlab-runner[2084]: prebuildscript: ""
gitlab-runner[2084]: postbuildscript: ""
gitlab-runner[2084]: debugtracedisabled: false
gitlab-runner[2084]: shell: ""
gitlab-runner[2084]: custombuilddir:
gitlab-runner[2084]: enabled: false
gitlab-runner[2084]: referees: null
gitlab-runner[2084]: cache:
gitlab-runner[2084]: type: ""
gitlab-runner[2084]: path: ""
gitlab-runner[2084]: shared: false
gitlab-runner[2084]: maxuploadedarchivesize: 0
gitlab-runner[2084]: s3:
gitlab-runner[2084]: serveraddress: ""
gitlab-runner[2084]: accesskey: ""
gitlab-runner[2084]: secretkey: ""
gitlab-runner[2084]: bucketname: ""
gitlab-runner[2084]: bucketlocation: ""
gitlab-runner[2084]: insecure: false
gitlab-runner[2084]: authenticationtype: ""
gitlab-runner[2084]: serversideencryption: ""
gitlab-runner[2084]: serversideencryptionkeyid: ""
gitlab-runner[2084]: gcs:
gitlab-runner[2084]: cachegcscredentials:
gitlab-runner[2084]: accessid: ""
gitlab-runner[2084]: privatekey: ""
gitlab-runner[2084]: credentialsfile: ""
gitlab-runner[2084]: bucketname: ""
gitlab-runner[2084]: azure:
gitlab-runner[2084]: cacheazurecredentials:
gitlab-runner[2084]: accountname: ""
gitlab-runner[2084]: accountkey: ""
gitlab-runner[2084]: containername: ""
gitlab-runner[2084]: storagedomain: ""
gitlab-runner[2084]: gracefulkilltimeout: null
gitlab-runner[2084]: forcekilltimeout: null
gitlab-runner[2084]: featureflags: {}
gitlab-runner[2084]: ssh: null
gitlab-runner[2084]: docker: null
gitlab-runner[2084]: parallels: null
gitlab-runner[2084]: virtualbox: null
gitlab-runner[2084]: machine: null
gitlab-runner[2084]: kubernetes: null
gitlab-runner[2084]: custom: null
gitlab-runner[2084]: autoscaler: null
gitlab-runner[2084]: sentrydsn: null
gitlab-runner[2084]: modtime: 2023-01-20T03:46:44.481453299Z
gitlab-runner[2084]: loaded: true
gitlab-runner[2084]: shutdowntimeout: 0
gitlab-runner[2084]: builds=0
gitlab-runner[2084]: Waiting for stop signal builds=0
gitlab-runner[2084]: listen_address not defined, metrics & debug endpoints disabled builds=0
gitlab-runner[2084]: [session_server].listen_address not defined, session endpoints disabled builds=0
gitlab-runner[2084]: Initializing executor providers builds=0
gitlab-runner[2084]: Feeding runners to channel builds=0
gitlab-runner[2084]: Feeding runner to channel builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Starting worker builds=0 worker=0
gitlab-runner[2084]: Processing runner builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Starting worker builds=0 worker=3
gitlab-runner[2084]: Starting worker builds=0 worker=2
gitlab-runner[2084]: Acquiring job slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Starting worker builds=0 worker=1
gitlab-runner[2084]: Dialing: tcp scm.at1.netcentrix.net:443 ...
gitlab-runner[2084]: Checking for jobs...nothing runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel builds=0
gitlab-runner[2084]: Feeding runner to channel builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs...nothing runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel builds=0
gitlab-runner[2084]: Feeding runner to channel builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs...nothing runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel builds=0
gitlab-runner[2084]: Feeding runner to channel builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs...nothing runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel builds=0
gitlab-runner[2084]: Feeding runner to channel builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs... received job=5590 repo_url=https://scm.at1.netcentrix.net/bluepenguin/bpp-dashboard.git runner=TKLEXHn3
gitlab-runner[2084]: Processing chain chain-leaf=[0xc0007c4b00 0xc000b5b600] context=certificate-chain-build resolve-full-chain=true
gitlab-runner[2084]: Certificate doesn't provide parent URL: exiting the loop Issuer=Certificate Authority IssuerCertURL=[] Serial=1 Subject=Certificate Authority context=certificate-chain-build
gitlab-runner[2084]: Requeued the runner builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Running with gitlab-runner 15.6.2 (c081f284) job=5590 project=44 runner=TKLEXHn3
gitlab-runner[2084]: on nodejs runner TKLEXHn3 job=5590 project=44 runner=TKLEXHn3
gitlab-runner[2084]: Resolving secrets job=5590 project=44 runner=TKLEXHn3
gitlab-runner[2084]: Dialing: tcp scm.at1.netcentrix.net:443 ...
gitlab-runner[2084]: Checking for jobs...nothing runner=TKLEXHn3
gitlab-runner[2084]: Appending trace to coordinator...ok code=202 job=5590 job-log=0-226 job-status=running runner=TKLEXHn3 sent-log=0-225 status=202 Accepted update-interval=1m0s
gitlab-runner[2084]: Updating job... bytesize=226 checksum=crc32:fd02fcb6 job=5590 runner=TKLEXHn3
gitlab-runner[2084]: Submitting job to coordinator...ok bytesize=226 checksum=crc32:fd02fcb6 code=200 job=5590 job-status= runner=TKLEXHn3 update-interval=0s
gitlab-runner[2084]: panic: runtime error: index out of range [0] with length 0
gitlab-runner[2084]: goroutine 36 [running]:
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/masker.(*masker).Write(0xc000742630, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/helpers/trace/internal/masker/masker.go:82 +0x4c5
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/masker.(*masker).Write(0xc000742660, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/helpers/trace/internal/masker/masker.go:146 +0x446
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/masker.(*Masker).Write(0xc000140a61?, {0xc000140a00?, 0x1f?, 0x4f7867?})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/helpers/trace/internal/masker/masker.go:51 +0x26
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer.(*tokenSanitizer).Write(0xc000818080, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer/token_masker.go:202 +0x54c
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer.(*TokenSanitizer).Write(0x6c?, {0xc000140a00?, 0x45399c?, 0xc0007b4450?})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer/token_masker.go:90 +0x26
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace.(*Buffer).Write(0xc000b58660, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/helpers/trace/buffer.go:132 +0xb3
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/network.(*clientJobTrace).Write(0xc000478270?, {0xc000140a00?, 0x1?, 0xc0007b44a0?})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/network/trace.go:66 +0x25
gitlab-runner[2084]: fmt.Fprint({0x7fdddee45478, 0xc0007c3080}, {0xc0007b44f0, 0x1, 0x1})
gitlab-runner[2084]: /usr/local/go/src/fmt/print.go:233 +0x75
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*BuildLogger).SendRawLog(...)
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/common/build_logger.go:36
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*BuildLogger).sendLog(0xc0007b45b0, 0xc0007b45a0, {0x1ff193d, 0xe}, {0xc000806770, 0x1, 0x1})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/common/build_logger.go:56 +0x15e
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*BuildLogger).SoftErrorln(...)
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/common/build_logger.go:106
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*Build).setTraceStatus(0xc000072c00, {0x23deeb8, 0xc0007c3080}, {0x23bd120, 0xc000742690})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/common/build.go:821 +0x3fc
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*Build).Run.func2()
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/common/build.go:888 +0x30
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*Build).Run(0xc000072c00, 0xc000b0c480, {0x23deeb8?, 0xc0007c3080})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/common/build.go:890 +0x90a
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processBuildOnRunner(0xc0001c5400, 0xc0000003c0?, 0xc0007b52d0?, {0x23dc0c0?, 0xc000491f20?}, {0x0, 0x0})
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/commands/multi.go:683 +0x398
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processRunner(0xc0001c5400, 0xc0007b59d8?, 0xc0000003c0, 0x0?)
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/commands/multi.go:645 +0x7ff
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processRunners(0xc0001c5400, 0x0?, 0xc0004622a0, 0xc0000507e0)
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/commands/multi.go:583 +0x225
gitlab-runner[2084]: created by gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).startWorkers
gitlab-runner[2084]: /builds/gitlab-org/gitlab-runner/commands/multi.go:568 +0x33
systemd[1]: gitlab-runner.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.