Pipeline passes all stages, but each stage does nothing

GitLab CI/CD
1

We are seeing a problem with a few of our projects where the pipeline completes all stages successfully, but none of the stages do anything. Indeed, they do not even fetch the repo or artifacts (as the modification time of the gitlab-runner home directory does not change).

Gitlab version 15.6 (Self-managed EE)
Gitlab runner version 15.7.1 (and now 15.7.2)

We have tried pointing the affected projects at different runners that have proven to successfully build other projects. (So, we suspect this is project related and not runner related.)

We have compared the CI/CD settings between affected projects and successful projects.

(May or may not be relevant, but we upgraded about a month ago by creating a new instance and exporting/importing the projects into the new instances.)

The jobs always complete in 0 or 1 seconds without error.

The job output for the build step is always limited to just:

Running with gitlab-runner 15.7.1 (6d480948)
on nodejs runner TKLEXHn3

Resolving secrets 00:00
[MASKED]

The job output for the deploy step is the same except for using a different runner.

Our .gitlab-ci.yml is:

stages:
  - build
  - deploy

build:
  stage:  build
  tags:   [ nodejs ]
  script:
    - ci/this-is-broken-build.sh

  artifacts:
    paths:
      - bpp-dashboard.*.tgz
    expire_in: 1 day

deploy:
  stage: deploy
  tags: [ static-deploy ]
  script:
    - ci/deploy.sh

The actual scripts are irrelevant as they have been verified to never execute.

Hopefully someone has seen this before or has fresh ideas as we have run out of ideas for diagnosing this.

2

Enabling the debug log on the runner would be my first idea to analyse if there is more going on beyond the secrets fetch.

Iā€™d also suggest keeping the GitLab server and runner version the same, to rule out any potential incompatibilities and make troubleshooting more focussed.

Last guess: Maybe the disk is full on the runner hosts.

1 Like
3

Thank you for these suggestions. Unfortunately, they did not change the behavior.

I implemented all 3 suggestions. The results were the same. However, I now see a panic in the system journal now (it was probably there before but I did not think to look for it there):

Journal messages 
systemd[1]: Started GitLab Runner.
gitlab-runner[2084]: Runtime platform                                    arch=amd64 os=linux pid=2084 revision=c081f284 version=15.6.2
gitlab-runner[2084]: Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
gitlab-runner[2084]: Checking runtime mode                               GOOS=linux uid=0
gitlab-runner[2084]: Running in system-mode.                           
gitlab-runner[2084]:                                                   
gitlab-runner[2084]: Configuration loaded                                builds=0
gitlab-runner[2084]: listenaddress: ""
gitlab-runner[2084]: sessionserver:
gitlab-runner[2084]:   listenaddress: ""
gitlab-runner[2084]:   advertiseaddress: ""
gitlab-runner[2084]:   sessiontimeout: 1800
gitlab-runner[2084]: concurrent: 4
gitlab-runner[2084]: checkinterval: 0
gitlab-runner[2084]: loglevel: debug
gitlab-runner[2084]: logformat: null
gitlab-runner[2084]: user: gitlab-runner
gitlab-runner[2084]: runners:
gitlab-runner[2084]: - name: nodejs runner
gitlab-runner[2084]:   limit: 0
gitlab-runner[2084]:   outputlimit: 0
gitlab-runner[2084]:   requestconcurrency: 0
gitlab-runner[2084]:   runnercredentials:
gitlab-runner[2084]:     url: https://scm.at1.netcentrix.net/
gitlab-runner[2084]:     id: 0
gitlab-runner[2084]:     token: XXXXXXXXXXXXXXXXXXXX
gitlab-runner[2084]:     tokenobtainedat: 0001-01-01T00:00:00Z
gitlab-runner[2084]:     tokenexpiresat: 0001-01-01T00:00:00Z
gitlab-runner[2084]:     tlscafile: ""
gitlab-runner[2084]:     tlscertfile: ""
gitlab-runner[2084]:     tlskeyfile: ""
gitlab-runner[2084]:   runnersettings:
gitlab-runner[2084]:     executor: shell
gitlab-runner[2084]:     buildsdir: ""
gitlab-runner[2084]:     cachedir: ""
gitlab-runner[2084]:     cloneurl: ""
gitlab-runner[2084]:     environment: []
gitlab-runner[2084]:     preclonescript: ""
gitlab-runner[2084]:     postclonescript: ""
gitlab-runner[2084]:     prebuildscript: ""
gitlab-runner[2084]:     postbuildscript: ""
gitlab-runner[2084]:     debugtracedisabled: false
gitlab-runner[2084]:     shell: ""
gitlab-runner[2084]:     custombuilddir:
gitlab-runner[2084]:       enabled: false
gitlab-runner[2084]:     referees: null
gitlab-runner[2084]:     cache:
gitlab-runner[2084]:       type: ""
gitlab-runner[2084]:       path: ""
gitlab-runner[2084]:       shared: false
gitlab-runner[2084]:       maxuploadedarchivesize: 0
gitlab-runner[2084]:       s3:
gitlab-runner[2084]:         serveraddress: ""
gitlab-runner[2084]:         accesskey: ""
gitlab-runner[2084]:         secretkey: ""
gitlab-runner[2084]:         bucketname: ""
gitlab-runner[2084]:         bucketlocation: ""
gitlab-runner[2084]:         insecure: false
gitlab-runner[2084]:         authenticationtype: ""
gitlab-runner[2084]:         serversideencryption: ""
gitlab-runner[2084]:         serversideencryptionkeyid: ""
gitlab-runner[2084]:       gcs:
gitlab-runner[2084]:         cachegcscredentials:
gitlab-runner[2084]:           accessid: ""
gitlab-runner[2084]:           privatekey: ""
gitlab-runner[2084]:         credentialsfile: ""
gitlab-runner[2084]:         bucketname: ""
gitlab-runner[2084]:       azure:
gitlab-runner[2084]:         cacheazurecredentials:
gitlab-runner[2084]:           accountname: ""
gitlab-runner[2084]:           accountkey: ""
gitlab-runner[2084]:         containername: ""
gitlab-runner[2084]:         storagedomain: ""
gitlab-runner[2084]:     gracefulkilltimeout: null
gitlab-runner[2084]:     forcekilltimeout: null
gitlab-runner[2084]:     featureflags: {}
gitlab-runner[2084]:     ssh: null
gitlab-runner[2084]:     docker: null
gitlab-runner[2084]:     parallels: null
gitlab-runner[2084]:     virtualbox: null
gitlab-runner[2084]:     machine: null
gitlab-runner[2084]:     kubernetes: null
gitlab-runner[2084]:     custom: null
gitlab-runner[2084]:     autoscaler: null
gitlab-runner[2084]: sentrydsn: null
gitlab-runner[2084]: modtime: 2023-01-20T03:46:44.481453299Z
gitlab-runner[2084]: loaded: true
gitlab-runner[2084]: shutdowntimeout: 0
gitlab-runner[2084]:   builds=0
gitlab-runner[2084]: Waiting for stop signal                             builds=0
gitlab-runner[2084]: listen_address not defined, metrics & debug endpoints disabled  builds=0
gitlab-runner[2084]: [session_server].listen_address not defined, session endpoints disabled  builds=0
gitlab-runner[2084]: Initializing executor providers                     builds=0
gitlab-runner[2084]: Feeding runners to channel                          builds=0
gitlab-runner[2084]: Feeding runner to channel                           builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Starting worker                                     builds=0 worker=0
gitlab-runner[2084]: Processing runner                                   builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider                    builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Starting worker                                     builds=0 worker=3
gitlab-runner[2084]: Starting worker                                     builds=0 worker=2
gitlab-runner[2084]: Acquiring job slot                                  builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot                              builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Starting worker                                     builds=0 worker=1
gitlab-runner[2084]: Dialing: tcp scm.at1.netcentrix.net:443 ...       
gitlab-runner[2084]: Checking for jobs...nothing                         runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel                          builds=0
gitlab-runner[2084]: Feeding runner to channel                           builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner                                   builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider                    builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot                                  builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot                              builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs...nothing                         runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel                          builds=0
gitlab-runner[2084]: Feeding runner to channel                           builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner                                   builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider                    builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot                                  builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot                              builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs...nothing                         runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel                          builds=0
gitlab-runner[2084]: Feeding runner to channel                           builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner                                   builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider                    builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot                                  builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot                              builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs...nothing                         runner=TKLEXHn3
gitlab-runner[2084]: Feeding runners to channel                          builds=0
gitlab-runner[2084]: Feeding runner to channel                           builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner                                   builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider                    builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot                                  builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot                              builds=0 runner=TKLEXHn3
gitlab-runner[2084]: Checking for jobs... received                       job=5590 repo_url=https://scm.at1.netcentrix.net/bluepenguin/bpp-dashboard.git runner=TKLEXHn3
gitlab-runner[2084]: Processing chain                                    chain-leaf=[0xc0007c4b00 0xc000b5b600] context=certificate-chain-build resolve-full-chain=true
gitlab-runner[2084]: Certificate doesn't provide parent URL: exiting the loop  Issuer=Certificate Authority IssuerCertURL=[] Serial=1 Subject=Certificate Authority context=certificate-chain-build
gitlab-runner[2084]: Requeued the runner                                 builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Processing runner                                   builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring executor from provider                    builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring job slot                                  builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Acquiring request slot                              builds=1 runner=TKLEXHn3
gitlab-runner[2084]: Running with gitlab-runner 15.6.2 (c081f284)        job=5590 project=44 runner=TKLEXHn3
gitlab-runner[2084]:   on nodejs runner TKLEXHn3                         job=5590 project=44 runner=TKLEXHn3
gitlab-runner[2084]: Resolving secrets                       job=5590 project=44 runner=TKLEXHn3
gitlab-runner[2084]: Dialing: tcp scm.at1.netcentrix.net:443 ...       
gitlab-runner[2084]: Checking for jobs...nothing                         runner=TKLEXHn3
gitlab-runner[2084]: Appending trace to coordinator...ok                 code=202 job=5590 job-log=0-226 job-status=running runner=TKLEXHn3 sent-log=0-225 status=202 Accepted update-interval=1m0s
gitlab-runner[2084]: Updating job...                                     bytesize=226 checksum=crc32:fd02fcb6 job=5590 runner=TKLEXHn3
gitlab-runner[2084]: Submitting job to coordinator...ok                  bytesize=226 checksum=crc32:fd02fcb6 code=200 job=5590 job-status= runner=TKLEXHn3 update-interval=0s
gitlab-runner[2084]: panic: runtime error: index out of range [0] with length 0
gitlab-runner[2084]: goroutine 36 [running]:
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/masker.(*masker).Write(0xc000742630, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/helpers/trace/internal/masker/masker.go:82 +0x4c5
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/masker.(*masker).Write(0xc000742660, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/helpers/trace/internal/masker/masker.go:146 +0x446
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/masker.(*Masker).Write(0xc000140a61?, {0xc000140a00?, 0x1f?, 0x4f7867?})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/helpers/trace/internal/masker/masker.go:51 +0x26
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer.(*tokenSanitizer).Write(0xc000818080, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer/token_masker.go:202 +0x54c
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer.(*TokenSanitizer).Write(0x6c?, {0xc000140a00?, 0x45399c?, 0xc0007b4450?})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/helpers/trace/internal/tokensanitizer/token_masker.go:90 +0x26
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/helpers/trace.(*Buffer).Write(0xc000b58660, {0xc000140a00, 0x6c, 0x80})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/helpers/trace/buffer.go:132 +0xb3
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/network.(*clientJobTrace).Write(0xc000478270?, {0xc000140a00?, 0x1?, 0xc0007b44a0?})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/network/trace.go:66 +0x25
gitlab-runner[2084]: fmt.Fprint({0x7fdddee45478, 0xc0007c3080}, {0xc0007b44f0, 0x1, 0x1})
gitlab-runner[2084]:         /usr/local/go/src/fmt/print.go:233 +0x75
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*BuildLogger).SendRawLog(...)
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/common/build_logger.go:36
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*BuildLogger).sendLog(0xc0007b45b0, 0xc0007b45a0, {0x1ff193d, 0xe}, {0xc000806770, 0x1, 0x1})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/common/build_logger.go:56 +0x15e
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*BuildLogger).SoftErrorln(...)
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/common/build_logger.go:106
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*Build).setTraceStatus(0xc000072c00, {0x23deeb8, 0xc0007c3080}, {0x23bd120, 0xc000742690})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/common/build.go:821 +0x3fc
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*Build).Run.func2()
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/common/build.go:888 +0x30
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/common.(*Build).Run(0xc000072c00, 0xc000b0c480, {0x23deeb8?, 0xc0007c3080})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/common/build.go:890 +0x90a
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processBuildOnRunner(0xc0001c5400, 0xc0000003c0?, 0xc0007b52d0?, {0x23dc0c0?, 0xc000491f20?}, {0x0, 0x0})
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/commands/multi.go:683 +0x398
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processRunner(0xc0001c5400, 0xc0007b59d8?, 0xc0000003c0, 0x0?)
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/commands/multi.go:645 +0x7ff
gitlab-runner[2084]: gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).processRunners(0xc0001c5400, 0x0?, 0xc0004622a0, 0xc0000507e0)
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/commands/multi.go:583 +0x225
gitlab-runner[2084]: created by gitlab.com/gitlab-org/gitlab-runner/commands.(*RunCommand).startWorkers
gitlab-runner[2084]:         /builds/gitlab-org/gitlab-runner/commands/multi.go:568 +0x33
systemd[1]: gitlab-runner.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.