Jobs linger in "Running" state long after finished according to logs

GitLab CI/CD
,
1

Jobs seem to linger in the running state, blocking followup jobs (and using up pipeline quotas :upside_down_face:), for a significant amount of time past when the job finishes. Currently I’m seeing jobs that are finished in under a minute linger in the running state for as long as five more minutes past the “Job succeeded” log message. Even those that have no artifact/cache interactions.

As a concrete example, I have a job that from start to finish according to the logs took 12s, but the duration according to CI is “3 minutes 26 seconds”.

Another job took roughly 32s, and the recorded duration is “2 minutes 43 seconds”.

Steps to reproduce

See the pipeline configuration below.

Configuration

include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@1.1.0
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: 1.1.0 # component version
      opentofu_version: 1.9.0

stages: [validate, test, build, deploy, cleanup]

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners