Pipelines not being created after upgrade to 17

GitLab CI/CD
1

Problem to solve

When a updated is pushed to a project git repository a pipeline should be created based on .gitlab-ci.yml

Our GitLab on-prem server auto upgraded to v17 during the morning of 5/17 and pipeline have not been working since. We have updated our runner via the new registration workflow. GitLab server admin reported data migration were not complete and had to manually run the migration.

Other symptoms:

  • If I manually create a “Run pipeline” all jobs are stuck " This job has not been triggered yet" “This job depends on upstream jobs that need to succeed in order for this job to be triggered”
  • “Canceling this job” stuck as not triggered and retrying put in Pending status but no runner is assigned.
  • With a job in Pending status if a pause/unpause a runner GitLab will assign the runner for 1 job only.
  • Not all Pipeline showing on the project pipelines page. For example a manually created Pipeline may not show on the pipelines page but can be accessed via ID (…/pipleines/5517)
  • Some Pipelines show status Running eventhough they have no running jobs.

Steps to reproduce

  1. Push a commit to GitLab project containin .gitlab-ci.yml and no pipeline is created

I’ve attempted to change CI/CD pipeline settings for projects, edit or remove .gitlab-ci.yml files, use the API to inspect pipelines and jobs, fork a project, and create a new project with a sample .gitlab-ci.yml file. The common thread is that I cannot get GitLab pipelines to function as they did before the upgrade. I haven’t found any documentation indicating fundamental changes in GitLab pipelines.

Configuration

any update to .gitlab-ci.yml seem to have no change so even the most basic file will not trigger a pipeline creation.

build-job:
  stage: build

  script:
    - echo "Hello, $GITLAB_USER_LOGIN!"

Versions

  • Self-managed
  • GitLab.com SaaS
  • Self-hosted Runners

Versions

  • GitLab Enterprise Edition v17.0.0-ee
  • GitLab Runner, 17.0.0

Some of our issues seem similar to The scheduler failed to assign job to the runner (#395628) · Issues · GitLab.org / GitLab · GitLab

2

Dito. Self-hosted on a on Ubuntu 22.04.4 system, running under KVM/QEMU. Installed via official GitLab repos.

3

I just fixed it. It looks like, the package didn’t get fully configured. After I ran the command to upgrade my system again, it showed me, that it had to configure GitLab (although it was already running on v17). After that, and a reboot, everything works.

4

Thank you for your input on this thread. It’s fantastic to hear that you were able to resolve this issue. Would you mind sharing the command you ran to “upgrade my system again”? It’s not clear to me if the command in question is via the package manager or GitLab itself.

5

It shouldn’t be needed for the package to be reinstalled again. Sometimes after an upgrade if it’s not working properly, at the end of the package installation it does suggest if the installation/upgrade is misbehaving to do:

gitlab-ctl reconfigure
gitlab-ctl restart

should be enough. Sometimes it may also be worth doing:

systemctl restart gitlab-runsvdir

that would save the need to reboot.

6

For me it’s nuitka upgrade. It then told me, that gitlab needs to be configured by dpkg, although it should’ve already done that after the upgrade to v17.

7

FIXED!
Thank you very much @iwalker and @Serpensin for your help.

Report from our GitLab system admin:

gitlab-ctl reconfigure

was failing due to Gitlab KAS

failed to parse "2024-05-15T09:09:49+0000" into RFC3339 compliant time object

We don’t use Kubernetes so disabled via /etc/gitlab/gitlab.rb

gitlab_kas['enable'] = false

Then the reconfigure would run. After reboot backlogged email and integrations messages flooded through. My pipelines returned to functioning normally.

1 Like