None of our pipelines running after update to GitLab 15

GitLab CI/CD
1

Just updated GitLab to 15 last night on a Ubuntu 20 host. Ever since, no pipeline is running any longer and we’ve tried all we can, but nothing is moving.

A simple .gitlab-ci.yml looks like this:

SanityCheck:
  tags:
    - ansible
  variables:
    GIT_STRATEGY: none
  script:
    - ascr sanity check ${CHAT_INPUT} ${ANSIBLE_EXTRA_ARGS}

A runner with the tag ansible is assigned to that project.

When we run the pipeline - either manually or by pushing to the repository - the pipeline gets created but all is greyed out. When opening the job, it says “This job depends on upstream jobs that need to succeed in order for this job to be triggered” but we have no idea what this job should depend upon upstream.

Using the editor or lonter, it shows all is OK and no indication about upstream dependencies. What could be wrong with this? Please help, this is really a big problem for us.

2

Have you tried checking the status of db-migrate

sudo gitlab-rake db:migrate:status

and then performing it manually

sudo gitlab-rake db:migrate

and restarting it?

sudo gitlab-ctl restart

3

Yes, I had tried that and all migrations had been completed successfully. But with gitlab-rake gitlab:check I found out that sidekiq had a problem and it turned out that they introduced the usage of the new port 8092 by a worker which had already been used by another service on our host. I then switched that other service to a different port and now sidekiq is working properly and does its job when it comes to pipelines - and all the other background tasks that had been failing too as it turned out over time.

Edit: a bit more detail can also be found here: https://gitlab.com/gitlab-org/gitlab/-/issues/363180

1 Like