Today I upgraded from 15.6.0 to 15.8.1, and jobs no longer run.
Jobs are still created, via push, schedules or API calls, and they appear as “pending” in the various pipelines.
Runners can still connect to the gitlab instance, I can see this in the logs:
==> /var/log/gitlab/nginx/gitlab_access.log <==
10.32.98.26 - - [06/Feb/2023:10:34:43 +0100] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-runner 15.8.1 (15-8-stable; go1.18.9; linux/amd64)" -
==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"","correlation_id":"01GRK17RZTNHB5N333AQG2HSPM","duration_ms":0,"host":"git.ovea.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"178.20.67.132:0","remote_ip":"178.20.67.132","route":"^/api/v4/jobs/request\\z","status":204,"system":"http","time":"2023-02-06T10:34:43+01:00","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 15.6.0 (15-6-stable; go1.18.8; linux/amd64)","written_bytes":0}
I don’t see any errors in the logs, and I am unsure what to check for next.
In the meantime, I rolled back to 15.6.0, and all pending jobs magically started running.