Observability::AlertQueryWorker fails and is retried continuously

I have a self managed instance of GitLab.

I noticed recently that the web interface responds slower than before.

While investigating a found an increase in background jobs (from 100.000 to 150.000 daily) and failed jobs (from 0 to 8000 daily). Looking at the retry queue there are only jobs for Observability::AlertQueryWorker but lots of them (several times per minute). They all fail because they try to access external IP addresses which are blocked and not accessible. The exact error message is “Net::OpenTimeout: Failed to open TCP connection”.

This might be related to a recent change.

My GitLab instance was recently updated to v17.4.2-ee.

My question is, what is this job and how can I disable this?

I found two relevant issues.