Sidekiq won't start due to write error to sidekiq_exporter.log

Hello,

I’m in the process of a long overdue update and my first step in the series of updates is to upgrade to v12.10.14. Now the 11 backbround_migration tasks aren’t running and it is recommended that they complete before doing the next upgrade. How do I get them to run?

running sudo gitlab-rake gitlab:check shows this:
Sidekiq: … Running? … no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section “Install Init Script”
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.

Running the recommended fix returns this:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
/opt/gitlab/embedded/service/gitlab-rails/bin/background_jobs_sk: line 27: /opt/gitlab/embedded/service/gitlab-rails/log/sidekiq.log: Permission denied

I’m seeing this in the sidekiq.log file.
{ “severity”:“WARN”,“time”:“2023-073T16:36:02.857Z”,**
“error_class”:“Errno::EIO”,**
“error_message”:“Input/output error @ io_write - /opt/gitlab/embedded/service/gitlab-rails/log/sidekiq_exporter.log”,“context”:“Exception during Sidekiq lifecycle event.”,**
“event”:“startup”,**
“error_backtrace”:[“lib/gitlab/daemon.rb:41:in `block in **
start’”,“lib/gitlab/daemon.rb:38:in `synchronize’”,“lib/gitlab/daemon.rb:38:in `start’”]**
}

Input/output error @ io_write - /opt/gitlab/embedded/service/gitlab-rails/log/sidekiq_exporter.log**

It looks like Sidekiq is trying to start, but then aborts due to being unable to write to the sidekiq_exporter.log file.

I’ve gone and updated the permissions on the sidekiq_exporter.log file to give everyone write permissions, but still get the write error.
sudo ls -l log/sidekiq_exporter.log returns
-rw-rw-rw- 1 git git 20540 Jul 13 12:35 log/sidekiq_exporter.log

What can I do to get Sidekiq to get started and run those background_migration jobs?

Thanks,
Bruce

OK, I feel sort of silly for not thinking of this sooner, but I found my issue. Turned out to be a corrupt sidekiq_exporter.log file. I deleted it and now background jobs are running again.

2 Likes

Thanks for sharing the solution @BruceH. Glad you got this figured out.