Gitlab-workhorse stopped working: "open config.toml: no such file or directory"

Hi,

We have a gitlab 8.20.0 server that has been running for a long time without issues.

Today suddenly gitlab-workhorse stopped working (all other services are fine):

gitlab-ctl tail gitlab-workhorse

....
{"build_time":"20200131.231313","level":"info","msg":"Starting","time":"2022-04-21T15:19:38+01:00","version":"v8.20.0"}
{"configFile":"config.toml","error":"open config.toml: no such file or directory","level":"fatal","msg":"Can not load config file","time":"2022-04-21T15:19:38+01:00"}

I can’t find any information on why this would happen, and it seems config.toml isn’t even part of the configuration of gitlab-workhorse.

Does anyone have any idea on how to solve this?

Thank you,
Eloy

2 Likes

I’m also having the same issue. A temporary fix for me would be manually creating a config.toml file in the var/opt/gitlab/gitlab-workhorse/ folder. The config file can be found at the omnibus-gitlab repo. However after more than 1 day I see that the config.toml file get deleted so I have to make it not removable using the chattr command. Today I see a new error which is:

{"configFile":"config.toml","error":"Near line 0 (last key parsed ''): bare keys cannot contain '\u003c'","level":"fatal","msg":"Can not load config file","time":"2022-04-25T02:21:47Z"}

Some of my temporary fix may seem to be a case of bad practice but I’m only a junior developer
so any criticism would be helpful. Thank you

We are having the same issue with gitlab 12.2 and cannot find any fix.

Also upgrading fails.

Hi All,

I’m also facing the same issue as described above. This happened suddenly, without any server upgrades. I’m trying to upgrade gitlab but no luck yet. Any solution would be appreciated.

Thank you.

An update:

Running gitlab-ctl reconfigure fixed the issue for us without any data loss.

However, just today we had a second issue, where the web interface for gitlab has continued working fine, but git access stopped working. Again gitlab-ctl reconfigure fixed the issue.

This is quite mysterious. We’re migrating to a new server running gitlab with docker, maybe that’s more stable.

Hello Everyone.

We experienced the same problem last week and we still didn’t resolve the issue.
Running Gitlab on a dedicated Linux server in an LXC container. I worked perfectly for more than 2 years.

I noticed that the issue is with GitLab workhorse, specifically with config.toml file which gets removed by the system automatically.

Running gitlab-ctl reconfigure helps, but only for a minute - after that GitLab page returns 502.

Any help would be much appreciated.

Klemen

Hi All,

I thought I’ll just give an update. Yesterday I’ve upgraded gitlab to version 14.6.2 and it worked fine for few hours after installing the upgrade. However, overnight the Linux VM has frozen. In the morning I had to restart the server and then I got 502 error again on gitlab after startup. When I run the reconfigure command it helps but only for a little while. Later gitlab will crush again with 502 error. I’m not sure if the config.toml error is causing the server to crush but it happened twice now in our case.
So the issue is still ongoing. Really not sure what the problem might be. Gitlab is running on Ubuntu 18.04.6 bionic.

Any help will be appreciated.

Following @eloy.villasclaras suggestions we got it to work without any data loss. However the problem seems to reappear a day later. Any new inputs are greatly appreciated