Gitlab-workhorse is missing config.toml

How to Use GitLab Self-managed
1

Hi All,

I’m facing problem with this issue, when see the logs, gitlab-workhorse is missing config.toml
And gitlab web show 502 error.

Thank you.

1 Like
3

Hi Jalunk,

Did you manage to solve this problem? We have the same issue, it just started happening suddenly.

Thank you,
Eloy

4

Same issue:

curl https://git.example.org/ --head

Give:

HTTP/2 502 
server: nginx

On server side:

sudo gitlab-ctl tail gitlab-workhorse

Give:

==> /var/log/gitlab/gitlab-workhorse/current <==
{"build_time":"20201022.044637","level":"info","msg":"Starting","time":"2022-04-23T08:48:06+02:00","version":"v8.39.0"}
{"configFile":"config.toml","error":"open config.toml: no such file or directory","level":"fatal","msg":"Can not load config file","time":"2022-04-23T08:48:06+02:00"}

sudo ls -la /var/opt/gitlab/gitlab-workhorse/

srwxrwxrwx  1 git  git           0 Apr 23 09:48 socket

I have try to upgrade gitlab, but that’s no solve the issue. (so I have rollback the upgrade)

In my case this could be like to this migration:

sudo gitlab-rake gitlab:storage:migrate_to_hashed

witch is a recommendation from:

sudo gitlab-rake gitlab:check

These files are missing:

  • /opt/gitlab/embedded/service/gitlab-shell/config.yml - I think it should be a link to /var/opt/gitlab/gitlab-shell/config.yml
  • but /var/opt/gitlab/gitlab-shell/config.yml is missing.

Same recent issue is describe here: Gitlab-workhorse stopped working: “open config.toml: no such file or directory” - Stack Overflow but no real solution.

5

I’m also having the same issue. A temporary fix for me would be manually creating a config.toml file in the var/opt/gitlab/gitlab-workhorse/ folder. The config file can be found at the omnibus-gitlab repo. However after more than 1 day I see that the config.toml file get deleted so I have to make it not removable using the chattr command. Today I see a new error which is:

{"configFile":"config.toml","error":"Near line 0 (last key parsed ''): bare keys cannot contain '\u003c'","level":"fatal","msg":"Can not load config file","time":"2022-04-25T02:21:47Z"}

Some of my temporary fix may seem to be a case of bad practice but I’m only a junior developer so any criticism would be helpful. Thank you

Update: turn out the config file above was not the correct one (wrong syntax) but somehow it would still make gitlab up and running for a few days. I change the config file and now will wait for any changes. The config file I used is here. Again I would appreciate any insight as I know my solution is not long-term

6

hi all,

still problem and then i’d reinstall gitab using file backup.

thank you.

7

Hi All,

I’m also facing the same issue as described above. This happened suddenly, without any server upgrades. I’m trying to upgrade gitlab but no luck yet. Any solution would be appreciated.

Thank you.

8

An update:

Running gitlab-ctl reconfigure fixed the issue for us without any data loss.

However, just today we had a second issue, where the web interface for gitlab has continued working fine, but git access stopped working. Again gitlab-ctl reconfigure fixed the issue.

This is quite mysterious. We’re migrating to a new server running gitlab with docker, maybe that’s more stable.