Hi,
I have deployed a new server with a gitlab instance running in docker version 17.11.4 . I already have an instance with the same version and I’m not facing this issue. I found the issue while checking “ps faux” :
runsvdir -P /opt/gitlab/service log: v sshd: fatal: unable to start log/./run: file does not exist runsv sshd: fatal: unable to start log/./run:
And in the file /opt/gitlab/service/sshd/log/run I have this :
#!/bin/shnexec svlogd -ttt /var/log/gitlab/sshd
On my other instance :
#!/bin/sh
exec svlogd -ttt /var/log/gitlab/sshd
Servers aren’t running the same docker & docker-compose version (29.4 vs 23.0 and 5.1 vs 1.29) and the server with the more recent version is the one with the issue.
I noticed the image are the same version but not the same ID :
- issue : sha256:5d9481078f79f7e103c684a51f37f61232f4a22d3d47ff9a8a652c783c99502b
- working : sha256:44db55a0159689280b597c7a7cec91435a26bbc1f389855e35d0f14c83136c91
Regards
/bin/shnexec looks odd, corrupted shebang. I think it should be /bin/sh <newline> exec ..., so to speak:
#!/bin/sh
exec svlogd ...
Maybe the newline character \n got malformed through backups/deployments on different OS with different carriage-return-line-feed (CRLF) characters.
I have checked all the “run” scripts under /opt/gitlab/sv and it’s the only one with the issue :
./sshd/log/run
./sshd/run
./logrotate/run
./logrotate/log/run
./redis/run
./redis/log/run
./gitaly/run
./gitaly/log/run
./postgresql/run
./postgresql/log/run
./gitlab-kas/run
./gitlab-kas/log/run
./puma/run
./puma/log/run
./sidekiq/run
./sidekiq/log/run
./gitlab-workhorse/run
./gitlab-workhorse/log/run
./nginx/run
./nginx/log/run
./gitlab-pages/run
./gitlab-pages/log/run
./gitlab-exporter/run
./gitlab-exporter/log/run
./redis-exporter/run
./redis-exporter/log/run
./postgres-exporter/run
./postgres-exporter/log/run
Ok, thanks for checking. Is the problem solved after editing the shebang lines to the correct format?
So it’s difficult to say because I have sshd logs now but I still have the error in the “ps faux” (maybe I didn’t run the proper command after fixing it).
What I tried : I edited the file, then ran a gitlab-ctl reconfigure (file is not reverted) but in my “ps faux” I still have the error “sshd: fatal: unable to start log/./run” .
I suppose it would require a full restart but as it’s running as a pod, the file is reverted when I restart it. A fix would be to create a volume just for that script.
I found that error because I wasn’t able to clone over SSH but it was working with HTTPS. After a restart the issue was solved whereas I didn’t modify the config.
I can try a couple of things if you want but I may upgrade the instance next week.
Thanks