Hello,
i’m experiencing from about one week a lot of problems on my GitLab installation that is on an Amazon EC2 instance with Ubuntu OS.
Starting from last week, my GitLab users were experiencing frequent 502 errors that became more frequent with time. Starting from monday, we started experiencing “ERR_CONNECTION_TIMED_OUT” issue and the only thing that i can do to restart correctly GitLab was to terminate and boot the EC2 instance.
I’ve tried the following to restart gitlab but didn’t work:
- sudo gitlab-ctl restart
- reboot of EC2 instance (instead of first terminating it)
Now the problem is occurring after 1 or two write operations (push, creation of a new project) and it has become impossible to use GitLab.
The strange fact is that at the boot Gitlab works perfectly and after a write operation it becomes unreachable: the EC2 instance becomes unreachable both on 80 and 443 port, while it’s reachable on 22 port (SSH).
Thinking that it was a problem of memory i’ve tried the following but without success:
- “df -h” command: there is a lot of free space (more than 20GB);
- increased memory of EC2 instance from t2.medium to t2.large
I’ve inspected logs and what i’ve found after the machine becomes unreachable is the following (cutting the log because it’s huge):
==> /var/log/gitlab/gitlab-workhorse/current <==
{“command”:[“exiftool”,"-all=","–IPTC:all","–XMP-iptcExt:all","-tagsFromFile","@","-ResolutionUnit","-XResolution","-YResolution","-YCbCrSubSampling","-YCbCrPositioning","-BitsPerSample","-ImageHeight","-ImageWidth","-ImageSize","-Copyright","-CopyrightNotice","-Orientation","-"],“correlation_id”:“uKp2L1bcGp3”,“error”:“exit status 1”,“level”:“info”,“msg”:“exiftool command failed”,“stderr”:" % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 4826 100 4826 0 0 4316 0 0:00:01 0:00:01 --:–:-- 4316\r100 4826 100 4826 0 0 4316 0 0:00:01 0:00:01 --:–:-- 4316\nERROR: You need to be root to run this script\nmodprobe: ERROR: could not insert ‘ip_tables’: Operation not permitted\niptables v1.6.0: can’t initialize iptables table filter': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\nmodprobe: ERROR: could not insert 'ip_tables': Operation not permitted\niptables v1.6.0: can't initialize iptables table filter’: Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\nmodprobe: ERROR: could not insert ‘ip_tables’: Operation not permitted\niptables v1.6.0: can’t initialize iptables table filter': Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\nmodprobe: ERROR: could not insert 'ip_tables': Operation not permitted\niptables v1.6.0: can't initialize iptables table filter’: Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\nchattr: No such file or directory while trying to stat /etc/ld.so.preload\nbash: line 15: /etc/ld.so.preload: Permission denied\nbash: line 24: //dev/fd/i: No such file or directory\nbash: line 24: echo: write error: Invalid argument\nmv: cannot stat ‘/tmp/dlr’: No such file or directory\nmv: cannot stat ‘/var/tmp/dlr’: No such file or directory\nno crontab for git\n/tmp/systemd-private-9f64df23dca548478613372b4de71655-systemd-timesyncd.service-8u9ZM6: no process found\n/var/tmp/cloud-init: no process found\n/var/tmp/systemd-pric36D-nice8f10aeae407ca41750e96b35ccfb-systemd-resolved.service-A4IRDe: no process found\n/var/tmp/systemd-private-8ff328d440b24723ba68d93b71c878d2-systemd-timesyncd.service-plNOh6: no process found\n/var/tmp/systemd-private-9f64df23dca548478613372b4de71655-systemd-timesyncd.service-4dhd8H: no process found\nls: cannot read symbolic link ‘/proc/1/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/1/root’: Permission denied\nls: cannot read symbolic link ‘/proc/1/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/10/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/10/root’: Permission denied\nls: cannot read symbolic link ‘/proc/10/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/1032/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/1032/root’: Permission denied\nls: cannot read symbolic link ‘/proc/1032/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/104/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/104/root’: Permission denied\nls: cannot read symbolic link ‘/proc/104/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/11/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/11/root’: Permission denied\nls: cannot read symbolic link ‘/proc/11/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/1195/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/1195/root’: Permission denied\nls: cannot read symbolic link ‘/proc/1195/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/1196/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/1196/root’: Permission denied\nls: cannot read symbolic link ‘/proc/1196/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/12/cwd’: Permission denied\nls: cannot read symbolic link ‘/proc/12/root’: Permission denied\nls: cannot read symbolic link ‘/proc/12/exe’: Permission denied\nls: cannot read symbolic link ‘/proc/1200/cwd’:
==> /var/log/gitlab/gitlab-workhorse/current <==
{“error”:“keywatcher: pubsub receive: EOF”,“level”:“error”,“msg”:“unknown error”,“time”:“2021-11-18T08:47:52Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-11-18T08:47:52Z”}
{“error”:“keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory”,“level”:“error”,“msg”:“unknown error”,“time”:“2021-11-18T08:47:52Z”}
{“address”:"/var/opt/gitlab/redis/redis.socket",“level”:“info”,“msg”:“redis: dialing”,“network”:“unix”,“time”:“2021-11-18T08:47:52Z”}