dkdlv
November 22, 2022, 8:51am
1
Hi all,
The path of life leaded me to monitor a gitlab instance that is pretty instable and on which build processes are very long (30 minutes or more) and the VM of the worker that is hosting is overloaded, mainly by postgres processes :
The machine has an uptime of 14 days and has processes that are apparently 5000 hours old…
I am searching for clues to understand what is happening (and i am no expert so, we can start with things than can be obvious tp you!)
Thanks in advance!
iwalker
November 22, 2022, 9:28am
2
Build processes? You are running gitlab-runner on the same server as Gitlab? Or can you explain what you mean exactly?
If yes, you are not supposed to run Gitlab Runner on the same server as Gitlab.
Screenshot doesn’t show much information to be honest. Best would be to copy/paste text from the console, for example from this command:
ps aux
To me postgres processes normally show far more:
root@gitlab:~# ps aux | grep -i postgres
gitlab-+ 277142 0.0 0.6 6203316 159328 ? Ss Oct11 2:18 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+ 277164 0.0 0.5 6203520 137328 ? Ss Oct11 0:50 postgres: checkpointer
gitlab-+ 277165 0.0 0.1 6203316 47792 ? Ss Oct11 0:43 postgres: background writer
gitlab-+ 277166 0.0 0.0 6203316 21420 ? Ss Oct11 0:47 postgres: walwriter
gitlab-+ 277167 0.0 0.0 6204016 7836 ? Ss Oct11 1:47 postgres: autovacuum launcher
gitlab-+ 277168 0.0 0.0 18596 5924 ? Ss Oct11 6:20 postgres: stats collector
gitlab-+ 277169 0.0 0.0 6203856 5984 ? Ss Oct11 0:02 postgres: logical replication launcher
root 2869123 0.0 0.0 2188 512 ? Ss Aug30 0:00 runsv postgresql
root 2869140 0.0 0.0 2336 1376 ? S Aug30 0:00 svlogd -tt /var/log/gitlab/postgresql
gitlab-+ 3367604 0.0 0.1 6214772 39596 ? Ss 10:16 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3367614 0.0 0.1 6214900 41016 ? Ss 10:16 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3367616 0.0 0.1 6214740 41092 ? Ss 10:16 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3370632 0.0 0.1 6215148 44044 ? Ss 10:22 0:00 postgres: gitlab gitlabhq_production [local] idle
root 3372926 0.0 0.0 8936 712 pts/0 S+ 10:28 0:00 grep -i postgres
yours say just postgres
. Which leads me to believe these are not postgres processes, but cryptominer processes for your server being potentially compromised.
Please post full output from the command I gave.
dkdlv
November 22, 2022, 9:45am
3
Hi and thank you for your answer.
For what I understood, runners are on separate containers, on the same node.
Here is the result of the ps command.
ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 5864 2020 ? Ss Nov07 0:00 /bin/bash /assets/wrapper
root 23 0.0 0.0 2536 1240 ? S Nov07 0:11 runsvdir -P /opt/gitlab/service log: ..................................................................................................................
root 29 0.0 0.0 2384 584 ? Ss Nov07 0:00 runsv sshd
root 30 0.0 0.0 2528 636 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/sshd
root 31 0.0 0.0 12180 2728 ? S Nov07 0:00 sshd: /usr/sbin/sshd -D -f /assets/sshd_config -e [listener] 0 of 100-200 startups
root 251 0.0 0.0 2384 1304 ? Ss Nov07 0:00 runsv logrotate
root 262 0.0 0.0 2528 568 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/logrotate
root 264 0.0 0.0 2384 580 ? Ss Nov07 0:00 runsv redis
gitlab-+ 266 1.6 0.1 76020 19136 ? Ssl Nov07 346:28 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
root 275 0.0 0.0 2528 1388 ? S Nov07 0:02 svlogd -tt /var/log/gitlab/redis
root 281 0.0 0.0 2384 648 ? Ss Nov07 0:00 runsv gitaly
git 283 0.0 0.0 1394748 8188 ? Ssl Nov07 1:59 /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
git 292 0.3 1.1 2094936 137160 ? Sl Nov07 68:17 /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
root 310 0.0 0.0 2528 1344 ? S Nov07 0:02 svlogd /var/log/gitlab/gitaly
git 327 0.1 0.3 2743648 46768 ? Sl Nov07 25:37 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 292 /var/opt/gitlab/gitaly/internal_sockets/ruby.0
git 328 0.1 0.6 2954672 83164 ? Sl Nov07 25:55 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 292 /var/opt/gitlab/gitaly/internal_sockets/ruby.1
root 406 0.0 0.0 2384 636 ? Ss Nov07 0:00 runsv postgresql
gitlab-+ 408 0.0 0.1 34920 19928 ? Ss Nov07 0:25 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
root 413 0.0 0.0 2528 1600 ? S Nov07 0:03 svlogd -tt /var/log/gitlab/postgresql
gitlab-+ 419 0.0 0.0 35028 6772 ? Ss Nov07 0:09 postgres: checkpointer
gitlab-+ 420 0.0 0.0 34920 5160 ? Ss Nov07 1:39 postgres: background writer
gitlab-+ 421 0.0 0.0 34920 3616 ? Ss Nov07 0:50 postgres: walwriter
gitlab-+ 422 0.0 0.0 35620 7936 ? Ss Nov07 1:24 postgres: autovacuum launcher
gitlab-+ 423 0.0 0.0 18068 4808 ? Ss Nov07 13:38 postgres: stats collector
gitlab-+ 424 0.0 0.0 35460 6160 ? Ss Nov07 0:00 postgres: logical replication launcher
root 500 0.0 0.0 2384 704 ? Ss Nov07 0:00 runsv puma
git 502 0.0 5.0 800272 615424 ? Ssl Nov07 12:37 puma 5.3.2 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]
root 511 0.0 0.0 2528 700 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/puma
root 513 0.0 0.0 2384 700 ? Ss Nov07 0:00 runsv sidekiq
git 515 0.0 0.2 117452 36248 ? Ssl Nov07 0:19 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -m 50 --timeout 25 *
git 520 5.5 8.4 2212124 1045148 ? Sl Nov07 1198:22 sidekiq 6.2.2 queues:authorized_project_update:authorized_project_update_project_create,authorized_project_update:authorized_project_update_project_gr
root 527 0.0 0.0 2528 1440 ? S Nov07 1:02 svlogd /var/log/gitlab/sidekiq
root 529 0.0 0.0 2384 700 ? Ss Nov07 0:00 runsv gitlab-workhorse
root 549 0.0 0.0 2528 1452 ? S Nov07 0:34 svlogd /var/log/gitlab/gitlab-workhorse
root 565 0.0 0.0 2384 636 ? Ss Nov07 0:00 runsv nginx
root 567 0.0 0.0 20116 5980 ? Ss Nov07 0:00 nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx
gitlab-+ 568 0.0 0.0 24748 7916 ? S Nov07 0:00 nginx: worker process
gitlab-+ 569 0.0 0.0 24368 6484 ? S Nov07 0:00 nginx: worker process
gitlab-+ 570 0.0 0.0 25240 8388 ? S Nov07 5:30 nginx: worker process
gitlab-+ 571 0.0 0.0 24368 6484 ? S Nov07 0:00 nginx: worker process
gitlab-+ 572 0.0 0.0 24764 7808 ? S Nov07 0:02 nginx: worker process
gitlab-+ 573 0.0 0.0 24368 6484 ? S Nov07 0:00 nginx: worker process
gitlab-+ 574 0.0 0.0 24368 6484 ? S Nov07 0:00 nginx: worker process
gitlab-+ 575 0.0 0.0 24368 6484 ? S Nov07 0:00 nginx: worker process
gitlab-+ 576 0.0 0.0 20332 2368 ? S Nov07 0:04 nginx: cache manager process
root 588 0.0 0.0 2528 696 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/nginx
root 594 0.0 0.0 2384 692 ? Ss Nov07 0:00 runsv gitlab-exporter
root 604 0.0 0.0 2528 1492 ? S Nov07 0:17 svlogd -tt /var/log/gitlab/gitlab-exporter
root 610 0.0 0.0 2384 640 ? Ss Nov07 0:00 runsv redis-exporter
root 802 0.0 0.0 2528 700 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/redis-exporter
root 819 0.0 0.0 2384 1256 ? Ss Nov07 0:47 runsv prometheus
root 844 0.0 0.0 2528 1484 ? S Nov07 0:29 svlogd -tt /var/log/gitlab/prometheus
root 853 0.0 0.0 2384 636 ? Ss Nov07 0:00 runsv alertmanager
root 871 0.0 0.0 2528 632 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/alertmanager
root 884 0.0 0.0 2384 632 ? Ss Nov07 0:00 runsv postgres-exporter
root 901 0.0 0.0 2528 632 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/postgres-exporter
root 908 0.0 0.0 2384 636 ? Ss Nov07 0:00 runsv grafana
root 931 0.0 0.0 2528 1440 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/grafana
git 972 0.1 0.3 1862256 46224 ? Ssl Nov07 38:07 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sockets/socket -authBackend h
git 987 0.7 0.5 256804 66492 ? Ssl Nov07 151:27 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/gitlab-exporter web -c /var/opt/gitlab/gitlab-exporter/gitlab-exporter.yml
gitlab-+ 989 0.0 0.2 1626336 27220 ? Ssl Nov07 15:21 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket
gitlab-+ 1014 0.0 0.1 1713556 20232 ? Ssl Nov07 16:22 /opt/gitlab/embedded/bin/alertmanager --web.listen-address=localhost:9093 --storage.path=/var/opt/gitlab/alertmanager/data --config.file=/var/opt/gitla
gitlab-+ 1025 0.2 0.2 1628492 25484 ? Ssl Nov07 47:55 /opt/gitlab/embedded/bin/postgres_exporter --web.listen-address=localhost:9187 --extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
gitlab-+ 1033 0.0 0.3 1828760 43364 ? Ssl Nov07 5:36 /opt/gitlab/embedded/bin/grafana-server -config /var/opt/gitlab/grafana/grafana.ini
gitlab-+ 1074 0.6 0.1 43340 22688 ? Ss Nov07 150:44 postgres: gitlab gitlabhq_production [local] idle
root 1094 0.0 0.0 5732 2148 ? S Nov07 0:00 /bin/bash /opt/gitlab/bin/gitlab-ctl tail
root 1095 0.0 0.3 95780 41220 ? S Nov07 0:00 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/omnibus-ctl gitlab /opt/gitlab/embedded/service/omnibus-ctl* tail
root 1103 0.0 0.0 2616 604 ? S Nov07 0:00 sh -c find -L /var/log/gitlab -type f -not -path '*/sasl/*' | grep -E -v '(config|lock|@|gzip|tgz|gz)' | xargs tail --follow=name --retry
root 1106 0.0 0.0 4564 484 ? S Nov07 0:00 xargs tail --follow=name --retry
root 1107 0.0 0.0 4304 500 ? S Nov07 2:37 tail --follow=name --retry /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/a
gitlab-+ 1108 0.5 0.3 66712 45724 ? Ss Nov07 122:32 postgres: gitlab-psql gitlabhq_production [local] idle
gitlab-+ 9206 58.0 0.2 48736 29192 ? Ss Nov15 5892:14 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 13332 58.5 0.2 47508 27308 ? Ss Nov15 5916:13 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 14100 57.9 0.2 48864 29088 ? Rs Nov15 5851:23 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16118 58.1 0.2 48120 27924 ? Ss Nov15 5865:12 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16122 57.5 0.2 47136 25960 ? Rs Nov15 5804:29 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16123 58.7 0.2 47544 26372 ? Rs Nov15 5927:09 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16124 58.0 0.2 48060 27180 ? Ss Nov15 5862:04 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 19439 0.8 2.1 3815544 269436 ? Ssl Nov16 77:23 /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitl
git 28146 0.4 6.4 1130452 791460 ? Sl 09:12 0:06 puma: cluster worker 6: 502 [gitlab-puma-worker]
gitlab-+ 28189 0.0 0.1 43904 22664 ? Ss 09:12 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git 28254 0.5 6.5 1165780 801800 ? Sl 09:13 0:06 puma: cluster worker 5: 502 [gitlab-puma-worker]
git 28317 0.4 6.4 1139156 788960 ? Sl 09:13 0:05 puma: cluster worker 1: 502 [gitlab-puma-worker]
gitlab-+ 28384 0.0 0.2 46580 25088 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28409 0.0 0.2 47064 27664 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git 28432 0.4 6.3 1116628 776724 ? Sl 09:14 0:05 puma: cluster worker 4: 502 [gitlab-puma-worker]
gitlab-+ 28461 0.0 0.1 41504 21452 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28488 0.0 0.1 41488 22536 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git 28575 0.6 6.3 1152468 786720 ? Sl 09:15 0:06 puma: cluster worker 7: 502 [gitlab-puma-worker]
git 28658 0.4 6.3 1137740 787048 ? Sl 09:16 0:05 puma: cluster worker 2: 502 [gitlab-puma-worker]
git 28770 0.6 6.4 1138136 791164 ? Sl 09:17 0:06 puma: cluster worker 0: 502 [gitlab-puma-worker]
gitlab-+ 28810 0.0 0.2 47060 26400 ? Ss 09:17 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git 28888 0.7 6.4 1130964 789532 ? Sl 09:18 0:06 puma: cluster worker 3: 502 [gitlab-puma-worker]
gitlab-+ 28918 0.0 0.2 48012 26812 ? Ss 09:18 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 29310 0.0 0.1 43680 23776 ? Ss 09:21 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 29985 0.0 0.1 41404 20556 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30076 0.0 0.1 44760 22308 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30077 0.0 0.1 41416 19592 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30078 0.0 0.1 41412 20016 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
root 30274 0.0 0.0 2612 596 ? Ss 09:27 0:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
root 30282 0.0 0.0 2512 520 ? S 09:27 0:00 sleep 600
root 30905 0.0 0.0 7656 3196 pts/1 R+ 09:33 0:00 ps aux
root 31537 0.0 0.0 5996 3812 pts/1 Ss Nov21 0:00 /bin/bash
gitlab-+ 32035 58.7 0.2 47180 26432 ? Rs Nov15 5909:46 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32329 57.9 0.1 42180 22884 ? Rs Nov15 5825:28 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32725 58.6 0.1 40924 19788 ? Rs Nov15 5902:59 postgres: gitlab gitlabhq_production [local] UPDATE
And more specifically:
ps aux | grep -i postgres
root 406 0.0 0.0 2384 636 ? Ss Nov07 0:00 runsv postgresql
gitlab-+ 408 0.0 0.1 34920 19928 ? Ss Nov07 0:25 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
root 413 0.0 0.0 2528 1600 ? S Nov07 0:03 svlogd -tt /var/log/gitlab/postgresql
gitlab-+ 419 0.0 0.0 35028 6772 ? Ss Nov07 0:09 postgres: checkpointer
gitlab-+ 420 0.0 0.0 34920 5160 ? Ss Nov07 1:39 postgres: background writer
gitlab-+ 421 0.0 0.0 34920 3616 ? Ss Nov07 0:50 postgres: walwriter
gitlab-+ 422 0.0 0.0 35620 7936 ? Ss Nov07 1:24 postgres: autovacuum launcher
gitlab-+ 423 0.0 0.0 18068 4808 ? Ss Nov07 13:39 postgres: stats collector
gitlab-+ 424 0.0 0.0 35460 6160 ? Ss Nov07 0:00 postgres: logical replication launcher
root 884 0.0 0.0 2384 632 ? Ss Nov07 0:00 runsv postgres-exporter
root 901 0.0 0.0 2528 632 ? S Nov07 0:00 svlogd -tt /var/log/gitlab/postgres-exporter
gitlab-+ 1025 0.2 0.1 1628492 24168 ? Ssl Nov07 47:55 /opt/gitlab/embedded/bin/postgres_exporter --web.listen-address=localhost:9187 --extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
gitlab-+ 1074 0.6 0.1 43340 22688 ? Ss Nov07 150:49 postgres: gitlab gitlabhq_production [local] idle
root 1107 0.0 0.0 4304 500 ? S Nov07 2:37 tail --follow=name --retry /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/auth.log /var/log/gitlab/gitlab-rails/grpc.log /var/log/gitlab/gitlab-rails/api_json.log /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/gitlab-rails/service_measurement.log /var/log/gitlab/gitlab-rails/application_json.log /var/log/gitlab/gitlab-rails/sidekiq_client.log /var/log/gitlab/alertmanager/current /var/log/gitlab/sidekiq/current /var/log/gitlab/postgres-exporter/current /var/log/gitlab/prometheus/current /var/log/gitlab/gitlab-exporter/current /var/log/gitlab/grafana/current /var/log/gitlab/redis/current /var/log/gitlab/gitlab-workhorse/current /var/log/gitlab/redis-exporter/current /var/log/gitlab/sshd/current /var/log/gitlab/postgresql/current /var/log/gitlab/nginx/gitlab_access.log /var/log/gitlab/nginx/gitlab_error.log /var/log/gitlab/nginx/error.log /var/log/gitlab/nginx/access.log /var/log/gitlab/nginx/current /var/log/gitlab/logrotate/current /var/log/gitlab/gitaly/current /var/log/gitlab/gitaly/gitaly_ruby_json.log /var/log/gitlab/puma/puma_stdout.log /var/log/gitlab/puma/current /var/log/gitlab/puma/puma_stderr.log
gitlab-+ 1108 0.5 0.3 66712 45724 ? Ss Nov07 122:35 postgres: gitlab-psql gitlabhq_production [local] idle
gitlab-+ 9206 58.0 0.2 48736 29192 ? Rs Nov15 5899:21 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 13332 58.5 0.2 47508 27308 ? Ss Nov15 5923:20 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 14100 57.9 0.2 48864 29088 ? Rs Nov15 5858:23 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16118 58.1 0.2 48120 27924 ? Rs Nov15 5872:15 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16122 57.5 0.2 47136 25960 ? Rs Nov15 5811:28 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16123 58.7 0.2 47544 26372 ? Rs Nov15 5934:15 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16124 58.0 0.2 48060 27180 ? Rs Nov15 5869:05 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 28189 0.0 0.1 44160 24256 ? Ss 09:12 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28384 0.0 0.2 46580 25524 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28409 0.0 0.2 47164 27776 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28461 0.0 0.2 48300 27100 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28488 0.0 0.2 48752 28896 ? Ss 09:14 0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28810 0.0 0.2 47220 27168 ? Ss 09:17 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28918 0.0 0.2 46244 25420 ? Ss 09:18 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 29310 0.0 0.2 45860 25228 ? Ss 09:21 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 29985 0.0 0.2 46796 25748 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30076 0.0 0.1 44032 22040 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30078 0.0 0.1 44604 22068 ? Ss 09:25 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32035 58.7 0.2 47180 26432 ? Rs Nov15 5916:54 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32103 0.0 0.1 42696 21776 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32104 0.0 0.1 44100 21912 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32193 0.0 0.1 41420 19860 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32194 0.0 0.1 40536 18924 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32195 0.0 0.1 42228 20940 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32196 0.0 0.1 39208 17792 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32197 0.0 0.1 42260 20612 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32219 0.0 0.1 39476 18120 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32220 0.0 0.1 39136 17696 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32221 0.0 0.1 39504 18548 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32222 0.0 0.1 39124 17812 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32223 0.0 0.1 39460 18556 ? Ss 09:45 0:00 postgres: gitlab gitlabhq_production [local] idle
root 32309 0.0 0.0 5196 2344 pts/0 S+ 09:46 0:00 grep --color=auto -i postgres
gitlab-+ 32329 57.9 0.1 42180 22884 ? Ss Nov15 5832:29 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32725 58.6 0.1 40924 19788 ? Rs Nov15 5910:07 postgres: gitlab gitlabhq_production [local] UPDATE
Again, thank you for your time
iwalker
November 22, 2022, 10:56am
4
And that’s the problem. As per the Gitlab documentation: GitLab Runner | GitLab
So, Gitlab on it’s own needs 4cpu and 8gb ram. Your top/htop shows your server has 12GB, so means only 4GB for Runners. You mentioned you have multiple runners also running.
My suggestion is you remove all the runners from this server, and put them on different VM’s. Also for performance, each runner should have their own VM. You should not be attempting to run multiple runners on the same machine. All you are doing is sharing resources between all runners, and everything will take longer to complete that way. Alternatively, create a single server for multiple container runners and ensure the server has enough cpu and ram for all of them. Eg: assuming 4cpu and 8gb of ram for each runner, and you have say 4 runners, that is 16cpu and 32gb ram.
But definitely, you should never have them on the Gitlab server, and this is why you have a problem.
iwalker
November 22, 2022, 11:00am
5
Also, once you have removed the runners from the Gitlab server, I suggest following the Gitlab upgrade docs, as 15.5.4 is the latest version. You don’t want to be running older versions due to security vulnerabilities. Upgrading GitLab | GitLab
According to the upgrade path, you need to upgrade through multiple versions:
the Gitlab docs will have all the commands you need to ensure you upgrade successfully. But only do that once you have removed all your runners from the Gitlab server.
dkdlv
November 22, 2022, 4:22pm
6
Hmm, verification taken, the gitlab server is on one node/VM and the runners are on other nodes/VM
XXX-gitlab-runner-fb77755cb-f6b8h 1/1 Running 1 57d 10.42.2.71 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-86b964977f-ssqmm 1/1 Running 3 295d 10.42.2.67 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-79ff9454b8-z54fb 1/1 Running 1 110d 10.42.2.59 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-6bd57f785c-xmnmr 1/1 Terminating 1 216d 10.42.2.63 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-dfb7f7c86-rxpm2 1/1 Running 1 117d 10.42.0.7 X.Y.Z.70 <none> <none>
XXX-gitlab-runner-5cdd4ddfc-dfkdv 1/1 Running 1 91d 10.42.2.60 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-566cdd9cb8-k6jmf 1/1 Running 5 315d 10.42.0.2 X.Y.Z.70 <none> <none>
XXX-gitlab-runner-7bf9d75b54-xbpvk 1/1 Running 3 309d 10.42.4.30 X.Y.Z.74 <none> <none>
XXX-gitlab-runner-5bbb58985-8gwt9 1/1 Running 0 17m 10.42.3.111 X.Y.Z.73 <none> <none>
XXX-gitlab-runner-d85bc669f-7b6xh 1/1 Running 1 83d 10.42.0.8 X.Y.Z.70 <none> <none>
XXX-gitlab-runner-cd8cf54c6-jwd25 1/1 Running 0 6h14m 10.42.3.109 X.Y.Z.73 <none> <none>
XXX-gitlab-6df5f7f9c8-wbbzx 1/1 Running 4 319d 10.42.1.59 X.Y.Z.71 <none> <none>
XXX-gitlab-runner-697fd4c97b-ttjv8 1/1 Running 1 116d 10.42.2.57 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-68d5f5c48-cf4ws 1/1 Running 3 313d 10.42.4.32 X.Y.Z.74 <none> <none>
XXX-gitlab-runner-57f6b9b5c9-hjjnn 1/1 Running 3 276d 10.42.2.70 X.Y.Z.72 <none> <none>
XXX-gitlab-runner-557ffff968-8cmls 1/1 Running 0 31h 10.42.2.84 X.Y.Z.72 <none> <none>
Only the VM that host gitlab’s node is overloaded.
Interesting fact that may be related: a runner crashed a few minutes ago, and that seemed to give a little breath to the gitlab server but I can’t understand the reason
The upgrade is planned, thanks for the advice!
dkdlv
December 1, 2022, 9:45am
7
A little news from my issue:
We had another current breakdown earlier this week, and the CPU graph is so :
I am not sure this is a normal behaviour…