CPU load on gitlab CE 14.5.2 (postgres processes)

Hi all,

The path of life leaded me to monitor a gitlab instance that is pretty instable and on which build processes are very long (30 minutes or more) and the VM of the worker that is hosting is overloaded, mainly by postgres processes :
image

The machine has an uptime of 14 days and has processes that are apparently 5000 hours old…

I am searching for clues to understand what is happening (and i am no expert so, we can start with things than can be obvious tp you!)

Thanks in advance!

Build processes? You are running gitlab-runner on the same server as Gitlab? Or can you explain what you mean exactly?

If yes, you are not supposed to run Gitlab Runner on the same server as Gitlab.

Screenshot doesn’t show much information to be honest. Best would be to copy/paste text from the console, for example from this command:

ps aux

To me postgres processes normally show far more:

root@gitlab:~# ps aux | grep -i postgres
gitlab-+  277142  0.0  0.6 6203316 159328 ?      Ss   Oct11   2:18 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+  277164  0.0  0.5 6203520 137328 ?      Ss   Oct11   0:50 postgres: checkpointer 
gitlab-+  277165  0.0  0.1 6203316 47792 ?       Ss   Oct11   0:43 postgres: background writer 
gitlab-+  277166  0.0  0.0 6203316 21420 ?       Ss   Oct11   0:47 postgres: walwriter 
gitlab-+  277167  0.0  0.0 6204016 7836 ?        Ss   Oct11   1:47 postgres: autovacuum launcher 
gitlab-+  277168  0.0  0.0  18596  5924 ?        Ss   Oct11   6:20 postgres: stats collector 
gitlab-+  277169  0.0  0.0 6203856 5984 ?        Ss   Oct11   0:02 postgres: logical replication launcher 
root     2869123  0.0  0.0   2188   512 ?        Ss   Aug30   0:00 runsv postgresql
root     2869140  0.0  0.0   2336  1376 ?        S    Aug30   0:00 svlogd -tt /var/log/gitlab/postgresql
gitlab-+ 3367604  0.0  0.1 6214772 39596 ?       Ss   10:16   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3367614  0.0  0.1 6214900 41016 ?       Ss   10:16   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3367616  0.0  0.1 6214740 41092 ?       Ss   10:16   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 3370632  0.0  0.1 6215148 44044 ?       Ss   10:22   0:00 postgres: gitlab gitlabhq_production [local] idle
root     3372926  0.0  0.0   8936   712 pts/0    S+   10:28   0:00 grep -i postgres

yours say just postgres. Which leads me to believe these are not postgres processes, but cryptominer processes for your server being potentially compromised.

Please post full output from the command I gave.

Hi and thank you for your answer.

For what I understood, runners are on separate containers, on the same node.

Here is the result of the ps command.

ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   5864  2020 ?        Ss   Nov07   0:00 /bin/bash /assets/wrapper
root        23  0.0  0.0   2536  1240 ?        S    Nov07   0:11 runsvdir -P /opt/gitlab/service log: ..................................................................................................................
root        29  0.0  0.0   2384   584 ?        Ss   Nov07   0:00 runsv sshd
root        30  0.0  0.0   2528   636 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/sshd
root        31  0.0  0.0  12180  2728 ?        S    Nov07   0:00 sshd: /usr/sbin/sshd -D -f /assets/sshd_config -e [listener] 0 of 100-200 startups
root       251  0.0  0.0   2384  1304 ?        Ss   Nov07   0:00 runsv logrotate
root       262  0.0  0.0   2528   568 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/logrotate
root       264  0.0  0.0   2384   580 ?        Ss   Nov07   0:00 runsv redis
gitlab-+   266  1.6  0.1  76020 19136 ?        Ssl  Nov07 346:28 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
root       275  0.0  0.0   2528  1388 ?        S    Nov07   0:02 svlogd -tt /var/log/gitlab/redis
root       281  0.0  0.0   2384   648 ?        Ss   Nov07   0:00 runsv gitaly
git        283  0.0  0.0 1394748 8188 ?        Ssl  Nov07   1:59 /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
git        292  0.3  1.1 2094936 137160 ?      Sl   Nov07  68:17 /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
root       310  0.0  0.0   2528  1344 ?        S    Nov07   0:02 svlogd /var/log/gitlab/gitaly
git        327  0.1  0.3 2743648 46768 ?       Sl   Nov07  25:37 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 292 /var/opt/gitlab/gitaly/internal_sockets/ruby.0
git        328  0.1  0.6 2954672 83164 ?       Sl   Nov07  25:55 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 292 /var/opt/gitlab/gitaly/internal_sockets/ruby.1
root       406  0.0  0.0   2384   636 ?        Ss   Nov07   0:00 runsv postgresql
gitlab-+   408  0.0  0.1  34920 19928 ?        Ss   Nov07   0:25 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
root       413  0.0  0.0   2528  1600 ?        S    Nov07   0:03 svlogd -tt /var/log/gitlab/postgresql
gitlab-+   419  0.0  0.0  35028  6772 ?        Ss   Nov07   0:09 postgres: checkpointer
gitlab-+   420  0.0  0.0  34920  5160 ?        Ss   Nov07   1:39 postgres: background writer
gitlab-+   421  0.0  0.0  34920  3616 ?        Ss   Nov07   0:50 postgres: walwriter
gitlab-+   422  0.0  0.0  35620  7936 ?        Ss   Nov07   1:24 postgres: autovacuum launcher
gitlab-+   423  0.0  0.0  18068  4808 ?        Ss   Nov07  13:38 postgres: stats collector
gitlab-+   424  0.0  0.0  35460  6160 ?        Ss   Nov07   0:00 postgres: logical replication launcher
root       500  0.0  0.0   2384   704 ?        Ss   Nov07   0:00 runsv puma
git        502  0.0  5.0 800272 615424 ?       Ssl  Nov07  12:37 puma 5.3.2 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]
root       511  0.0  0.0   2528   700 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/puma
root       513  0.0  0.0   2384   700 ?        Ss   Nov07   0:00 runsv sidekiq
git        515  0.0  0.2 117452 36248 ?        Ssl  Nov07   0:19 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -m 50 --timeout 25 *
git        520  5.5  8.4 2212124 1045148 ?     Sl   Nov07 1198:22 sidekiq 6.2.2 queues:authorized_project_update:authorized_project_update_project_create,authorized_project_update:authorized_project_update_project_gr
root       527  0.0  0.0   2528  1440 ?        S    Nov07   1:02 svlogd /var/log/gitlab/sidekiq
root       529  0.0  0.0   2384   700 ?        Ss   Nov07   0:00 runsv gitlab-workhorse
root       549  0.0  0.0   2528  1452 ?        S    Nov07   0:34 svlogd /var/log/gitlab/gitlab-workhorse
root       565  0.0  0.0   2384   636 ?        Ss   Nov07   0:00 runsv nginx
root       567  0.0  0.0  20116  5980 ?        Ss   Nov07   0:00 nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx
gitlab-+   568  0.0  0.0  24748  7916 ?        S    Nov07   0:00 nginx: worker process
gitlab-+   569  0.0  0.0  24368  6484 ?        S    Nov07   0:00 nginx: worker process
gitlab-+   570  0.0  0.0  25240  8388 ?        S    Nov07   5:30 nginx: worker process
gitlab-+   571  0.0  0.0  24368  6484 ?        S    Nov07   0:00 nginx: worker process
gitlab-+   572  0.0  0.0  24764  7808 ?        S    Nov07   0:02 nginx: worker process
gitlab-+   573  0.0  0.0  24368  6484 ?        S    Nov07   0:00 nginx: worker process
gitlab-+   574  0.0  0.0  24368  6484 ?        S    Nov07   0:00 nginx: worker process
gitlab-+   575  0.0  0.0  24368  6484 ?        S    Nov07   0:00 nginx: worker process
gitlab-+   576  0.0  0.0  20332  2368 ?        S    Nov07   0:04 nginx: cache manager process
root       588  0.0  0.0   2528   696 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/nginx
root       594  0.0  0.0   2384   692 ?        Ss   Nov07   0:00 runsv gitlab-exporter
root       604  0.0  0.0   2528  1492 ?        S    Nov07   0:17 svlogd -tt /var/log/gitlab/gitlab-exporter
root       610  0.0  0.0   2384   640 ?        Ss   Nov07   0:00 runsv redis-exporter
root       802  0.0  0.0   2528   700 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/redis-exporter
root       819  0.0  0.0   2384  1256 ?        Ss   Nov07   0:47 runsv prometheus
root       844  0.0  0.0   2528  1484 ?        S    Nov07   0:29 svlogd -tt /var/log/gitlab/prometheus
root       853  0.0  0.0   2384   636 ?        Ss   Nov07   0:00 runsv alertmanager
root       871  0.0  0.0   2528   632 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/alertmanager
root       884  0.0  0.0   2384   632 ?        Ss   Nov07   0:00 runsv postgres-exporter
root       901  0.0  0.0   2528   632 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/postgres-exporter
root       908  0.0  0.0   2384   636 ?        Ss   Nov07   0:00 runsv grafana
root       931  0.0  0.0   2528  1440 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/grafana
git        972  0.1  0.3 1862256 46224 ?       Ssl  Nov07  38:07 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sockets/socket -authBackend h
git        987  0.7  0.5 256804 66492 ?        Ssl  Nov07 151:27 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/gitlab-exporter web -c /var/opt/gitlab/gitlab-exporter/gitlab-exporter.yml
gitlab-+   989  0.0  0.2 1626336 27220 ?       Ssl  Nov07  15:21 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket
gitlab-+  1014  0.0  0.1 1713556 20232 ?       Ssl  Nov07  16:22 /opt/gitlab/embedded/bin/alertmanager --web.listen-address=localhost:9093 --storage.path=/var/opt/gitlab/alertmanager/data --config.file=/var/opt/gitla
gitlab-+  1025  0.2  0.2 1628492 25484 ?       Ssl  Nov07  47:55 /opt/gitlab/embedded/bin/postgres_exporter --web.listen-address=localhost:9187 --extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
gitlab-+  1033  0.0  0.3 1828760 43364 ?       Ssl  Nov07   5:36 /opt/gitlab/embedded/bin/grafana-server -config /var/opt/gitlab/grafana/grafana.ini
gitlab-+  1074  0.6  0.1  43340 22688 ?        Ss   Nov07 150:44 postgres: gitlab gitlabhq_production [local] idle
root      1094  0.0  0.0   5732  2148 ?        S    Nov07   0:00 /bin/bash /opt/gitlab/bin/gitlab-ctl tail
root      1095  0.0  0.3  95780 41220 ?        S    Nov07   0:00 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/omnibus-ctl gitlab /opt/gitlab/embedded/service/omnibus-ctl* tail
root      1103  0.0  0.0   2616   604 ?        S    Nov07   0:00 sh -c find -L /var/log/gitlab -type f -not -path '*/sasl/*' | grep -E -v '(config|lock|@|gzip|tgz|gz)' | xargs tail --follow=name --retry
root      1106  0.0  0.0   4564   484 ?        S    Nov07   0:00 xargs tail --follow=name --retry
root      1107  0.0  0.0   4304   500 ?        S    Nov07   2:37 tail --follow=name --retry /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/a
gitlab-+  1108  0.5  0.3  66712 45724 ?        Ss   Nov07 122:32 postgres: gitlab-psql gitlabhq_production [local] idle
gitlab-+  9206 58.0  0.2  48736 29192 ?        Ss   Nov15 5892:14 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 13332 58.5  0.2  47508 27308 ?        Ss   Nov15 5916:13 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 14100 57.9  0.2  48864 29088 ?        Rs   Nov15 5851:23 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16118 58.1  0.2  48120 27924 ?        Ss   Nov15 5865:12 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16122 57.5  0.2  47136 25960 ?        Rs   Nov15 5804:29 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16123 58.7  0.2  47544 26372 ?        Rs   Nov15 5927:09 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16124 58.0  0.2  48060 27180 ?        Ss   Nov15 5862:04 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 19439  0.8  2.1 3815544 269436 ?      Ssl  Nov16  77:23 /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitl
git      28146  0.4  6.4 1130452 791460 ?      Sl   09:12   0:06 puma: cluster worker 6: 502 [gitlab-puma-worker]
gitlab-+ 28189  0.0  0.1  43904 22664 ?        Ss   09:12   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git      28254  0.5  6.5 1165780 801800 ?      Sl   09:13   0:06 puma: cluster worker 5: 502 [gitlab-puma-worker]
git      28317  0.4  6.4 1139156 788960 ?      Sl   09:13   0:05 puma: cluster worker 1: 502 [gitlab-puma-worker]
gitlab-+ 28384  0.0  0.2  46580 25088 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28409  0.0  0.2  47064 27664 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git      28432  0.4  6.3 1116628 776724 ?      Sl   09:14   0:05 puma: cluster worker 4: 502 [gitlab-puma-worker]
gitlab-+ 28461  0.0  0.1  41504 21452 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28488  0.0  0.1  41488 22536 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git      28575  0.6  6.3 1152468 786720 ?      Sl   09:15   0:06 puma: cluster worker 7: 502 [gitlab-puma-worker]
git      28658  0.4  6.3 1137740 787048 ?      Sl   09:16   0:05 puma: cluster worker 2: 502 [gitlab-puma-worker]
git      28770  0.6  6.4 1138136 791164 ?      Sl   09:17   0:06 puma: cluster worker 0: 502 [gitlab-puma-worker]
gitlab-+ 28810  0.0  0.2  47060 26400 ?        Ss   09:17   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
git      28888  0.7  6.4 1130964 789532 ?      Sl   09:18   0:06 puma: cluster worker 3: 502 [gitlab-puma-worker]
gitlab-+ 28918  0.0  0.2  48012 26812 ?        Ss   09:18   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 29310  0.0  0.1  43680 23776 ?        Ss   09:21   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 29985  0.0  0.1  41404 20556 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30076  0.0  0.1  44760 22308 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30077  0.0  0.1  41416 19592 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30078  0.0  0.1  41412 20016 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
root     30274  0.0  0.0   2612   596 ?        Ss   09:27   0:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
root     30282  0.0  0.0   2512   520 ?        S    09:27   0:00 sleep 600
root     30905  0.0  0.0   7656  3196 pts/1    R+   09:33   0:00 ps aux
root     31537  0.0  0.0   5996  3812 pts/1    Ss   Nov21   0:00 /bin/bash
gitlab-+ 32035 58.7  0.2  47180 26432 ?        Rs   Nov15 5909:46 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32329 57.9  0.1  42180 22884 ?        Rs   Nov15 5825:28 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32725 58.6  0.1  40924 19788 ?        Rs   Nov15 5902:59 postgres: gitlab gitlabhq_production [local] UPDATE

And more specifically:

 ps aux | grep -i postgres
root       406  0.0  0.0   2384   636 ?        Ss   Nov07   0:00 runsv postgresql
gitlab-+   408  0.0  0.1  34920 19928 ?        Ss   Nov07   0:25 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
root       413  0.0  0.0   2528  1600 ?        S    Nov07   0:03 svlogd -tt /var/log/gitlab/postgresql
gitlab-+   419  0.0  0.0  35028  6772 ?        Ss   Nov07   0:09 postgres: checkpointer
gitlab-+   420  0.0  0.0  34920  5160 ?        Ss   Nov07   1:39 postgres: background writer
gitlab-+   421  0.0  0.0  34920  3616 ?        Ss   Nov07   0:50 postgres: walwriter
gitlab-+   422  0.0  0.0  35620  7936 ?        Ss   Nov07   1:24 postgres: autovacuum launcher
gitlab-+   423  0.0  0.0  18068  4808 ?        Ss   Nov07  13:39 postgres: stats collector
gitlab-+   424  0.0  0.0  35460  6160 ?        Ss   Nov07   0:00 postgres: logical replication launcher
root       884  0.0  0.0   2384   632 ?        Ss   Nov07   0:00 runsv postgres-exporter
root       901  0.0  0.0   2528   632 ?        S    Nov07   0:00 svlogd -tt /var/log/gitlab/postgres-exporter
gitlab-+  1025  0.2  0.1 1628492 24168 ?       Ssl  Nov07  47:55 /opt/gitlab/embedded/bin/postgres_exporter --web.listen-address=localhost:9187 --extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
gitlab-+  1074  0.6  0.1  43340 22688 ?        Ss   Nov07 150:49 postgres: gitlab gitlabhq_production [local] idle
root      1107  0.0  0.0   4304   500 ?        S    Nov07   2:37 tail --follow=name --retry /var/log/gitlab/gitlab-rails/application.log /var/log/gitlab/gitlab-rails/production_json.log /var/log/gitlab/gitlab-rails/auth.log /var/log/gitlab/gitlab-rails/grpc.log /var/log/gitlab/gitlab-rails/api_json.log /var/log/gitlab/gitlab-rails/production.log /var/log/gitlab/gitlab-rails/service_measurement.log /var/log/gitlab/gitlab-rails/application_json.log /var/log/gitlab/gitlab-rails/sidekiq_client.log /var/log/gitlab/alertmanager/current /var/log/gitlab/sidekiq/current /var/log/gitlab/postgres-exporter/current /var/log/gitlab/prometheus/current /var/log/gitlab/gitlab-exporter/current /var/log/gitlab/grafana/current /var/log/gitlab/redis/current /var/log/gitlab/gitlab-workhorse/current /var/log/gitlab/redis-exporter/current /var/log/gitlab/sshd/current /var/log/gitlab/postgresql/current /var/log/gitlab/nginx/gitlab_access.log /var/log/gitlab/nginx/gitlab_error.log /var/log/gitlab/nginx/error.log /var/log/gitlab/nginx/access.log /var/log/gitlab/nginx/current /var/log/gitlab/logrotate/current /var/log/gitlab/gitaly/current /var/log/gitlab/gitaly/gitaly_ruby_json.log /var/log/gitlab/puma/puma_stdout.log /var/log/gitlab/puma/current /var/log/gitlab/puma/puma_stderr.log
gitlab-+  1108  0.5  0.3  66712 45724 ?        Ss   Nov07 122:35 postgres: gitlab-psql gitlabhq_production [local] idle
gitlab-+  9206 58.0  0.2  48736 29192 ?        Rs   Nov15 5899:21 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 13332 58.5  0.2  47508 27308 ?        Ss   Nov15 5923:20 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 14100 57.9  0.2  48864 29088 ?        Rs   Nov15 5858:23 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16118 58.1  0.2  48120 27924 ?        Rs   Nov15 5872:15 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16122 57.5  0.2  47136 25960 ?        Rs   Nov15 5811:28 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16123 58.7  0.2  47544 26372 ?        Rs   Nov15 5934:15 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 16124 58.0  0.2  48060 27180 ?        Rs   Nov15 5869:05 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 28189  0.0  0.1  44160 24256 ?        Ss   09:12   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28384  0.0  0.2  46580 25524 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28409  0.0  0.2  47164 27776 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28461  0.0  0.2  48300 27100 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28488  0.0  0.2  48752 28896 ?        Ss   09:14   0:00 postgres: gitlab gitlabhq_production [local] UPDATE waiting
gitlab-+ 28810  0.0  0.2  47220 27168 ?        Ss   09:17   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 28918  0.0  0.2  46244 25420 ?        Ss   09:18   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 29310  0.0  0.2  45860 25228 ?        Ss   09:21   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 29985  0.0  0.2  46796 25748 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30076  0.0  0.1  44032 22040 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 30078  0.0  0.1  44604 22068 ?        Ss   09:25   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32035 58.7  0.2  47180 26432 ?        Rs   Nov15 5916:54 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32103  0.0  0.1  42696 21776 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32104  0.0  0.1  44100 21912 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32193  0.0  0.1  41420 19860 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32194  0.0  0.1  40536 18924 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32195  0.0  0.1  42228 20940 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32196  0.0  0.1  39208 17792 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32197  0.0  0.1  42260 20612 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32219  0.0  0.1  39476 18120 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32220  0.0  0.1  39136 17696 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32221  0.0  0.1  39504 18548 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32222  0.0  0.1  39124 17812 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
gitlab-+ 32223  0.0  0.1  39460 18556 ?        Ss   09:45   0:00 postgres: gitlab gitlabhq_production [local] idle
root     32309  0.0  0.0   5196  2344 pts/0    S+   09:46   0:00 grep --color=auto -i postgres
gitlab-+ 32329 57.9  0.1  42180 22884 ?        Ss   Nov15 5832:29 postgres: gitlab gitlabhq_production [local] UPDATE
gitlab-+ 32725 58.6  0.1  40924 19788 ?        Rs   Nov15 5910:07 postgres: gitlab gitlabhq_production [local] UPDATE

Again, thank you for your time

And that’s the problem. As per the Gitlab documentation: GitLab Runner | GitLab

So, Gitlab on it’s own needs 4cpu and 8gb ram. Your top/htop shows your server has 12GB, so means only 4GB for Runners. You mentioned you have multiple runners also running.

My suggestion is you remove all the runners from this server, and put them on different VM’s. Also for performance, each runner should have their own VM. You should not be attempting to run multiple runners on the same machine. All you are doing is sharing resources between all runners, and everything will take longer to complete that way. Alternatively, create a single server for multiple container runners and ensure the server has enough cpu and ram for all of them. Eg: assuming 4cpu and 8gb of ram for each runner, and you have say 4 runners, that is 16cpu and 32gb ram.

But definitely, you should never have them on the Gitlab server, and this is why you have a problem.

Also, once you have removed the runners from the Gitlab server, I suggest following the Gitlab upgrade docs, as 15.5.4 is the latest version. You don’t want to be running older versions due to security vulnerabilities. Upgrading GitLab | GitLab

According to the upgrade path, you need to upgrade through multiple versions:

the Gitlab docs will have all the commands you need to ensure you upgrade successfully. But only do that once you have removed all your runners from the Gitlab server.

Hmm, verification taken, the gitlab server is on one node/VM and the runners are on other nodes/VM

XXX-gitlab-runner-fb77755cb-f6b8h     1/1     Running       1          57d     10.42.2.71    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-86b964977f-ssqmm    1/1     Running       3          295d    10.42.2.67    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-79ff9454b8-z54fb    1/1     Running       1          110d    10.42.2.59    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-6bd57f785c-xmnmr    1/1     Terminating   1          216d    10.42.2.63    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-dfb7f7c86-rxpm2     1/1     Running       1          117d    10.42.0.7     X.Y.Z.70   <none>           <none>
XXX-gitlab-runner-5cdd4ddfc-dfkdv     1/1     Running       1          91d     10.42.2.60    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-566cdd9cb8-k6jmf    1/1     Running       5          315d    10.42.0.2     X.Y.Z.70   <none>           <none>
XXX-gitlab-runner-7bf9d75b54-xbpvk    1/1     Running       3          309d    10.42.4.30    X.Y.Z.74   <none>           <none>
XXX-gitlab-runner-5bbb58985-8gwt9     1/1     Running       0          17m     10.42.3.111   X.Y.Z.73   <none>           <none>
XXX-gitlab-runner-d85bc669f-7b6xh     1/1     Running       1          83d     10.42.0.8     X.Y.Z.70   <none>           <none>
XXX-gitlab-runner-cd8cf54c6-jwd25     1/1     Running       0          6h14m   10.42.3.109   X.Y.Z.73   <none>           <none>
XXX-gitlab-6df5f7f9c8-wbbzx           1/1     Running       4          319d    10.42.1.59    X.Y.Z.71   <none>           <none>
XXX-gitlab-runner-697fd4c97b-ttjv8    1/1     Running       1          116d    10.42.2.57    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-68d5f5c48-cf4ws     1/1     Running       3          313d    10.42.4.32    X.Y.Z.74   <none>           <none>
XXX-gitlab-runner-57f6b9b5c9-hjjnn    1/1     Running       3          276d    10.42.2.70    X.Y.Z.72   <none>           <none>
XXX-gitlab-runner-557ffff968-8cmls    1/1     Running       0          31h     10.42.2.84    X.Y.Z.72   <none>           <none>

Only the VM that host gitlab’s node is overloaded.

Interesting fact that may be related: a runner crashed a few minutes ago, and that seemed to give a little breath to the gitlab server but I can’t understand the reason
image

The upgrade is planned, thanks for the advice!