Gitlab 502 all sockets connection refused

How to Use GitLab Self-managed
1

Hi all, I hope you’re all well.

Our gitlab CE instance returns 502 error when trying to access the webpages, and we can’t pull or push to the repos either.

I have spent since the 25/11/2021 trying to fix this myself and am pretty much stuck.

The version I am running is GitLab Community Edition 13.9.2, I am using Puma not Unicorn and when running sudo gitlab-ctl status all services are OK, however running sudo gitlab-ctl tail and I can see that all of the sockets are saying connection refused.

The other issue I am having is that trying to run any of the rake commands and this simply returns the message “Killed”.

We had an underlying issue with the repos last week where we could not read from them (I think someone deleted some large files from the server, that may have caused this)

And when I looked into that issue, after I did sudo gitlab-ctl reconfigure we seemed to start getting these issues.

The server itself has 40 cores, core 1-20 is at 100% all the time regardless of if gitlab is running or not. It has 128GB of RAM of which around 50% is free since I restarted mysqld.

The server is normally only used for mysqld, but it does have gitlab running on it as well.

I would greatly appreciate any help / advice I can get from you guys and the community.

Regards,
Mike D

2

Could be you have some processes running as git user? Check this, because sounds like your server has been compromised: Gitlab , git juma - #2 by iwalker

3

Hello iwalker, thanks for your reply. Here is the result of ps aux | grep “git”

git       4005  3.7  0.0 1539508 56552 ?       Ssl  Nov25 209:26 /mnt/tm8node3/archive/sxhvp/cqoicqtegp
git       7132 95.7  0.0 274384 73792 ?        Rs   11:44   0:08 puma 5.1.1 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]
git       7148  0.0  0.0 113176  1224 ?        S    11:44   0:00 /bin/bash -c ps axf -o "pid"|while read procid do         ls -l /proc/$procid/fd | grep /tmp         if [ $? -ne 1 ]         then                 ls -l /proc/$procid/fd| grep -a -E "xxx"                 if [ $? -ne 0 ]                 then                         kill -9 $procid ??????echo zhaodao $procid                 else                         echo "don't kill"$procid                 fi         fi done
git       7150 13.2  0.0 113176   852 ?        S    11:44   0:00 /bin/bash -c ps axf -o "pid"|while read procid do         ls -l /proc/$procid/fd | grep /tmp         if [ $? -ne 1 ]         then                 ls -l /proc/$procid/fd| grep -a -E "xxx"                 if [ $? -ne 0 ]                 then                         kill -9 $procid ??????echo zhaodao $procid                 else                         echo "don't kill"$procid                 fi         fi done
git       7939  5.8  0.0 153476 15808 ?        Ssl  11:44   0:00 ruby /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -m 50 --timeout 25 *
git       8137 99.2  0.0 229468 50344 ?        R    11:44   0:04 /opt/gitlab/embedded/bin/sidekiq -c50 -eproduction -t25 -gqueues:authorized_project_update:authorized_project_update_project_create,authorized_project_update:authorized_project_update_project_group_link_create,authorized_project_update:authorized_project_update_user_refresh_over_user_range,authorized_project_update:authorized_project_update_user_refresh_with_low_urgency,auto_devops:auto_devops_disable,auto_merge:auto_merge_process,chaos:chaos_cpu_spin,chaos:chaos_db_spin,chaos:chaos_kill,chaos:chaos_leak_mem,chaos:chaos_sleep,container_repository:cleanup_container_repository,container_repository:container_expiration_policies_cleanup_container_repository,container_repository:delete_container_repository,cronjob:admin_email,cronjob:analytics_instance_statistics_count_job_trigger,cronjob:authorized_project_update_periodic_recalculate,cronjob:ci_archive_traces_cron,cronjob:ci_pipeline_artifacts_expire_artifacts,cronjob:ci_platform_metrics_update_cron,cronjob:ci_schedule_delete_objects_cron,cronjob:container_expir
git       9456  0.0  0.0 709908 13956 ?        Sl   Nov25   0:17 SSHD2
root     11437  0.0  0.0 112712   976 pts/0    S+   11:44   0:00 grep --color=auto git
root     18849  0.0  0.0   4380   504 ?        Ss   10:23   0:00 runsvdir -P /opt/gitlab/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
root     18866  0.0  0.0   4228   356 ?        Ss   10:23   0:00 runsv gitaly
root     18870  0.0  0.0   4228   352 ?        Ss   10:23   0:00 runsv gitlab-workhorse
root     18880  0.0  0.0   4228   356 ?        Ss   10:23   0:00 runsv gitlab-exporter
root     18881  0.0  0.0   4372   556 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/postgresql
root     18882  0.0  0.0   4372   552 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/prometheus
root     18883  0.0  0.0   4372   552 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/grafana
root     18884  0.0  0.0   4372   552 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/alertmanager
root     18885  0.0  0.0   4372   348 ?        S    10:23   0:00 svlogd /var/log/gitlab/gitaly
root     18886  0.0  0.0   4372   348 ?        S    10:23   0:00 svlogd /var/log/gitlab/sidekiq
root     18887  0.0  0.0   4372   556 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/redis-exporter
root     18888  0.0  0.0   4372   552 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/redis
root     18889  0.0  0.0   4372   556 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/node-exporter
root     18890  0.0  0.0   4372   348 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/nginx
root     18891  0.0  0.0   4372   552 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/postgres-exporter
gitlab-+ 18892  0.0  0.0 1814312 22008 ?       Ssl  10:23   0:01 /opt/gitlab/embedded/bin/grafana-server -config /var/opt/gitlab/grafana/grafana.ini
gitlab-+ 18893  0.0  0.0 2154544 68492 ?       Ss   10:23   0:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/postgresql/data
gitlab-+ 18894  0.8  0.0 2015548 100168 ?      Ssl  10:23   0:41 /opt/gitlab/embedded/bin/prometheus --web.listen-address=localhost:9090 --storage.tsdb.path=/var/opt/gitlab/prometheus/data --config.file=/var/opt/gitlab/prometheus/prometheus.yml
gitlab-+ 18895  0.1  0.0 1639476 12996 ?       Ssl  10:23   0:05 /opt/gitlab/embedded/bin/alertmanager --web.listen-address=localhost:9093 --storage.path=/var/opt/gitlab/alertmanager/data --config.file=/var/opt/gitlab/alertmanager/alertmanager.yml
git      18896  0.0  0.0 1076684 1768 ?        Ssl  10:23   0:00 /opt/gitlab/embedded/bin/gitaly-wrapper /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
root     18897  0.0  0.0   4372   552 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/registry
gitlab-+ 18899  0.0  0.0 1173640 15668 ?       Ssl  10:23   0:04 /opt/gitlab/embedded/bin/redis_exporter --web.listen-address=localhost:9121 --redis.addr=unix:///var/opt/gitlab/redis/redis.socket
root     18900  0.0  0.0   4372   352 ?        S    10:23   0:00 svlogd /var/log/gitlab/gitlab-workhorse
gitlab-+ 18901  0.3  0.0  69412  5028 ?        Ssl  10:23   0:15 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
root     18902  0.0  0.0   4372   556 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/puma
root     18903  0.0  0.0   4372   352 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/logrotate
gitlab-+ 18905  1.5  0.0 723348 11484 ?        Ssl  10:23   1:13 /opt/gitlab/embedded/bin/node_exporter --web.listen-address=localhost:9100 --collector.mountstats --collector.runit --collector.runit.servicedir=/opt/gitlab/sv --collector.textfile.directory=/var/opt/gitlab/node-exporter/textfile_collector
root     18906  0.0  0.0  50572  3948 ?        Ss   10:23   0:00 nginx: master process /opt/gitlab/embedded/sbin/nginx -p /var/opt/gitlab/nginx
gitlab-+ 18908  0.3  0.0 1346272 11276 ?       Ssl  10:23   0:15 /opt/gitlab/embedded/bin/postgres_exporter --web.listen-address=localhost:9187 --extend.query-path=/var/opt/gitlab/postgres-exporter/queries.yaml
registry 18909  0.0  0.0 1560476 7652 ?        Ssl  10:23   0:00 /opt/gitlab/embedded/bin/registry serve ./config.yml
root     18910  0.0  0.0   4372   556 ?        S    10:23   0:00 svlogd -tt /var/log/gitlab/gitlab-exporter
git      18911  0.0  0.0 1346732 23304 ?       Ssl  10:23   0:03 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sockets/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr  -proxyHeadersTimeout 3m0s -prometheusListenAddr localhost:9229 -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret -logFormat json -config config.toml
git      18914  1.6  0.0 280148 40260 ?        Ssl  10:23   1:19 /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/gitlab-exporter web -c /var/opt/gitlab/gitlab-exporter/gitlab-exporter.yml
git      18988  0.4  0.0 1813832 26376 ?       Sl   10:23   0:20 /opt/gitlab/embedded/bin/gitaly /var/opt/gitlab/gitaly/config.toml
gitlab-+ 18998  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19001  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19003  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19004  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19005  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19006  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19007  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19008  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19009  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19010  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19011  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19013  0.0  0.0  56828  7604 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19014  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19015  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19016  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19019  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19020  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19021  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19022  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19023  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19024  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19026  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19027  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19028  0.0  0.0  56828  7608 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19029  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19030  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19031  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19035  0.0  0.0  57152  8020 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19036  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19037  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19038  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19042  0.0  0.0  56828  7604 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19043  0.0  0.0  56828  7604 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19044  0.0  0.0  57160  7780 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19047  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19048  0.0  0.0  56828  5864 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19049  0.0  0.0  56828  7832 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19050  0.0  0.0  56828  7824 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19051  0.0  0.0  56828  7612 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19053  0.0  0.0  56828  7604 ?        S    10:23   0:00 nginx: worker process
gitlab-+ 19103  0.0  0.0  52656  1900 ?        S    10:23   0:00 nginx: cache manager process
gitlab-+ 19173  0.0  0.0 2154676 18632 ?       Ss   10:23   0:00 postgres: checkpointer
gitlab-+ 19174  0.0  0.0 2154544 18444 ?       Ss   10:23   0:00 postgres: background writer
gitlab-+ 19176  0.0  0.0 2154544 2352 ?        Ss   10:23   0:00 postgres: walwriter
gitlab-+ 19178  0.0  0.0 2155088 3484 ?        Ss   10:23   0:00 postgres: autovacuum launcher
gitlab-+ 19179  0.0  0.0  32632  2800 ?        Ss   10:23   0:01 postgres: stats collector
gitlab-+ 19180  0.0  0.0 2155104 3076 ?        Ss   10:23   0:00 postgres: logical replication launcher
gitlab-+ 19716  0.6  0.0 2285368 40892 ?       Ss   10:23   0:30 postgres: gitlab-psql gitlabhq_production [local] idle
git      22683  0.3  0.0 3001064 72288 ?       Sl   10:23   0:15 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 18988 /var/opt/gitlab/gitaly/internal_sockets/ruby.0
git      22688  0.3  0.0 3001064 72420 ?       Sl   10:23   0:15 ruby /opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby 18988 /var/opt/gitlab/gitaly/internal_sockets/ruby.1
gitlab-+ 26846  1.8  0.0 2264732 22696 ?       Ss   10:23   1:31 postgres: gitlab gitlabhq_production [local] idle
root     34078  0.0  0.0  11680  1460 ?        Ss   11:23   0:00 /bin/sh /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper
root     35141  1.3  0.0  47704 14968 ?        Ssl  Nov26  61:53 /usr/lib/gitlab-runner/gitlab-runner run --working-directory /etc/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
git      35936 2046  3.6 7818536 4828820 ?     Ssl  Nov28 17374:54 /usr/java/latest/bin/java -Djava.util.logging.config.file=/data/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx5120M -server -XX:+UseParallelGC -Djavax.net.debug=ssl -Dignore.endorsed.dirs= -classpath /data/apache-tomcat/bin/bootstrap.jar:/data/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/data/apache-tomcat -Dcatalina.home=/data/apache-tomcat -Djava.io.tmpdir=/data/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

And here is if I run the following first:

gitlab-ctl stop
systemctl stop gitlab-runsvdir

git       4005  3.7  0.0 1539508 54040 ?       Ssl  Nov25 209:31 /mnt/tm8node3/archive/sxhvp/cqoicqtegp
git       9456  0.0  0.0 709908 13956 ?        Sl   Nov25   0:17 SSHD2
root     35141  1.3  0.0  47704 15004 ?        Ssl  Nov26  61:54 /usr/lib/gitlab-runner/gitlab-runner run --working-directory /etc/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
git      35936 2046  3.6 7818536 4828820 ?     Ssl  Nov28 17416:06 /usr/java/latest/bin/java -Djava.util.logging.config.file=/data/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx5120M -server -XX:+UseParallelGC -Djavax.net.debug=ssl -Dignore.endorsed.dirs= -classpath /data/apache-tomcat/bin/bootstrap.jar:/data/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/data/apache-tomcat -Dcatalina.home=/data/apache-tomcat -Djava.io.tmpdir=/data/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start
git      37650  0.0  0.0 113176  1224 ?        S    11:46   0:00 /bin/bash -c ps axf -o "pid"|while read procid do         ls -l /proc/$procid/fd | grep /tmp         if [ $? -ne 1 ]         then                 ls -l /proc/$procid/fd| grep -a -E "xxx"                 if [ $? -ne 0 ]                 then                         kill -9 $procid ??????echo zhaodao $procid                 else                         echo "don't kill"$procid                 fi         fi done
git      37652 14.5  0.0 113176   800 ?        S    11:46   0:00 /bin/bash -c ps axf -o "pid"|while read procid do         ls -l /proc/$procid/fd | grep /tmp         if [ $? -ne 1 ]         then                 ls -l /proc/$procid/fd| grep -a -E "xxx"                 if [ $? -ne 0 ]                 then                         kill -9 $procid ??????echo zhaodao $procid                 else                         echo "don't kill"$procid                 fi         fi done
root     38257  0.0  0.0 112712   976 pts/0    S+   11:46   0:00 grep --color=auto git

Any further helper would be appreciated.

Regards,
Mike

4

Hi iwalker,

Thanks for your reply. After running sudo gitlab-ctl stop and systemctl stop gitlab-runsvdir followed by ps aux | grep "git" this is the result:

git       4005  3.7  0.0 1539508 54244 ?       Ssl  Nov25 209:55 /mnt/tm8node3/archive/sxhvp/cqoicqtegp
git       9456  0.0  0.0 709908 13908 ?        Sl   Nov25   0:17 SSHD2
git      12643  0.0  0.0 113176  1224 ?        S    11:58   0:00 /bin/bash -c ps axf -o "pid"|while read procid do         ls -l /proc/$procid/fd | grep /tmp         if [ $? -ne 1 ]         then                 ls -l /proc/$procid/fd| grep -a -E "xxx"                 if [ $? -ne 0 ]                 then                         kill -9 $procid ??????echo zhaodao $procid                 else                         echo "don't kill"$procid                 fi         fi done
git      12645  0.0  0.0 113176   796 ?        S    11:58   0:00 /bin/bash -c ps axf -o "pid"|while read procid do         ls -l /proc/$procid/fd | grep /tmp         if [ $? -ne 1 ]         then                 ls -l /proc/$procid/fd| grep -a -E "xxx"                 if [ $? -ne 0 ]                 then                         kill -9 $procid ??????echo zhaodao $procid                 else                         echo "don't kill"$procid                 fi         fi done
root     12694  0.0  0.0 112712   972 pts/0    S+   11:58   0:00 grep --color=auto git
root     35141  1.3  0.0  47704 14948 ?        Ssl  Nov26  62:25 /usr/lib/gitlab-runner/gitlab-runner run --working-directory /etc/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner
git      35936 2045  3.6 7818536 4828820 ?     Ssl  Nov28 17647:10 /usr/java/latest/bin/java -Djava.util.logging.config.file=/data/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx5120M -server -XX:+UseParallelGC -Djavax.net.debug=ssl -Dignore.endorsed.dirs= -classpath /data/apache-tomcat/bin/bootstrap.jar:/data/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/data/apache-tomcat -Dcatalina.home=/data/apache-tomcat -Djava.io.tmpdir=/data/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

Regards,
Mike

5

There shouldn’t be any processes running as git user. Kill them, and remove them from the server as these are cryptominers. Not even java should be running as git either or apache-tomcat.

6

WOW!

That’s fixed it, but unfortunatly I am back to the original issue I had - I wonder if you know how to fix this as well - and I owe you a beer!

I cannot read from the repos at the moment.

image

image
image1587×1059 46.4 KB

Best Wishes,
Mike

7

Iwalker,

Looks like the apache tomcat process has started back up again.

Taking over all the CPU cores

8

You will have to remove that as well to stop it from starting up.

9

Hi iwalker,

It actually looks like something one of our developers setup in the past, but cant find what is starting it - we dont even appear to have tomcat installed.

So unsure how to prevent that from starting up again / or removing it at the moment.

It does even look like we have apache tomcat installed as a service. So this is very strange.

Regards,
Mike

10

Hey again iwalker,

So now I am back to 502 error, even after killing the apache tomcat process.

Connection refused on the sockets again. It felt great when that 502 went away because of this. But now It’s here all the time.

Thanks for everything and regards,
Mike D

11

Hey,

I was going to remove tomcat, as we are not using it for anything but then some gitlab dependencies were shown and I aborted:

[root@telemetry usr]# sudo yum remove tomcat-el-2.2-api-7.0.76-9.el7_6.noarch
Loaded plugins: fastestmirror, langpacks, versionlock
Resolving Dependencies
--> Running transaction check
---> Package tomcat-el-2.2-api.noarch 0:7.0.76-9.el7_6 will be erased
--> Processing Dependency: tomcat-el-2.2-api = 7.0.76-9.el7_6 for package: tomcat-lib-7.0.76-9.el7_6.noarch
--> Running transaction check
---> Package tomcat-lib.noarch 0:7.0.76-9.el7_6 will be erased
--> Finished Dependency Resolution
base/7/x86_64                                                                                               | 3.6 kB  00:00:00
docker-ce-stable/x86_64                                                                                     | 3.5 kB  00:00:00
drivesrvr                                                                                                   | 2.9 kB  00:00:00
epel/x86_64/metalink                                                                                        |  30 kB  00:00:00
epel/x86_64                                                                                                 | 4.7 kB  00:00:00
epel/x86_64/updateinfo                                                                                      | 1.0 MB  00:00:00
epel/x86_64/primary_db                                                                                      | 7.0 MB  00:00:00
extras/7/x86_64                                                                                             | 2.9 kB  00:00:00
gitlab_gitlab-ce/x86_64/signature                                                                           |  862 B  00:00:00
gitlab_gitlab-ce/x86_64/signature                                                                           | 1.0 kB  00:00:00 !!!
gitlab_gitlab-ce-source/signature                                                                           |  862 B  00:00:00
gitlab_gitlab-ce-source/signature                                                                           |  951 B  00:00:00 !!!
mysql-connectors-community/x86_64                                                                           | 2.6 kB  00:00:00
mysql-tools-community/x86_64                                                                                | 2.6 kB  00:00:00
mysql80-community/x86_64                                                                                    | 2.6 kB  00:00:00
rackspace                                                                                                   | 2.9 kB  00:00:00
remi-php72                                                                                                  | 3.0 kB  00:00:00
remi-php72/primary_db                                                                                       | 258 kB  00:00:00
remi-safe                                                                                                   | 3.0 kB  00:00:00
remi-safe/primary_db                                                                                        | 2.1 MB  00:00:00
runner_gitlab-runner/x86_64/signature                                                                       |  862 B  00:00:00
runner_gitlab-runner/x86_64/signature                                                                       | 1.0 kB  00:00:00 !!!
runner_gitlab-runner-source/signature                                                                       |  862 B  00:00:00
runner_gitlab-runner-source/signature                                                                       |  951 B  00:00:00 !!!
updates/7/x86_64                                                                                            | 2.9 kB  00:00:00
zabbix/x86_64                                                                                               | 2.9 kB  00:00:00
zabbix-non-supported/x86_64                                                                                 | 2.9 kB  00:00:00

Dependencies Resolved

===================================================================================================================================
 Package                             Arch                     Version                             Repository                  Size
===================================================================================================================================
Removing:
 tomcat-el-2.2-api                   noarch                   7.0.76-9.el7_6                      @updates                   113 k
Removing for dependencies:
 tomcat-lib                          noarch                   7.0.76-9.el7_6                      @updates                   4.2 M

Transaction Summary
===================================================================================================================================
Remove  1 Package (+1 Dependent package)

Installed size: 4.4 M
Is this ok [y/N]: n
Exiting on user command

tomcat / java do not appear to be used by anything I care about on this server - so I was going to remove them. But the above worried me.

What do you think iwalker please?

Regards,
Mike

12

Well it’s only going to remove tomcat-el and tomcat-lib so there isn’t any problem there. Gitlab doesn’t use tomcat or java anyway. Potentially they could be running on ports that Gitlab wants so could be a source of the problems.

Best would be to uninstall those two tomcat packages as per your output. Then when that is done:

gitlab-ctl status

and perhaps even restart the gitlab services at this point and check the status again to make sure all is running.

13

Thanks iwalker.

I have removed tomcat via the command above, but the following still finds a way to start again:

git      19621 2242  3.5 7621900 4699480 ?     Ssl  08:29 315:24 /usr/java/latest/bin/java -Djava.util.logging.config.file=/data/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx5120M -server -XX:+UseParallelGC -Djavax.net.debug=ssl -Dignore.endorsed.dirs= -classpath /data/apache-tomcat/bin/bootstrap.jar:/data/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/data/apache-tomcat -Dcatalina.home=/data/apache-tomcat -Djava.io.tmpdir=/data/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start

This is what status looks like after I killed it:

run: alertmanager: (pid 22866) 10s; run: log: (pid 2882) 1305s
run: gitaly: (pid 22903) 10s; run: log: (pid 2893) 1305s
run: gitlab-exporter: (pid 22947) 9s; run: log: (pid 2892) 1305s
run: gitlab-workhorse: (pid 22961) 9s; run: log: (pid 2886) 1305s
run: grafana: (pid 22986) 9s; run: log: (pid 2880) 1305s
run: logrotate: (pid 23022) 8s; run: log: (pid 2913) 1305s
run: nginx: (pid 23453) 8s; run: log: (pid 2908) 1305s
run: node-exporter: (pid 23943) 7s; run: log: (pid 2890) 1305s
run: postgres-exporter: (pid 24520) 7s; run: log: (pid 2898) 1305s
run: postgresql: (pid 24986) 6s; run: log: (pid 2894) 1305s
run: prometheus: (pid 25435) 6s; run: log: (pid 2884) 1305s
run: puma: (pid 32038) 0s; run: log: (pid 2885) 1305s
run: redis: (pid 26367) 5s; run: log: (pid 2887) 1305s
run: redis-exporter: (pid 26796) 5s; run: log: (pid 2881) 1305s
run: registry: (pid 27244) 4s; run: log: (pid 2901) 1305s
run: sidekiq: (pid 27630) 4s; run: log: (pid 2883) 1305s

All of the gitlab services always start up fine according to that.

Please see the latest logs for each service:

alertmanager

[root@telemetry ~]# sudo gitlab-ctl tail alertmanager
==> /var/log/gitlab/alertmanager/state <==

==> /var/log/gitlab/alertmanager/current <==
2021-11-30_08:21:31.15665 level=info ts=2021-11-30T08:21:31.153Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..."
2021-11-30_08:22:37.44483 level=info ts=2021-11-30T08:22:37.444Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=master, revision=)"
2021-11-30_08:22:37.44492 level=info ts=2021-11-30T08:22:37.444Z caller=main.go:217 build_context="(go=go1.14.7, user=GitLab-Omnibus, date=)"
2021-11-30_08:22:37.46176 level=info ts=2021-11-30T08:22:37.461Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.17.0.1 port=9094
2021-11-30_08:22:37.46457 level=info ts=2021-11-30T08:22:37.464Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s
2021-11-30_08:22:37.49922 level=info ts=2021-11-30T08:22:37.499Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml
2021-11-30_08:22:37.50024 level=info ts=2021-11-30T08:22:37.500Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml
2021-11-30_08:22:37.50322 level=info ts=2021-11-30T08:22:37.503Z caller=main.go:485 msg=Listening address=localhost:9093
2021-11-30_08:22:39.46471 level=info ts=2021-11-30T08:22:39.464Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000058041s
2021-11-30_08:22:47.46529 level=info ts=2021-11-30T08:22:47.465Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000657923s

gitaly

[root@telemetry ~]# sudo gitlab-ctl tail gitaly
==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <==
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:08Z","grpc.time_ms":0.184,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5608,"correlation_id":"016bd6fde32f183334cc71e7a7eea640","time":"2021-11-30T08:27:08.067Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:08Z","grpc.time_ms":0.181,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5606,"correlation_id":"717590c69db00f532c58e6d71c748153","time":"2021-11-30T08:27:08.067Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:23Z","grpc.time_ms":0.178,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5608,"correlation_id":"8d3bd88190f3932e4848ad2b62ccb068","time":"2021-11-30T08:27:23.068Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:23Z","grpc.time_ms":0.173,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5606,"correlation_id":"d520114f7ae84f8b9a1be05f2f2e02a9","time":"2021-11-30T08:27:23.068Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:38Z","grpc.time_ms":0.154,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5608,"correlation_id":"917fb4d4f51b3fd9f7ccfeea7e6b2def","time":"2021-11-30T08:27:38.069Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:38Z","grpc.time_ms":0.194,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5606,"correlation_id":"0d5b7d1a4b4a4fd983a9e1c50548d373","time":"2021-11-30T08:27:38.069Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:53Z","grpc.time_ms":0.18,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5608,"correlation_id":"573aebdd959b5f951df3610eb8f486fb","time":"2021-11-30T08:27:53.070Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:27:53Z","grpc.time_ms":0.231,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5606,"correlation_id":"0c57e5a8c946d787cc37478448e71228","time":"2021-11-30T08:27:53.070Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:28:08Z","grpc.time_ms":0.205,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5606,"correlation_id":"c6e14c162b4bc28d00a9493df94b19c2","time":"2021-11-30T08:28:08.071Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-11-30T08:28:08Z","grpc.time_ms":0.225,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":5608,"correlation_id":"b14cb1228a6558a6e7cf7847a861d85a","time":"2021-11-30T08:28:08.071Z"}

==> /var/log/gitlab/gitaly/state <==

==> /var/log/gitlab/gitaly/current <==
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","supervisor.pid":4047,"time":"2021-11-30T08:22:38.036Z"}
{"level":"warning","msg":"grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1  \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: connection refused\". Reconnecting...","pid":2983,"system":"system","time":"2021-11-30T08:22:38.036Z"}
{"level":"info","msg":"PID 4047 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-11-30T08:22:38.354Z"}
{"level":"info","msg":"PID 4045 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-11-30T08:22:38.355Z"}
{"error":"signal: killed","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-11-30T08:22:39.673Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":5606,"time":"2021-11-30T08:22:39.674Z"}
{"error":"signal: killed","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-11-30T08:22:39.675Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","supervisor.pid":5608,"time":"2021-11-30T08:22:39.675Z"}
{"level":"info","msg":"PID 5608 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2021-11-30T08:22:39.983Z"}
{"level":"info","msg":"PID 5606 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","2983","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-11-30T08:22:39.984Z"}

==> /var/log/gitlab/gitaly/gitlab-shell.log <==

==> /var/log/gitlab/gitaly/gitaly_hooks.log <==

gitlab-exporter

[root@telemetry ~]# sudo gitlab-ctl tail gitlab-exporter
==> /var/log/gitlab/gitlab-exporter/state <==

==> /var/log/gitlab/gitlab-exporter/current <==
2021-11-30_08:28:05.88510 ::1 - - [30/Nov/2021:08:28:05 UTC] "GET /database HTTP/1.1" 200 2565
2021-11-30_08:28:05.88512 - -> /database
2021-11-30_08:28:09.60515 ::1 - - [30/Nov/2021:08:28:09 UTC] "GET /sidekiq HTTP/1.1" 200 70296
2021-11-30_08:28:09.60516 - -> /sidekiq
2021-11-30_08:28:20.58340 ::1 - - [30/Nov/2021:08:28:20 UTC] "GET /ruby HTTP/1.1" 200 996
2021-11-30_08:28:20.58341 - -> /ruby
2021-11-30_08:28:20.91265 ::1 - - [30/Nov/2021:08:28:20 UTC] "GET /database HTTP/1.1" 200 2565
2021-11-30_08:28:20.91267 - -> /database
2021-11-30_08:28:24.60450 ::1 - - [30/Nov/2021:08:28:24 UTC] "GET /sidekiq HTTP/1.1" 200 70296
2021-11-30_08:28:24.60452 - -> /sidekiq
2021-11-30_08:28:35.57613 ::1 - - [30/Nov/2021:08:28:35 UTC] "GET /ruby HTTP/1.1" 200 996
2021-11-30_08:28:35.57615 - -> /ruby
2021-11-30_08:28:35.88154 ::1 - - [30/Nov/2021:08:28:35 UTC] "GET /database HTTP/1.1" 200 2565
2021-11-30_08:28:35.88156 - -> /database
2021-11-30_08:28:39.62881 ::1 - - [30/Nov/2021:08:28:39 UTC] "GET /sidekiq HTTP/1.1" 200 70296
2021-11-30_08:28:39.62883 - -> /sidekiq
2021-11-30_08:28:50.57137 ::1 - - [30/Nov/2021:08:28:50 UTC] "GET /ruby HTTP/1.1" 200 996
2021-11-30_08:28:50.57140 - -> /ruby
2021-11-30_08:28:50.88117 ::1 - - [30/Nov/2021:08:28:50 UTC] "GET /database HTTP/1.1" 200 2565
2021-11-30_08:28:50.88119 - -> /database
2021-11-30_08:28:54.62264 ::1 - - [30/Nov/2021:08:28:54 UTC] "GET /sidekiq HTTP/1.1" 200 70296
2021-11-30_08:28:54.62267 - -> /sidekiq

gitlab-workhorse

[root@telemetry ~]# sudo gitlab-ctl tail gitlab-workhorse
==> /var/log/gitlab/gitlab-workhorse/state <==

==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"01FNQZAVWRNH81GQPBEWBH55SJ","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-11-30T08:28:15Z","uri":"/favicon.ico"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FNQZAVWRNH81GQPBEWBH55SJ","duration_ms":0,"host":"gitlab.mypolicy.co.uk:9191","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mypolicy.co.uk:9191/","remote_addr":"193.117.129.74:0","remote_ip":"193.117.129.74","route":"","status":502,"system":"http","time":"2021-11-30T08:28:15Z","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","written_bytes":2940}
{"correlation_id":"01FNQZAW5FBCJ648A1E3JY0KWK","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"POST","msg":"","time":"2021-11-30T08:28:15Z","uri":"/api/v4/internal/allowed"}
{"content_type":"text/plain","correlation_id":"01FNQZAW5FBCJ648A1E3JY0KWK","duration_ms":0,"host":"unix","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/","status":502,"system":"http","time":"2021-11-30T08:28:15Z","ttfb_ms":0,"uri":"/api/v4/internal/allowed","user_agent":"GitLab-Shell","written_bytes":24}
{"correlation_id":"01FNQZBH91PW6N9B2CVGQF9JN1","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"POST","msg":"","time":"2021-11-30T08:28:37Z","uri":"/api/v4/jobs/request"}
{"content_type":"text/plain","correlation_id":"01FNQZBH91PW6N9B2CVGQF9JN1","duration_ms":0,"host":"gitlab.mypolicy.co.uk:9191","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"^/api/v4/jobs/request\\z","status":502,"system":"http","time":"2021-11-30T08:28:37Z","ttfb_ms":0,"uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 11.11.2 (; go1.8.7; linux/amd64)","written_bytes":24}
{"correlation_id":"01FNQZBHX8X0J3RC8MXZWTAGE2","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-11-30T08:28:38Z","uri":"/"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FNQZBHX8X0J3RC8MXZWTAGE2","duration_ms":1,"host":"gitlab.mypolicy.co.uk:9191","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mypolicy.co.uk:9191/","remote_addr":"193.117.129.74:0","remote_ip":"193.117.129.74","route":"","status":502,"system":"http","time":"2021-11-30T08:28:38Z","ttfb_ms":1,"uri":"/","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","written_bytes":2940}
{"correlation_id":"01FNQZBHZP8XMQPSDYNYJ003VB","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-11-30T08:28:38Z","uri":"/favicon.ico"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FNQZBHZP8XMQPSDYNYJ003VB","duration_ms":0,"host":"gitlab.mypolicy.co.uk:9191","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mypolicy.co.uk:9191/","remote_addr":"193.117.129.74:0","remote_ip":"193.117.129.74","route":"","status":502,"system":"http","time":"2021-11-30T08:28:38Z","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","written_bytes":2940}
{"correlation_id":"01FNQZCHDNK2B4AYDX1NGTBWVT","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-11-30T08:29:10Z","uri":"/"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FNQZCHDNK2B4AYDX1NGTBWVT","duration_ms":1,"host":"gitlab.mypolicy.co.uk:9191","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mypolicy.co.uk:9191/","remote_addr":"193.117.129.74:0","remote_ip":"193.117.129.74","route":"","status":502,"system":"http","time":"2021-11-30T08:29:10Z","ttfb_ms":1,"uri":"/","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","written_bytes":2940}
{"correlation_id":"01FNQZCHJ4JF60W3V1ERFKHD21","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-11-30T08:29:10Z","uri":"/favicon.ico"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FNQZCHJ4JF60W3V1ERFKHD21","duration_ms":0,"host":"gitlab.mypolicy.co.uk:9191","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mypolicy.co.uk:9191/","remote_addr":"193.117.129.74:0","remote_ip":"193.117.129.74","route":"","status":502,"system":"http","time":"2021-11-30T08:29:10Z","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36","written_bytes":2940}

grafana

[root@telemetry ~]# sudo gitlab-ctl tail grafana
==> /var/log/gitlab/grafana/state <==

==> /var/log/gitlab/grafana/current <==
2021-11-30_08:22:37.45571 t=2021-11-30T08:22:37+0000 lvl=info msg="Path Home" logger=settings path=/var/opt/gitlab/grafana
2021-11-30_08:22:37.45571 t=2021-11-30T08:22:37+0000 lvl=info msg="Path Data" logger=settings path=/var/opt/gitlab/grafana/data
2021-11-30_08:22:37.45572 t=2021-11-30T08:22:37+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/gitlab/grafana
2021-11-30_08:22:37.45572 t=2021-11-30T08:22:37+0000 lvl=info msg="Path Plugins" logger=settings path=/var/opt/gitlab/grafana/data/plugins
2021-11-30_08:22:37.45572 t=2021-11-30T08:22:37+0000 lvl=info msg="Path Provisioning" logger=settings path=/var/opt/gitlab/grafana/provisioning
2021-11-30_08:22:37.45573 t=2021-11-30T08:22:37+0000 lvl=info msg="App mode production" logger=settings
2021-11-30_08:22:37.45693 t=2021-11-30T08:22:37+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
2021-11-30_08:22:37.45860 t=2021-11-30T08:22:37+0000 lvl=info msg="Starting DB migrations" logger=migrator
2021-11-30_08:22:37.47227 t=2021-11-30T08:22:37+0000 lvl=info msg="Starting plugin search" logger=plugins
2021-11-30_08:22:37.56174 t=2021-11-30T08:22:37+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3000 protocol=http subUrl=/-/grafana socket=

logrotate

[root@telemetry ~]# sudo gitlab-ctl tail logrotate
==> /var/log/gitlab/logrotate/state <==

==> /var/log/gitlab/logrotate/current <==
2021-11-29_12:26:51.65376 Received TERM from runit, sending to process group (-PID)
2021-11-29_13:54:19.93311 Received TERM from runit, sending to process group (-PID)
2021-11-29_13:57:08.11288 Received TERM from runit, sending to process group (-PID)
2021-11-29_14:05:29.53906 Received TERM from runit, sending to process group (-PID)
2021-11-29_14:06:21.51084 Received TERM from runit, sending to process group (-PID)
2021-11-29_14:12:58.75198 Received TERM from runit, sending to process group (-PID)
2021-11-29_14:13:34.21314 Received TERM from runit, sending to process group (-PID)
2021-11-29_16:05:39.26274 Received TERM from runit, sending to process group (-PID)
2021-11-29_16:14:05.80529 Received TERM from runit, sending to process group (-PID)
2021-11-30_08:21:34.12732 Received TERM from runit, sending to process group (-PID)

nginx

[root@telemetry ~]# sudo gitlab-ctl tail nginx
==> /var/log/gitlab/nginx/gitlab_registry_error.log <==

==> /var/log/gitlab/nginx/state <==

==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
78.128.112.18 - - [30/Nov/2021:03:58:17 +0000] "-  -" 400 150 "" "-" -

==> /var/log/gitlab/nginx/current <==

==> /var/log/gitlab/nginx/error.log <==

==> /var/log/gitlab/nginx/gitlab_access.log <==
193.117.129.74 - - [30/Nov/2021:08:28:38 +0000] "GET / HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:28:38 +0000] "GET /favicon.ico HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:29:10 +0000] "GET / HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:29:10 +0000] "GET /favicon.ico HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:29:30 +0000] "GET / HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:29:30 +0000] "GET /favicon.ico HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
127.0.0.1 - - [30/Nov/2021:08:29:37 +0000] "POST /api/v4/jobs/request HTTP/1.1" 502 24 "" "gitlab-runner 11.11.2 (; go1.8.7; linux/amd64)" -
193.117.129.74 - - [30/Nov/2021:08:30:25 +0000] "GET / HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:30:25 +0000] "GET /favicon.ico HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
127.0.0.1 - - [30/Nov/2021:08:30:37 +0000] "POST /api/v4/jobs/request HTTP/1.1" 502 24 "" "gitlab-runner 11.11.2 (; go1.8.7; linux/amd64)" -

==> /var/log/gitlab/nginx/gitlab_error.log <==

==> /var/log/gitlab/nginx/access.log <==

==> /var/log/gitlab/nginx/gitlab_access.log <==
193.117.129.74 - - [30/Nov/2021:08:31:02 +0000] "GET / HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:31:02 +0000] "GET /favicon.ico HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:31:11 +0000] "GET / HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
193.117.129.74 - - [30/Nov/2021:08:31:11 +0000] "GET /favicon.ico HTTP/2.0" 502 2940 "https://gitlab.mypolicy.co.uk:9191/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -

node-exporter

[root@telemetry ~]# sudo gitlab-ctl tail node-exporter
==> /var/log/gitlab/node-exporter/state <==

==> /var/log/gitlab/node-exporter/current <==
2021-11-30_08:22:37.35806 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=thermal_zone
2021-11-30_08:22:37.35806 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=time
2021-11-30_08:22:37.35808 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=timex
2021-11-30_08:22:37.35808 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=udp_queues
2021-11-30_08:22:37.35809 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=uname
2021-11-30_08:22:37.35809 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=vmstat
2021-11-30_08:22:37.35809 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=xfs
2021-11-30_08:22:37.35810 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:112 collector=zfs
2021-11-30_08:22:37.35820 level=info ts=2021-11-30T08:22:37.358Z caller=node_exporter.go:191 msg="Listening on" address=localhost:9100
2021-11-30_08:22:37.35822 level=info ts=2021-11-30T08:22:37.358Z caller=tls_config.go:170 msg="TLS is disabled and it cannot be enabled on the fly." http2=false

postgres-exporter

[root@telemetry ~]# sudo gitlab-ctl tail postgres-exporter
==> /var/log/gitlab/postgres-exporter/state <==

==> /var/log/gitlab/postgres-exporter/current <==
2021-11-29_16:14:00.52846 time="2021-11-29T16:14:00Z" level=info msg="Semantic Version Changed on \"/var/opt/gitlab/postgresql:5432\": 0.0.0 -> 12.5.0" source="postgres_exporter.go:1405"
2021-11-29_16:14:00.79395 time="2021-11-29T16:14:00Z" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1672"
2021-11-29_16:16:14.31726 time="2021-11-29T16:16:14Z" level=info msg="Established new database connection to \"/var/opt/gitlab/postgresql:5432\"." source="postgres_exporter.go:878"
2021-11-29_16:16:15.31795 time="2021-11-29T16:16:15Z" level=info msg="Established new database connection to \"/var/opt/gitlab/postgresql:5432\"." source="postgres_exporter.go:878"
2021-11-29_16:16:15.32510 time="2021-11-29T16:16:15Z" level=info msg="Semantic Version Changed on \"/var/opt/gitlab/postgresql:5432\": 0.0.0 -> 12.5.0" source="postgres_exporter.go:1405"
2021-11-29_16:16:15.47123 time="2021-11-29T16:16:15Z" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1672"
2021-11-30_08:22:37.31900 time="2021-11-30T08:22:37Z" level=info msg="Established new database connection to \"/var/opt/gitlab/postgresql:5432\"." source="postgres_exporter.go:878"
2021-11-30_08:22:38.39407 time="2021-11-30T08:22:38Z" level=info msg="Established new database connection to \"/var/opt/gitlab/postgresql:5432\"." source="postgres_exporter.go:878"
2021-11-30_08:22:38.41162 time="2021-11-30T08:22:38Z" level=info msg="Semantic Version Changed on \"/var/opt/gitlab/postgresql:5432\": 0.0.0 -> 12.5.0" source="postgres_exporter.go:1405"
2021-11-30_08:22:38.74087 time="2021-11-30T08:22:38Z" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1672"

postgresql

[root@telemetry ~]# sudo gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/state <==

==> /var/log/gitlab/postgresql/current <==
2021-11-30_08:21:35.83051 LOG:  received fast shutdown request
2021-11-30_08:21:35.83155 LOG:  aborting any active transactions
2021-11-30_08:21:35.83432 LOG:  background worker "logical replication launcher" (PID 7415) exited with exit code 1
2021-11-30_08:21:35.83605 LOG:  shutting down
2021-11-30_08:21:35.87963 LOG:  database system is shut down
2021-11-30_08:22:37.31684 LOG:  starting PostgreSQL 12.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2021-11-30_08:22:37.31995 LOG:  listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2021-11-30_08:22:37.39312 LOG:  database system was shut down at 2021-11-30 08:21:35 GMT
2021-11-30_08:22:37.39318 FATAL:  the database system is starting up
2021-11-30_08:22:37.41975 LOG:  database system is ready to accept connections

prometheus

[root@telemetry ~]# sudo gitlab-ctl tail prometheus
==> /var/log/gitlab/prometheus/state <==

==> /var/log/gitlab/prometheus/current <==
2021-11-30_08:22:39.00144 level=info ts=2021-11-30T08:22:39.001Z caller=main.go:735 msg="TSDB started"
2021-11-30_08:22:39.00146 level=info ts=2021-11-30T08:22:39.001Z caller=main.go:861 msg="Loading configuration file" filename=/var/opt/gitlab/prometheus/prometheus.yml
2021-11-30_08:22:39.00283 level=error ts=2021-11-30T08:22:39.002Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-11-30_08:22:39.00286 level=error ts=2021-11-30T08:22:39.002Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-11-30_08:22:39.00288 level=error ts=2021-11-30T08:22:39.002Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-11-30_08:22:39.01202 level=info ts=2021-11-30T08:22:39.011Z caller=main.go:892 msg="Completed loading of configuration file" filename=/var/opt/gitlab/prometheus/prometheus.yml totalDuration=10.52342ms remote_storage=1.833µs web_handler=674ns query_engine=1.573µs scrape=75.135µs scrape_sd=384.337µs notify=37.381µs notify_sd=23.051µs rules=8.836477ms
2021-11-30_08:22:39.01204 level=info ts=2021-11-30T08:22:39.012Z caller=main.go:684 msg="Server is ready to receive web requests."
2021-11-30_08:22:44.01256 level=error ts=2021-11-30T08:22:44.012Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-nodes
2021-11-30_08:22:44.01259 level=error ts=2021-11-30T08:22:44.012Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-cadvisor
2021-11-30_08:22:44.01264 level=error ts=2021-11-30T08:22:44.012Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-pods

Looking through these logs, I think redis doesn’t appear to be working correctly within gitlab.

Regards,
Mike

14

Here is the rest of the logs (was too much for a single post)

puma

[root@telemetry ~]# sudo gitlab-ctl tail puma
==> /var/log/gitlab/puma/state <==

==> /var/log/gitlab/puma/current <==
2021-11-30_08:33:08.00542 {"timestamp":"2021-11-30T08:33:08.005Z","pid":22548,"message":"* Preloading application"}
2021-11-30_08:33:24.80212 {"timestamp":"2021-11-30T08:33:24.801Z","pid":31559,"message":"Puma starting in cluster mode..."}
2021-11-30_08:33:24.80216 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"* Puma version: 5.1.1 (ruby 2.7.2-p137) (\"At Your Service\")"}
2021-11-30_08:33:24.80221 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"*  Min threads: 4"}
2021-11-30_08:33:24.80224 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"*  Max threads: 4"}
2021-11-30_08:33:24.80226 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"*  Environment: production"}
2021-11-30_08:33:24.80230 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"*   Master PID: 31559"}
2021-11-30_08:33:24.80233 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"*      Workers: 40"}
2021-11-30_08:33:24.80236 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"*     Restarts: (✔) hot (✖) phased"}
2021-11-30_08:33:24.80239 {"timestamp":"2021-11-30T08:33:24.802Z","pid":31559,"message":"* Preloading application"}

==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2021-11-30T03:20:44.030Z","pid":31520,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.030Z","pid":31554,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.030Z","pid":31843,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.031Z","pid":31514,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.031Z","pid":31530,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.031Z","pid":31548,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.031Z","pid":31524,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.033Z","pid":31830,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.034Z","pid":32066,"message":"! Detected parent died, dying"}
{"timestamp":"2021-11-30T03:20:44.035Z","pid":31894,"message":"! Detected parent died, dying"}

==> /var/log/gitlab/puma/puma_stderr.log <==
=== puma startup: 2021-11-30 03:00:37 +0000 ===
=== puma startup: 2021-11-30 03:02:23 +0000 ===
=== puma startup: 2021-11-30 03:07:48 +0000 ===
=== puma startup: 2021-11-30 03:09:38 +0000 ===
=== puma startup: 2021-11-30 03:11:25 +0000 ===
=== puma startup: 2021-11-30 03:13:15 +0000 ===
=== puma startup: 2021-11-30 03:15:01 +0000 ===
=== puma startup: 2021-11-30 03:16:53 +0000 ===
=== puma startup: 2021-11-30 03:18:40 +0000 ===
=== puma startup: 2021-11-30 03:20:25 +0000 ===

==> /var/log/gitlab/puma/current <==
2021-11-30_08:33:42.34939 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"Puma starting in cluster mode..."}
2021-11-30_08:33:42.34952 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"* Puma version: 5.1.1 (ruby 2.7.2-p137) (\"At Your Service\")"}
2021-11-30_08:33:42.34963 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"*  Min threads: 4"}
2021-11-30_08:33:42.34972 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"*  Max threads: 4"}
2021-11-30_08:33:42.34984 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"*  Environment: production"}
2021-11-30_08:33:42.34994 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"*   Master PID: 40819"}
2021-11-30_08:33:42.35003 {"timestamp":"2021-11-30T08:33:42.349Z","pid":40819,"message":"*      Workers: 40"}
2021-11-30_08:33:42.35017 {"timestamp":"2021-11-30T08:33:42.350Z","pid":40819,"message":"*     Restarts: (✔) hot (✖) phased"}
2021-11-30_08:33:42.35026 {"timestamp":"2021-11-30T08:33:42.350Z","pid":40819,"message":"* Preloading application"}
2021-11-30_08:33:59.52531 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"Puma starting in cluster mode..."}
2021-11-30_08:33:59.52545 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"* Puma version: 5.1.1 (ruby 2.7.2-p137) (\"At Your Service\")"}
2021-11-30_08:33:59.52555 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"*  Min threads: 4"}
2021-11-30_08:33:59.52563 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"*  Max threads: 4"}
2021-11-30_08:33:59.52566 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"*  Environment: production"}
2021-11-30_08:33:59.52569 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"*   Master PID: 9538"}
2021-11-30_08:33:59.52572 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"*      Workers: 40"}
2021-11-30_08:33:59.52577 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"*     Restarts: (✔) hot (✖) phased"}
2021-11-30_08:33:59.52580 {"timestamp":"2021-11-30T08:33:59.525Z","pid":9538,"message":"* Preloading application"}

redis

[root@telemetry ~]# sudo gitlab-ctl tail redis
==> /var/log/gitlab/redis/state <==

==> /var/log/gitlab/redis/current <==
2021-11-30_08:22:37.26690 2905:M 30 Nov 2021 08:22:37.266 * Loading RDB produced by version 6.0.10
2021-11-30_08:22:37.26691 2905:M 30 Nov 2021 08:22:37.266 * RDB age 60 seconds
2021-11-30_08:22:37.26693 2905:M 30 Nov 2021 08:22:37.266 * RDB memory usage when created 2.17 Mb
2021-11-30_08:22:37.27515 2905:M 30 Nov 2021 08:22:37.275 * DB loaded from disk: 0.008 seconds
2021-11-30_08:22:37.27516 2905:M 30 Nov 2021 08:22:37.275 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket
2021-11-30_08:31:33.36620 2905:M 30 Nov 2021 08:31:33.365 * 10 changes in 300 seconds. Saving...
2021-11-30_08:31:33.36783 2905:M 30 Nov 2021 08:31:33.366 * Background saving started by pid 29486
2021-11-30_08:31:33.41076 29486:C 30 Nov 2021 08:31:33.409 * DB saved on disk
2021-11-30_08:31:33.41078 29486:C 30 Nov 2021 08:31:33.410 * RDB: 0 MB of memory used by copy-on-write
2021-11-30_08:31:33.46976 2905:M 30 Nov 2021 08:31:33.469 * Background saving terminated with success

redis-exporter

[root@telemetry ~]# sudo gitlab-ctl tail registry
==> /var/log/gitlab/registry/state <==

==> /var/log/gitlab/registry/current <==
2021-11-29_16:16:14.34222 time="2021-11-29T16:16:14Z" level=info msg="listening on 127.0.0.1:5000" environment=production go_version=go1.14.7 instance_id=91f4bbf2-fa07-428d-bb8f-e6c76ae94960 service=registry version=v3.0.0-gitlab
2021-11-29_16:32:14.34094 time="2021-11-29T16:32:14Z" level=info msg="PurgeUploads starting: olderThan=2021-11-22 16:32:14.340164795 +0000 UTC m=-603839.977033963, actuallyDelete=true"
2021-11-29_16:32:14.34734 time="2021-11-29T16:32:14Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=0"
2021-11-29_16:32:14.34738 time="2021-11-29T16:32:14Z" level=info msg="Starting upload purge in 24h0m0s" environment=production go_version=go1.14.7 instance_id=91f4bbf2-fa07-428d-bb8f-e6c76ae94960 service=registry version=v3.0.0-gitlab
2021-11-30_08:21:37.97027 time="2021-11-30T08:21:37Z" level=info msg="attempting to stop server gracefully..." http_drain_timeout=0s quit_signal=terminated
2021-11-30_08:21:37.97066 time="2021-11-30T08:21:37Z" level=info msg="graceful shutdown successful" http_drain_timeout=0s quit_signal=terminated
2021-11-30_08:22:37.42654 time="2021-11-30T08:22:37Z" level=info msg="redis not configured" environment=production go_version=go1.14.7 instance_id=5ebb18d6-31fb-4169-9fb4-eb53e0b3de32 service=registry version=v3.0.0-gitlab
2021-11-30_08:22:37.42664 time="2021-11-30T08:22:37Z" level=info msg="using inmemory blob descriptor cache" environment=production go_version=go1.14.7 instance_id=5ebb18d6-31fb-4169-9fb4-eb53e0b3de32 service=registry version=v3.0.0-gitlab
2021-11-30_08:22:37.42811 time="2021-11-30T08:22:37Z" level=info msg="Starting upload purge in 32m0s" environment=production go_version=go1.14.7 instance_id=5ebb18d6-31fb-4169-9fb4-eb53e0b3de32 service=registry version=v3.0.0-gitlab
2021-11-30_08:22:37.43238 time="2021-11-30T08:22:37Z" level=info msg="listening on 127.0.0.1:5000" environment=production go_version=go1.14.7 instance_id=5ebb18d6-31fb-4169-9fb4-eb53e0b3de32 service=registry version=v3.0.0-gitlab

sidekiq

[root@telemetry ~]# sudo gitlab-ctl tail sidekiq
==> /var/log/gitlab/sidekiq/state <==

==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2021-11-30T08:34:52.997Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-30T08:34:53.325Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2021-11-30T08:35:08.339Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-30T08:35:08.663Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2021-11-30T08:35:23.676Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-30T08:35:24.022Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2021-11-30T08:35:39.035Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-30T08:35:39.349Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2021-11-30T08:35:59.368Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-30T08:35:59.707Z","message":"Starting cluster with 1 processes"}
{"severity":"INFO","time":"2021-11-30T08:36:14.721Z","message":"A worker terminated, shutting down the cluster"}
{"severity":"INFO","time":"2021-11-30T08:36:15.065Z","message":"Starting cluster with 1 processes"}
15

You need to look at all paths in there are remove them. For example:

/data/apache-tomcat/conf/logging.properties 
/dev/./urandom
/data/apache-tomcat/bin/bootstrap.jar
/data/apache-tomcat/bin/tomcat-juli.jar
/data/apache-tomcat 
/data/apache-tomcat
/data/apache-tomcat/temp

in essence delete the entire /data/apache-tomcat directory, or even the entire /data directory unless you are using this for something. Also the hidden .urandom directory under /dev. This is not normal. Normal would be /dev/urandom (eg: not hidden). The /dev/.urandom is hidden and should not exist and therefore this is trying to hide from detection.

Once you have guaranteed that tomcat isn’t running it’s fake processes, only then can you take a look at getting gitlab working properly. You will also have to make sure no other processes are running as the git user - ask here for verification if you are unsure which ones are valid or not.

16

I think I have ran out of replies, as I am a new user. But I tried to get into /data dir yesterday and it’s saying that it doesn’t exist.

17

Hi again,

There is no /data folder which is wierd and it’s /dev/urandom (which is for generating random numbers in cryptography I read online).

Also I am now looking in the /proc/21090 folder for this apache tomcat stuff:

cat cmdline gives (I deleted the folder /usr/java):

/usr/java/latest/bin/java -Djava.util.logging.config.file=/data/apache-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M -Xmx5120M -server -XX:+UseParallelGC -Djavax.net.debug=ssl -Dignore.endorsed.dirs= -classpath /data/apache-tomcat/bin/bootstrap.jar:/data/apache-tomcat/bin/tomcat-juli.jar -Dcatalina.base=/data/apache-tomcat -Dcatalina.home=/data/apache-tomcat -Djava.io.tmpdir=/data/apache-tomcat/temp org.apache.catalina.startup.Bootstrap

cwd is /

fd folder gives

lr-x------. 1 git git 64 Nov 30 10:55 0 -> /dev/null
l-wx------. 1 git git 64 Nov 30 10:55 1 -> /dev/null
lr-x------. 1 git git 64 Nov 30 10:55 10 -> /dev/null
lrwx------. 1 git git 64 Nov 30 11:00 11 -> socket:[2757606308]
lrwx------. 1 git git 64 Nov 30 10:55 2 -> anon_inode:[eventpoll]
lr-x------. 1 git git 64 Nov 30 10:55 3 -> pipe:[2757374680]
l-wx------. 1 git git 64 Nov 30 10:55 4 -> pipe:[2757374680]
lr-x------. 1 git git 64 Nov 30 10:55 5 -> pipe:[2757197746]
l-wx------. 1 git git 64 Nov 30 10:55 6 -> pipe:[2757197746]
lrwx------. 1 git git 64 Nov 30 10:55 7 -> anon_inode:[eventfd]
lrwx------. 1 git git 64 Nov 30 11:00 8 -> anon_inode:[eventfd]
lrwx------. 1 git git 64 Nov 30 10:55 9 -> anon_inode:[eventfd]

cat environ gives us

=/mnt/tm8node3/archive/sxhvp/cqoicqtegpXDG_SESSION_ID=1128023SHELL=/bin/shUSER=gitPATH=/usr/bin:/binPWD=/var/opt/gitlabLANG=en_US.UTF-8SHLVL=1HOME=/var/opt/gitlabLOGNAME=gitXDG_RUNTIME_DIR=/run/user/994_=/mnt/tm8node3/archive/sxhvp/cqoicqtegpXW_DAEMON_IDX=1

and root is /

This tm8node3 archive is one of our servers which is mounted, I have looked and this process (the tomcat one) was creating these folders but with no files in them on the mounted server i.e. /mnt/tm8node3/archive/sxhvp/cqoic and this started happening on the 24/11

Me and a colleague have gone to that server and changed permissions so that the ubuntu user that it’s mounted on cannot write to this archive folder above. But the process still keeps starting itself and consuming alot of CPU cores under the git user.

Hope this makes sense to you.

image
image1054×960 61.5 KB

Regards,
Mike

18

Looking into this process more we have:

[root@telemetry rc.d]# ps j 21090
PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND
1 21090 21090 21090 ? -1 Ssl 994 754:48 /usr/java/latest/bin/ja
[root@telemetry rc.d]# pstree -sg 21090
systemd(1)───8qwz9rsc9vw6yqm(21090)─┬─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
├─{8qwz9rsc9vw6yqm}(21090)
└─{8qwz9rsc9vw6yqm}(21090)

19

Hey,

Gitlab has been working today after periodically killing those processes, then I have been trying to run some rake commands, before it all goes offline again.

The results of sudo gitlab-rake gitlab:check:

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.16.1 ? ... OK (13.16.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
<<ALL SAY YES>>
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... no
  Try fixing it:
  mkdir ~/gitlab-check-backup-1638280384
  sudo mv /var/opt/gitlab/.ssh/authorized ~/gitlab-check-backup-1638280384
  For more information see:
  doc/ssh/README.md in section "SSH on the GitLab server"
  Please fix the error above and rerun the checks.
Active users: ... 25
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
20

I resolved the above SSH config error and made it default. By following the commands listed above.