How to troubleshoot time synchronization on gitlab pods

Community
1

Hi everyone,

Firstly, very new to Devops/kubernetes. I inherited a GitLab installation and have had maybe 3 months using it now and I must say I love it!

I am learning as much as I can and today I had an issue with GitLab for the first time and I am here to find out how best to diagnose the problem.

Components of my install:
GitLab 4.10.2-ee
GitLab Shell13.25.1
GitLab Workhorsev14.10.2
GitLab APIv4
Ruby2.7.5p203
Rails6.1.4.7
PostgreSQL12.7
Redis6.0.9

Symptoms

  1. 500 errors pages
  2. The odd 502 error page
  3. Messages like: “An error occurred while fetching folder content.”
  4. There was an error fetching the pipelines. Try again in a few moments or contact your support team.

etc.

What I have done thus far
I performed a rolling restart of all the gitlab pods in the hope this would resolve the issue.

> NAME                                                     READY   STATUS      RESTARTS   AGE
> gitlab-production-gitaly-0                               1/1     Running     0          157d
> gitlab-production-gitlab-exporter-7fcb888456-sctvg       1/1     Running     0          29m
> gitlab-production-gitlab-runner-96fb7757c-2q5xx          1/1     Running     0          31m
> gitlab-production-gitlab-shell-b967f9c55-bv9tx           1/1     Running     0          32m
> gitlab-production-gitlab-shell-b967f9c55-gssxx           1/1     Running     0          33m
> gitlab-production-grafana-app-78c5f499f4-mrk5d           2/2     Running     0          34m
> gitlab-production-mailroom-877c44b66-w5qbx               1/1     Running     0          35m
> gitlab-production-migrations-39-wmk7n                    0/1     Completed   0          157d
> gitlab-production-postgresql-0                           2/2     Running     0          204d
> gitlab-production-prometheus-server-6fdc878559-hh7gj     2/2     Running     0          36m
> gitlab-production-redis-master-0                         2/2     Running     1          204d
> gitlab-production-registry-86d4b5d5c4-2zvm2              1/1     Running     0          38m
> gitlab-production-registry-86d4b5d5c4-btx7h              1/1     Running     0          38m
> gitlab-production-sidekiq-all-in-1-v2-7c5b987987-7vk4g   1/1     Running     0          39m
> gitlab-production-toolbox-767fb64d66-kr6wv               1/1     Running     0          41m
> gitlab-production-webservice-default-f6d4c87bf-d4nlm     2/2     Running     0          21m
> gitlab-production-webservice-default-f6d4c87bf-wqnql     2/2     Running     7          46m

I was unsure how to restart some of them as you can see by the age.

I had a look at the logs by performing the following command:
kubectl logs deploy/gitlab-production-webservice-default -n gitlab-production --tail=2 -f

But there were no logs when the errors occured.

I checked disk space
Which seemed fine.

Q: Does the community have any advise/guidelines in terms of how to go about troubleshooting the installation?

2

The issue:
Turns out that our one nodes time was out of sync and this was causing the headache.
The solution
To solve the problem, I installed ntp on each of our nodes and synced them to a common time zone.
The outcome
The second I did this, everything started to work normally again.

This post pointed me in the right direction: gRPC permission denied errors that lead to unusable gitlab instance (#3084) · Issues · GitLab.org / gitaly · GitLab
Used this post for sync: How To Set Up Time Synchronization on Ubuntu 18.04 | DigitalOcean

1 Like
3

Glad you were able to find the solution @chateaux - thanks for sharing it with others who might come across your post :tada: