Grafana for Gitaly after GitLab switches to Gitaly server

Hi everyone,

I’m using the out-of-the box Grafana that came with self-installation of GitLab (omnibus). The boards include one for Gitaly (name=‘GitLab Omnibus - Gitaly’).

For Gitaly, I previously had the repository stored locally on the same main GitLab server. Since this is deprecated for version 15. While on 14.9.x, I’ve switched from the local Gitaly to its separate Gitlay server ( run-gitaly-on-its-own-server ).

After the switch, general functionality such as repository reads are fine. But the Gitaly data for from the default Grafana-Gitaly board no longer shows. I’m not sure where to configure to point to the new data source. Any help is appreciated.

Currently using default; so have not tinkered with any of grafana settings: /var/opt/gitlab/grafana/grafana.ini

Current setup:
myGitlab_server (grafana is running)
myGitaly_server (Gitaly server’s self grafana is disabled, as of now)

Version: Gitlab-ce 14.9.2 (omnibus, self-installation).

I believe the JSON of the Gitaly Grafana JSON is from here:

  /opt/gitlab/embedded/service/grafana-dashboards/gitaly.json

From [ GitLab - Gitaly dashboard for Grafana | Grafana Labs ] came as part of the default install.

Thanks in advance,

Adding additional details.

On the main GitLab server: (/etc/gitlab/gitlab.rb), where both Prometheus and Grafana also lives, I have the following ‘custom config’ to include the remote Gitaly server’s IP (11.22.33.44)

 gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '::1/128','11.22.33.44']
 prometheus['scrape_configs'] = [
   {
     'job_name': 'gitaly',
     'static_configs' => [
       'targets' => ['11.22.33.44:9236'],
     ],
   },
 ]

On main gitlab’s
[ /var/opt/gitlab/prometheus/prometheus.yml ] I can see the config came through from the Omnibus reconfigure.

scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090

...

- job_name: gitaly
  static_configs:
  - targets:
    - 11.22.33.44:9236

It looks similar to the example here: Monitoring GitLab with Prometheus | GitLab 's step 10’s scrape configs where with its job name and targets.

=====================================================================
Then on the Gitaly server itself (/etc/gitlab/gitlab.rb). I have the prometheus_address set with main GitLab’s server’s IP address. Which I’m guessing it’s supposed to “listen” to the main monitoring server IP (77.77.77.77)

# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
puma['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
grafana['enable'] = false
gitlab_exporter['enable'] = true
gitlab_kas['enable'] = false

# If you run a separate monitoring node you can disable these services.
prometheus['enable'] = false
alertmanager['enable'] = false

gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '::1/128','77.77.77.77']
gitlab_rails['prometheus_address'] = '77.77.77.77:9090'

I tried toggle the Gitaly box’s prometheus on/off; didn’t make much of a difference.

Was modeling it after [ Monitoring GitLab with Prometheus | GitLab ]

" Step 6. On all GitLab Rails(Puma, Sidekiq) servers, set the Prometheus server IP address and listen port.

For:
gitlab_rails[‘prometheus_address’] = ‘77.77.77.77:9090’
"

Not sure if these steps are correct or if there are additional steps. When launching Grafana and going to GitLab’s default dashboards: the Gitaly dashboard still no data coming in.

thanks in advance,