API call 502s to on-premises Gitlab Enterprise Edition cluster - rate limiting?

Hi all,

Our Gitlab setup is an on-premises Enterprise Edition cluster (1 primary box, 2 boxes dedicated to processing Sidekiq queues) running version 11.10.4-ee.

I’ve built a small application to process some analytics from Gitlab. This is written in Python, using the Python Gitlab API library (https://github.com/python-gitlab/python-gitlab).

Each individual component works on its own. I have tested these locally (using docker-compose) and in our Kubernetes cluster. If I SSH on to the server, I can see the API calls in the Nginx logs, all returning 200 responses.

Now that I have a few components on the Kubernetes cluster running on different schedules, I’m noticing an increasing number of 502 responses in the Kubernetes and Nginx logs.

I think this must be some kind of rate limiting. So my main question is this: is there a setting somewhere in gitlab.rb which enforces rate limiting for API calls?

I can see these two:

gitlab_rails['rate_limit_requests_per_period'] = 250
gitlab_rails['rate_limit_period'] = 60

But I don’t think my applications are making 250 calls per minute, so I’m not sure if those settings are active.

Two follow-up questions:

  • Is there a way to see if these settings are currently active on Gitlab?
  • If I make a change to these settings, how do I make them take effect with as little downtime as possible?

Thanks in advance