503 when trying to view branches on gitlab UI or delete branch (from UI or git client)

Hi Guys, We have self hosted gitlab ce 11.6.2 and we are in a strange situation.

One of the projects is having issues to load /branches or delete branch in their project. The project has reached around 2GB. I can see these logs in production.log

Get all branches:

    Started GET "/abstesting/abstesting/branches" for 127.0.0.1 at 2020-11-05 20:54:19 +0000
    Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2020-11-05 20:54:19 +0000
    Processing by Projects::BranchesController#index as HTML
    Parameters: {"namespace_id"=>"abstesting", "project_id"=>"abstesting"}
    Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2020-11-05 20:54:20 +0000
    .
    .
    .
    Gitlab::Git::CommandError (4:Deadline Exceeded):
    lib/gitlab/git/wraps_gitaly_errors.rb:13:in `rescue in wrapped_gitaly_errors'
    lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
    lib/gitlab/git/repository.rb:419:in `merged_branch_names'
    app/models/repository.rb:910:in `merged_branch_names'
    lib/gitlab/metrics/instrumentation.rb:161:in `block in merged_branch_names'
    lib/gitlab/metrics/method_call.rb:36:in `measure'
    lib/gitlab/metrics/instrumentation.rb:161:in `merged_branch_names'
    app/controllers/projects/branches_controller.rb:26:in `block (2 levels) in index'
    app/controllers/projects/branches_controller.rb:16:in `index'
    lib/gitlab/i18n.rb:55:in `with_locale'
    lib/gitlab/i18n.rb:61:in `with_user_locale'
    app/controllers/application_controller.rb:420:in `set_locale'
    lib/gitlab/middleware/multipart.rb:103:in `call'
    lib/gitlab/request_profiler/middleware.rb:16:in `call'
    lib/gitlab/middleware/go.rb:19:in `call'
    lib/gitlab/etag_caching/middleware.rb:13:in `call'
    lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
    lib/gitlab/correlation_id.rb:15:in `use_id'
    lib/gitlab/middleware/correlation_id.rb:15:in `call'
    lib/gitlab/middleware/rails_queue_duration.rb:24:in `call'
    lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
    lib/gitlab/metrics/transaction.rb:55:in `run'
    lib/gitlab/metrics/rack_middleware.rb:17:in `call'
    lib/gitlab/middleware/read_only/controller.rb:40:in `call'
    lib/gitlab/middleware/read_only.rb:18:in `call'
    lib/gitlab/middleware/basic_health_check.rb:25:in `call'
    lib/gitlab/request_context.rb:20:in `call'
    lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
    lib/gitlab/middleware/release_env.rb:13:in `call'
    Completed 503 Service Unavailable in 11445ms (Views: 9.7ms | ActiveRecord: 85.7ms)

To add more info, we also have around 700 branches! and we are not even being able to delete the branch, it also gives us 503!

Can someone please shed some light as to what could be the issue. I also tried to see an alternative to delete a branch from git server itself using git-rake but I couldn’t find anything in the docs to perform that action.