After upgrade EE 9.5.1 -> 10.0, creating a merge request gives 503

How to Use GitLab
1

On ubuntu 16.04, after upgrading gitlab Enterprise Edition from 9.5.1 -> 10.0, clicking on some existing merge requests returns a 503 and attempts to create new merge requests return 503’s.

Running gitlab-ctl tail and trying to create a merge request, I see this in the logs:


==> /var/log/gitlab/gitlab-rails/production.log <==

GRPC::Unavailable (14:Connect Failed):
  lib/gitlab/gitaly_client.rb:58:in `call'
  lib/gitlab/gitaly_client/repository_service.rb:13:in `exists?'
  lib/gitlab/git/repository.rb:75:in `exists?'
  lib/gitlab/git/repository.rb:884:in `empty_repo?'
  lib/gitlab/git/repository.rb:803:in `with_repo_branch_commit'
  lib/gitlab/git/repository.rb:844:in `compare_source_branch'
  app/models/repository.rb:1111:in `compare_source_branch'
  app/services/compare_service.rb:14:in `execute'
  app/services/merge_requests/build_service.rb:59:in `compare_branches'
  app/services/merge_requests/build_service.rb:13:in `execute'
  app/controllers/projects/merge_requests/creations_controller.rb:99:in `build_merge_request'
  lib/gitlab/i18n.rb:46:in `with_locale'
  lib/gitlab/i18n.rb:52:in `with_user_locale'
  app/controllers/application_controller.rb:334:in `set_locale'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/jira/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/readonly_geo.rb:30:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'

Completed 503 Service Unavailable in 32ms (Views: 4.9ms | ActiveRecord: 2.2ms | Elasticsearch: 0.0ms)
1 Like
2

This issue is caused by the tinyproxy service we are running on our host. We run tiny proxy because gitlab doesn’t honor/have a no_proxy setting we can configure.

GRPC requests were also being sent through tinyproxy which resulted in the errors listed above.

3

We have this problem as well. Even after upgrade 10.0.1. Any workarounds?

4

We have the problem here as well. It is a default Omnibus installation (EE).

5

Same issue here, reverted gitlab-ce to version 9.5.5. Merge requests work again.

6

Same here :frowning:

@aberkel it is possible to revert without restoring data from a backup?

7

We first created a backup of gitlab ofcourse. But for now don’t have any issues. When reverting always be sure you can get the situation back from where you start. Didn’t have any issues during downgrade.

8

I think gitlab always makes a backup before upgrade.

9

In the meantime we’ve been using gitlab without noticing this issue, so our backup is outdated by now unfortunately.

1 Like
10

Gitlab is dumping databases while upgrading/downgrading yes. But to make it easyer for you… create a machine backup if it’s virtual. We didn’t restore anything after downgrading.

11

i created an issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/38333

1 Like
12

I think we are running into the same issue. However, this only affects some merge requests – can this really be caused by a proxy misbehavior? Naively, I would expect a proxy to either work for all merge requests or break them all.

13

We had to completely disable any local proxying in order to get around this issue. This only works for us because we no longer need the slack integrations which required a proxy.

14

We have this problem as well. Even after upgrade 10.0.2. (merge requests, editing files online)