Can't load changes/discussions on merge requests of specific project

How to Use GitLab
1

We’ve been using GitLab for quite some time now, but recently we’re facing some problems on a specific project.

Sometimes on specific merge requests, we’re not able to load the content on the changes tab, neither on the discussion tab.

We already increased the following parameters on gitlab.rb:

unicorn[‘worker_timeout’] = 120
unicorn[‘worker_processes’] = 9 #// We’re running this on a 8-core 16gb VM.
unicorn[‘worker_memory_limit_min’] = 2048*(1024 ** 2)
unicorn[‘worker_memory_limit_max’] = 4096*(1024 ** 2)
nginx[‘keepalive_timeout’] = 120

With the above config, we’ve managed to decrease this problem by 50%, but still happens on a lot of merge requests.

The weird thing is that when we try to call the merge requests’ changes and dicussions trough the API it answes back instantly, but trough the GitLab UI it keeps loading until 502 Timeout.

We also already tried to increase the unicorn and nginx timeout to 10min, just to check what would happen, but still, it keeps loading the discussions/changes until times out.

For now, the workaround is whenever this happens to a merge, close and re-open the merge request.

We’re running:

GitLab 11.3.0 (17bd59a)
GitLab Shell 8.3.3
GitLab Workhorse v6.1.0
GitLab API v4
Ruby 2.4.4p296
Rails 4.2.10
postgresql 9.6.10

2

The error message you provided indicates that GitLab 14.10 introduced a change where applying the noexec option to a mount point, such as /var, can cause permission denied errors when forking processes in Gitaly. To resolve this issue, you can either remove the noexec option from the file system mount or change the Gitaly runtime directory to a location without the noexec option set.

Here are the steps to resolve this issue:

Option 1: Remove the noexec Option from the File System Mount (Recommended if possible):

  1. Open the file system mount configuration file. The location of this file may vary depending on your Linux distribution. Common locations include /etc/fstab or a file in the /etc/fstab.d/ directory.

  2. Locate the line that specifies the mount point (e.g., /var) and includes the noexec option. It might look something like this:

    /dev/sda1 /var ext4 defaults,noexec 0 0

  3. Remove the noexec option from the line, so it looks like this:

    /dev/sda1 /var ext4 defaults 0 0

  4. Save the changes to the configuration file.

  5. After saving the changes, remount the file system or reboot your server for the changes to take effect:

    sudo mount -o remount /var

Option 2: Change the Gitaly Runtime Directory (Use if Option 1 is not possible):

  1. Edit the GitLab configuration file /etc/gitlab/gitlab.rb:

    sudo nano /etc/gitlab/gitlab.rb

  2. Add the following line to specify a Gitaly runtime directory without the noexec option set. Replace <PATH_WITH_EXEC_PERM> with a suitable directory path:

    gitaly['runtime_dir'] = '<PATH_WITH_EXEC_PERM>'

    For example:

    gitaly['runtime_dir'] = '/opt/gitlab-gitaly'

  3. Save the changes to the configuration file.

  4. Reconfigure GitLab to apply the new configuration:

    sudo gitlab-ctl reconfigure

By following one of these options, you should be able to resolve the “permission denied” errors related to forking processes in Gitaly caused by the noexec option on the mount point.