502 when I goto create a project Gitaly looks responisble?

Looking at the logs of Gitaly they look like this.

$
{"level":"info","msg":"7fbcb53b2000-7fbcb53b6000 r--s 00000000 fd:0f 262612                     /opt/gitlab/embedded/bin/ruby","supervisor.args":["bundle","exec","bin/ruby-cd",$
{"level":"info","msg":"7fbcb53b6000-7fbcb53b8000 rw-p 00000000 00:00 0 ","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service$
{"level":"info","msg":"7fbcb53b8000-7fbcb53b9000 r--p 00027000 fd:0f 787056                     /lib/x86_64-linux-gnu/ld-2.27.so","supervisor.args":["bundle","exec","bin/ruby-c$
{"level":"info","msg":"7fbcb53b9000-7fbcb53ba000 rw-p 00028000 fd:0f 787056                     /lib/x86_64-linux-gnu/ld-2.27.so","supervisor.args":["bundle","exec","bin/ruby-c$
{"level":"info","msg":"7fbcb53ba000-7fbcb53bb000 rw-p 00000000 00:00 0 ","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service$
{"level":"info","msg":"7ffd65672000-7ffd66071000 rw-p 00000000 00:00 0                          [stack]","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gital$
{"level":"info","msg":"7ffd66184000-7ffd66187000 r--p 00000000 00:00 0                          [vvar]","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly$
{"level":"info","msg":"7ffd66187000-7ffd66188000 r-xp 00000000 00:00 0                          [vdso]","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly$
{"level":"info","msg":"ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gi$
{"error":"signal: aborted","level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/$
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","6976","/var/opt/g$
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","6976","/var/opt/g$
{"level":"warning","msg":"opening circuit breaker","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gital$
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/rub$
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/rub$
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/rub$
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/rub$
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/rub$
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/rub$

Its a refresh install of gitlab-ce and havent changed any of the settings. I’m not using a DNS but an IP on our local network to connect. When I do ‘gitlab-ctrl status’ it all states its okay.

But when I do ‘gitlab-rake gitlab:check’ i get this.


Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:49: [BUG] Illegal instruction at 0x00007fb7f5c67ba3
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0066 p:---- s:0367 e:000366 CFUNC  :initialize
c:0065 p:---- s:0364 e:000363 CFUNC  :new
c:0064 p:0205 s:0357 e:000356 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:49
c:0063 p:0039 s:0349 e:000348 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:104
c:0062 p:0016 s:0336 e:000335 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:158 [FINISH]

How can I fix this?

Whats the core issue?

Thanks :slight_smile:

1 Like

We are facing the same issue here.

We are able to pull commits from an existing repository but unable to push new commits to our server.

If we manually enter the appropriate Url, we can access the various “GitLab Issues” of a project. However, as soon as the Gitlab page consulted involves accessing the files of the repository, the page fails immediately with Error502.

It looks like what makes the interconnection between Gitlab and the actual git repository which contains the files, tags, branches etc is broken. That would explain why we are still able to pull existing commits from our repository but when pushing new ones we obtain the following error message:

My assumption is that Git checks with Gitlab for specific rules to enforce (on some branches only specified users may be able to push …). But with the interconnection between the two broken it suffers an internal failure and fails with the following message:

user@desktop:~/my_project/$ git push
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 4 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (7/7), 143.40 KiB | 5.12 MiB/s, done.
Total 7 (delta 5), reused 0 (delta 0)
remote: GitLab: Internal API error (502)
To <server url>:<gitlab group>/<repository name>.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@<server url>:<gitlab group>/<repository name>.git'

What does your
gitlab-rake gitlab:check
Output? We are running in a VM/VE are you guys?

Output of gitlab-rake gitlab:check was the following:

[...]
Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/zeitwerk-2.4.2/lib/zeitwerk/kernel.rb:26: [BUG] Illegal instruction at 0x00007f76710beba3
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

Then we re-installed the zeitwerk package manually and it became:

[...]
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:367: [BUG] Illegal instruction at 0x00007f79bf84eba3
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0063 p:---- s:0350 e:000349 CFUNC  :run_batch
c:0062 p:0088 s:0345 e:000344 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:367
c:0061 p:0009 s:0336 e:000335 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:180
c:0060 p:0013 s:0333 e:000332 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170
c:0059 p:0093 s:0326 e:000325 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:179

We are using a Ubuntu v18.04 VM. It looks like we are both in very similar environments.

Yeah. Gitlab Does NOT like using Virtual Enviroments & Virtual Machines it seems. I was using Proxmox as the VE with a Ubuntu 18.04 & and even tried a 20.04 the 20.04 really hated it.

What VM/VE software you using?

Proxmox is free and it has prebuilt container with Gitlab in if you need it.

I run Gitlab in a VM on Debian and have even upgraded between distros in the mean time (from Debian 9 to Debian 10 and upgrading Gitlab accordingly), so I don’t believe the situation is that Gitlab doesn’t like virtual environments. There must be something unique to your installation. I’ve ran Gitlab in VMware as well as on KVM environments as well as Openstack at OVH (also most likely based on KVM). Proxmox is also based on KVM unless you are using containers. But containers could be causing other issues.

You mentioned it was a fresh install. What changes did you make to /etc/gitlab/gitlab.rb? Perhaps that can help us figure out where the problem might be. Are you also configured for https on the external_url? Or just http with the IP address of the server?

Also, are you using a standard VM with an Ubuntu install? Or a container? Perhaps something with the container configuration (if using LXC), might be causing some of the instability issues. If so, perhaps try as a standard VM that utilises KVM instead. But let’s take a look at your config changes done during install, and we can then see from there.

You can try:

gitlab-ctl restart

I also find somethings when something isn’t working right to also do this:

systemctl restart gitlab-runsvdir

since some issues can be caused by this needing a restart. Or even a reboot can help. Some instability, especially after an upgrade, can be fixed by running reconfigure, so:

gitlab-ctl reconfigure
gitlab-ctl restart

let us know how you get on.

I used both a Container and started VM. The config change i did was set my IP I didnt use SSL nor DNS. In the end my manager did it. I’m not sure how but it took him most of the day. I think we downgraded in the end.

I did Gitlab reconfigure after all gitlab.rb changes.

It was strange I will ask him how he got it working after the weekend.

Would be good to know how, just in case someone else experiences it, it can help others here. For mine I use an internally accessible host.domain.name and pretty much a self-signed certficate for SSL - generated by an internal CA on Linux - so that the certificate is the cert + CA cert, and key. Another one I have is publicly available, so in that instance it uses LetsEncrypt for it’s certificate. But perfectly fine to use it without SSL if you prefer. Some options :slight_smile:

I’m getting the same issue here with the omnibus packages (CE) on Ubuntu Focal, which I’m assuming has been occurring since the latest update. However, I haven’t used GitLab for a couple of weeks so it could’ve been the 14.2.0 update - I’m currently downgrading to 14.2.0 to check, will update when done.

I agree with iwalker, if you do have the details of the fix this would be greatly appreciated.

I can confirm that downgrading to 14.2.0 didn’t help, for reference this the error snippet output from gitlab-rake gitlab:status

/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:367: [BUG] Illegal instruction at 0x00007fd93ef32ba3
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0063 p:---- s:0350 e:000349 CFUNC  :run_batch
c:0062 p:0088 s:0345 e:000344 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:367
c:0061 p:0009 s:0336 e:000335 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:180
c:0060 p:0013 s:0333 e:000332 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/interceptors.rb:170
c:0059 p:0093 s:0326 e:000325 METHOD /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:179
c:0058 p:0067 s:0308 e:000307 BLOCK  /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:171 [FINISH]

Further investigation on the above error looks like this is a known issue being looked at by the gRPC team: gRPC Ruby client crashes with illegal instruction on older AMD processors · Issue #27095 · grpc/grpc · GitHub

I can also confirm that GitLab is running on a KVM VM but the physical server has an AMD Opteron 4180

Workaround

For anyone experiencing this issue, the work around that worked for me was downgrading the omnibus package on Ubuntu Focal to 14.1.3 using the following commands:

sudo apt install gitlab-ce=14.1.3-ce.0

# Optional
echo "gitlab-ce hold" | sudo dpkg --set-selections

Note: The last command stops apt deploying updates for gitlab-ce, which I needed due to having automatic update routines in place.

3 Likes