Puma crashes after the update to gitlab-ce/bionic,now 14.2.0-ce.0 amd64 tonight

After tonight’s update I couldn’t log in to my self hosted omnibus Gitlab-ce on Ubuntu 18.04 and I saw an unusal high load. Debugging this, I see puma crashing and trying to restart over and over again (from /var/log/gitlab/puma/puma_stderr.log

=== puma startup: 2021-08-23 14:33:46 +0200 ===
[BUG] Illegal instruction at 0x00007f543c0e6ba3
ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0001 p:---- s:0003 e:000002 (none) [FINISH]


-- Machine register context ------------------------------------------------
 RIP: 0x00007f543c0e6ba3 RBP: 0x0000000000005a2c RSP: 0x00007f5412dfb500
 RAX: 0x00000000612395b8 RBX: 0x00007f543c5afa00 RCX: 0x0000000100000000
 RDX: 0x0000000135f9927e RDI: 0x00000000612395b8 RSI: 0x00000000d7e649f8
  R8: 0x0000000035f9927e  R9: 0x0000000000000000 R10: 0x7fffffffffffffff
 R11: 0x80000000612395b7 R12: 0x7fffffffffffffff R13: 0x0000000013fb584f
 R14: 0x00007f543c5afad8 R15: 0x00007f543c5afae8 EFL: 0x0000000000010212

-- C level backtrace information -------------------------------------------
/opt/gitlab/embedded/lib/libruby.so.2.7(rb_vm_bugreport+0x79f) [0x7f545c8705ff] vm_dump.c:755
/opt/gitlab/embedded/lib/libruby.so.2.7(rb_bug_for_fatal_signal+0xdd) [0x7f545c690f2d] error.c:660
/opt/gitlab/embedded/lib/libruby.so.2.7(sigill+0x44) [0x7f545c7d4314] signal.c:962
/lib/x86_64-linux-gnu/libc.so.6(0x7f545c215040) [0x7f545c215040]
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/2.7/grpc_c.so(0x7f543c0e6ba3) [0x7f543c0e6ba3]
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/2.7/grpc_c.so(0x7f543c0e8809) [0x7f543c0e8809]
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/2.7/grpc_c.so(0x7f543c0bf563) [0x7f543c0bf563]
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/2.7/grpc_c.so(0x7f543c2531e3) [0x7f543c2531e3]
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/grpc-1.38.0-x86_64-linux/src/ruby/lib/grpc/2.7/grpc_c.so(0x7f543c0c1298) [0x7f543c0c1298]
/lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb) [0x7f545b47b6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f545c2f771f]

-- Other runtime information -----------------------------------------------

all in all 12.000 lines of output per crash.
Tried a sudo gitlab-ctl stop / reconfigure /start, but no suceess.
Anybody else seing this issue and ideas how to fix this?

There is meanwhile an issue for this problem: puma fails to start with illegal instructions (#339110) · Issues · GitLab.org / GitLab · GitLab

1 Like