Mattermost broken after omnibus upgrade to 18.3

How to Use GitLab Upgrade
1

Problem to solve

After upgrading from 18.2 to 18.3 on a Rocky Linux 8 system, Mattermost no longer starts.

Several attempts with gitlab-ctl reconfigure and gitlab-ctl restart vs. gitlab-ctl stop|start failed. The result of gitlab-ctl status mattermostremains the same:

down: mattermost: 0s, normally up, want up; run: log: (pid 1505) 6764448s

After checking the log file if found recurring entries like this:

2025-08-22_04:56:58.77632 /opt/gitlab/embedded/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/gitlab/embedded/bin/mattermost)
2025-08-22_04:56:59.78278 /opt/gitlab/embedded/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/gitlab/embedded/bin/mattermost)
2025-08-22_04:56:59.78284 /opt/gitlab/embedded/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/gitlab/embedded/bin/mattermost)
2025-08-22_04:57:00.78758 /opt/gitlab/embedded/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/gitlab/embedded/bin/mattermost)

I was able to confirm this with ldd:

[redacted@redacted ~]$ ldd /opt/gitlab/embedded/bin/mattermost
/opt/gitlab/embedded/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /opt/gitlab/embedded/bin/mattermost)
/opt/gitlab/embedded/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /opt/gitlab/embedded/bin/mattermost)
	linux-vdso.so.1 (0x00007fffa64d3000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fd6bf530000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd6bf907000)

All other GitLab services are working as expected.

Steps to reproduce

dnf upgrade gitlab-ce

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated

Versions

  • GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info):
System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	3.2.8
Gem Version:	3.6.9
Bundler Version:2.7.1
Rake Version:	13.0.6
Redis Version:	7.2.10
Sidekiq Version:7.3.9
Go Version:	unknown

GitLab information
Version:	18.3.0
Revision:	f54d72cc517
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	16.8
URL:		https://xxx.xxx.xxx
HTTP Clone URL:	https://xxx.xxx.xxx/some-group/some-project.git
SSH Clone URL:	git@xxx.xxx.xxx:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	14.44.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	18.3.0
- default Git Version: 	2.50.1.gl1

2

Same here. When I run gitlab-ctl reconfigure` I’m also seeing the following warning that’s probably related:

Recipe: mattermost::enable
  * account[Mattermost user and group] action create
    * group[Mattermost user and group] action create (up to date)
    * linux_user[Mattermost user and group] action create (up to date)
     (up to date)
  * directory[/var/opt/gitlab/mattermost] action create (up to date)
  * directory[/var/opt/gitlab/mattermost/data] action create (up to date)
  * directory[/var/opt/gitlab/mattermost/plugins] action create (up to date)
  * directory[/var/opt/gitlab/mattermost/client-plugins] action create (up to date)
  * directory[/var/log/gitlab/mattermost] action create (up to date)
  * file[/var/log/gitlab/mattermost/mattermost.log] action create (up to date)
  * postgresql_user[gitlab_mattermost] action create
    * execute[create gitlab_mattermost postgresql user] action run (skipped due to not_if)
     (up to date)
  * execute[create mattermost_production database] action run (skipped due to not_if)
  * ruby_block[authorize mattermost with gitlab] action run[2025-08-22T09:36:31+02:00] WARN: Connecting to GitLab to generate new app_id and app_secret for GitLab Mattermost.
[2025-08-22T09:36:54+02:00] WARN: Something went wrong while trying to update gitlab-secrets.json. Check the file permissions and try reconfiguring again.
[2025-08-22T09:36:54+02:00] INFO: ruby_block[authorize mattermost with gitlab] called

    - execute the ruby block authorize mattermost with gitlab
  * ruby_block[populate mattermost configuration options] action run[2025-08-22T09:36:54+02:00] INFO: ruby_block[populate mattermost configuration options] called

    - execute the ruby block populate mattermost configuration options
  * remote_file[/var/opt/gitlab/mattermost/config.json] action create_if_missing (up to date)
  * env_dir[/opt/gitlab/etc/mattermost/env] action create
3

One more thing: alertmanager isn’t starting either. That never happened before in our case and I’m therefore wondering if it has to do with it. The alertmanager log shows things like this:

2025-08-22_09:13:42.87192 time=2025-08-22T09:13:42.871Z level=WARN source=cluster.go:178 msg="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" component=cluster
2025-08-22_09:13:42.87374 time=2025-08-22T09:13:42.873Z level=ERROR source=main.go:259 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
4

Here is an issue with the same topic: Missing or wrong dependency for Mattermost in GitLab 18.3.0 (#9267) · Issues · GitLab.org / omnibus-gitlab · GitLab

It also contains a suggested workaround, will give that a try.

EDIT: The described workaround fixes the issue.

1 Like