401 Unauthorized on push after update 15.4 to 15.8.3 (omnibus)

After upgrading gitlab-ce (omnibus) 15.4 to 15.8.3 the following error occurs in all projects when doing push or merge only.

git push origin master
Numbering objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 291 bytes | 291.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: GitLab: 401 Unauthorized
To gitlab.xxxxx:IT-DEVOPS/aws-stack.git
! [remote rejected] master → master (pre-receive hook declined)
error: Pushing some references to ‘gitlab.xxxxx:IT-DEVOPS/aws-stack.git’ failed

:~# gitlab-ctl status
run: alertmanager: (pid 4904) 1903s; run: log: (pid 4887) 1903s
run: gitaly: (pid 4860) 1903s; run: log: (pid 4857) 1903s
run: gitlab-exporter: (pid 4903) 1903s; run: log: (pid 4886) 1903s
run: gitlab-kas: (pid 4863) 1903s; run: log: (pid 4859) 1903s
run: gitlab-workhorse: (pid 4898) 1903s; run: log: (pid 4877) 1903s
run: logrotate: (pid 4895) 1903s; run: log: (pid 4873) 1903s
run: nginx: (pid 4902) 1903s; run: log: (pid 4882) 1903s
run: node-exporter: (pid 4861) 1903s; run: log: (pid 4858) 1903s
run: prometheus: (pid 4900) 1903s; run: log: (pid 4880) 1903s
run: cougar: (pid 6420) 675s; run: log: (pid 4876) 1903s
run: redis: (pid 4892) 1903s; run: log: (pid 4862) 1903s
run: redis-exporter: (pid 4894) 1903s; run: log: (pid 4872) 1903s
run: sidekiq: (pid 6393) 681s; run: log: (pid 4881) 1903s

UI error in merge

Any chance something happened to the /etc/gitlab/gitlab-secrets.json ? Sometime ago, I set up a test omnibus install and forgot to bring over my source setup’s gitlab-secrets.json. Got 401 errors then when pushing. Wish I had more to suggest but that’s it.

It is an upgrade, I tried to restore the gitlab-secrets.json and the error persists

Just wondering if you ever found a solution to this? We have had this start happening on our self-hosted gitlab after an upgrade from 15.7.x to 15.11.x. So far tried everything. The issue seems to be transient and only happens on some pushes/merges and retrying resolves it regularly but it still very annoying!