TL;DR 500 errors were reported when accessing CI/CD setting under projects. I probably messed up the reconfig/restore order. Resetting secrets as found here fixed ‘most’ things except still getting 500 errors when access admin area > CI/CD > Runners.
Version Gitlab Community Editon 16.10.2 (Omnibus/Repo)
The story
- Several months ago, I updated gitlab from 13.x to 16.8 as preparation for a move to a new VM/OS.
That process went smoothly following all the recommended hard stops. - I made a backup of the server at this point and backup of the /etc/gitlab directory for ssl, secrets and gitlab.rb.
- I built a new VM, installed ce 16.8.2 and extracted the /etc/gitlab in place and restored the backup.
I can’t recall if I ran a gitlab reconfigure before the actual restore but this was just a ‘dry run’. - After 2 weeks without any complaints, I backed up the entire production server in an overnight maintenance and restored it over the top on the newer server.
- Several months go by with developers happy and I destroy the old server.
- During this time I push several updates out to stay current and we’re at gitlab-ce-16.10.2.
- I was notified of some 500 errors accessing Project CI/CD variables and troubleshooting indicated my mistake that never got caught during testing.
- The rake doctor secrets check showed problems in multiple areas matching the secrets mismatch issue.
- The fix seemed to be here and running through the steps has cleared up almost everything.
The Current Problem(s)
(which are probably related)
A 500 error when access the /admin/runners (Admin Area > CI/CD > Runners).
And
Outstanding error(s) with rake doctor secrets check (previously much higher)
[root@gitlab01 ~]# gitlab-rake gitlab:doctor:secrets VERBOSE=1
I, [2024-04-09T17:08:22.275377 #1762562] INFO -- : - ApplicationSetting failures: 1
D, [2024-04-09T17:08:22.275468 #1762562] DEBUG -- : - ApplicationSetting[1]: customers_dot_jwt_signing_key, runners_registration_token, error_tracking_access_token
When checking the application_settings table there are not entries for
customers_dot_jwt_signing_key
error_tracking_access_token
I do find:
encrypted_customers_dot_jwt_signing_key
encrypted_customers_dot_jwt_signing_key_iv
error_tracking_access_token_encrypted
I’m guessing I should be able to remove those entries and the rake check is just looking for out of data information.
UPDATE application_settings SET encrypted_customers_dot_jwt_signing_key = null, encrypted_customers_dot_jwt_signing_key_iv = null, error_tracking_access_token_encrypted = null;
Any advice, constructive feedback or information is greatly appreciated.
Sincerely,
Scott C.