Gitlab Migration to New Server and Reset Secrets

Community
1

TL;DR 500 errors were reported when accessing CI/CD setting under projects. I probably messed up the reconfig/restore order. Resetting secrets as found here fixed ‘most’ things except still getting 500 errors when access admin area > CI/CD > Runners.

Version Gitlab Community Editon 16.10.2 (Omnibus/Repo)

The story

  • Several months ago, I updated gitlab from 13.x to 16.8 as preparation for a move to a new VM/OS.
    That process went smoothly following all the recommended hard stops.
  • I made a backup of the server at this point and backup of the /etc/gitlab directory for ssl, secrets and gitlab.rb.
  • I built a new VM, installed ce 16.8.2 and extracted the /etc/gitlab in place and restored the backup.
    I can’t recall if I ran a gitlab reconfigure before the actual restore but this was just a ‘dry run’.
  • After 2 weeks without any complaints, I backed up the entire production server in an overnight maintenance and restored it over the top on the newer server.
  • Several months go by with developers happy and I destroy the old server.
  • During this time I push several updates out to stay current and we’re at gitlab-ce-16.10.2.
  • I was notified of some 500 errors accessing Project CI/CD variables and troubleshooting indicated my mistake that never got caught during testing.
  • The rake doctor secrets check showed problems in multiple areas matching the secrets mismatch issue.
  • The fix seemed to be here and running through the steps has cleared up almost everything.

The Current Problem(s)
(which are probably related)

A 500 error when access the /admin/runners (Admin Area > CI/CD > Runners).
And
Outstanding error(s) with rake doctor secrets check (previously much higher)

[root@gitlab01 ~]# gitlab-rake gitlab:doctor:secrets VERBOSE=1
I, [2024-04-09T17:08:22.275377 #1762562]  INFO -- : - ApplicationSetting failures: 1
D, [2024-04-09T17:08:22.275468 #1762562] DEBUG -- :   - ApplicationSetting[1]: customers_dot_jwt_signing_key, runners_registration_token, error_tracking_access_token

When checking the application_settings table there are not entries for

customers_dot_jwt_signing_key
error_tracking_access_token

I do find:

encrypted_customers_dot_jwt_signing_key
encrypted_customers_dot_jwt_signing_key_iv
error_tracking_access_token_encrypted

I’m guessing I should be able to remove those entries and the rake check is just looking for out of data information.

UPDATE application_settings SET encrypted_customers_dot_jwt_signing_key = null, encrypted_customers_dot_jwt_signing_key_iv = null, error_tracking_access_token_encrypted = null;

Any advice, constructive feedback or information is greatly appreciated.

Sincerely,

Scott C.

2

Decided to try the fix I mentioned. Backed up the entire instance with gitlab-backup and as well as a VM snapshot.
Running the command gitlab-psql and then executing:

UPDATE application_settings SET encrypted_customers_dot_jwt_signing_key = null, encrypted_customers_dot_jwt_signing_key_iv = null, error_tracking_access_token_encrypted = null;

Afterward, I could navigate to the WebUI Admin Area > CI/CD > Runners.
After a screenshot to note info/tagging, deleted them and asking our developers to recreate.
So far everyone is reporting successful build/deployments again.
Noting here for future reference.

1 Like
3

Thank you so much. It helped me so much.
I took the Proxmox-template “debian-12-turnkey-gitlab_18.1-1_amd64.tar.gz”
and I had always the 500 failure at the root login.