Ci_jwt_signing_key setting has no effect in Docker image

I’m currently attempting to set up a GitLab server in a Docker container using the gitlab/gitlab-ce:16.3.5-ce.0 Docker image. As part of my configuration, I would like to set the ci_jwt_signing_key secret value to an RSA key I’ve generated. I’m attempting to do this by setting the GITLAB_OMNIBUS_CONFIG environment variable to gitlab_rails['ci_jwt_signing_key'] = "<RSA key PEM contents>", but this does not seem to have any effect.

For example:

openssl genrsa -out key.pem
container="$(docker run -p 80:80 -d -e GITLAB_OMNIBUS_CONFIG="gitlab_rails['ci_jwt_signing_key'] = \"$(cat key.pem)\"" gitlab/gitlab-ce:16.3.5-ce.0)"

Notably, I can confirm that Omnibus is correctly reading the key PEM that I set and writing it to both /etc/gitlab/gitlab-secrets.json and /var/opt/gitlab/gitlab-rails/etc/secrets.yml:

cat key.pem # for comparison
docker exec $container cat /etc/gitlab/gitlab-secrets.json | jq -r '.gitlab_rails.ci_jwt_signing_key'
docker exec $container cat /var/opt/gitlab/gitlab-rails/etc/secrets.yml | yq -r '.production.ci_jwt_signing_key'

However, the actual setting value in GitLab is incorrect:

echo 'puts Gitlab::CurrentSettings.ci_jwt_signing_key' | docker exec -i $container gitlab-rails console | tail -n +10 | head -n -2 > actual.pem
cat actual.pem # does not match key.pem

And, unsurprisingly, I am unable to validate JWTs signed using the private key I provided with the public key JWKS at https://localhost:/-/jwks, which I can test using smallstep:

curl http://localhost/-/jwks > gitlab.jwks
step crypto jwt sign -iss 'test' --aud 'gitlab' --sub 'test' --exp "$(date -d 'tomorrow' +%s)" --key key.pem | step crypto jwt verify --jwks gitlab.jwks --iss 'test' --aud 'gitlab'
# cannot find key with kid ... on gitlab.jwks

But I am able to using the private key obtained through the gitlab-rails console:

step crypto jwt sign -iss 'test' --aud 'gitlab' --sub 'test' --exp "$(date -d 'tomorrow' +%s)" --key actual.pem | step crypto jwt verify --jwks gitlab.jwks --iss 'test' --aud 'gitlab'
# succeeds

I have also tried adding package['generate_default_secrets'] = false to GITLAB_OMNIBUS_CONFIG but this causes the GitLab setup process to fail with permissions errors while accessing /var/opt/gitlab/gitlab-rails/etc/secrets.yml.

What am I doing wrong here? Is this a bug in the Omnibus recipes?

So I looked into this for a while and finally figured out what’s going on: it looks like the Omnibus package was correctly updated to set the ci_jwt_signing_key secret in /etc/gitlab/gitlab-secrets.json and /var/opt/gitlab/gitlab-rails/etc/secrets.yml, but GitLab was never updated to read the setting from the Rails secrets. Instead, it was updated to read it from the application settings stored in the database, and as a result will always ignore what’s in gitlab-secrets.json and secrets.yml and generate a new key during the post-deployment migration.

In order for the gitlab_rails['ci_jwt_signing_key'] setting to work as intended, I believe references in GitLab to

Gitlab::CurrentSettings.ci_jwt_signing_key

should be updated to

Rails.application.secrets.ci_jwt_signing_key

Alternatively, the Omnibus package could be updated to read the gitlab_rails['ci_jwt_signing_key'] setting and populate Gitlab::CurrentSettings.ci_jwt_signing_key with it during deployment, rather than Rails.application.secrets.ci_jwt_signing_key which is not actually used by GitLab.

My workaround for this for now is to create a file with the following contents and mount it into the GitLab Docker container at /opt/gitlab/embedded/service/gitlab-rails/db/fixtures/production/100_ci_jwt_signing_key.rb:

Gitlab::CurrentSettings.current_application_settings.tap do |settings|
  settings.ci_jwt_signing_key = ENV['GITLAB_CI_JWT_SIGNING_KEY']
end.save

Then I set the GITLAB_CI_JWT_SIGNING_KEY environment variable in my Docker container, and the post-deployment migration will set the GitLab application setting correctly.