"Bad decrypt" due to `gitlab-secrets.json` reset after upgrading Ubuntu (16.04 -> 18.04) and enabling Container Registry

How to Use GitLab
1

I just wanted to share that we experienced problems after upgrading our Omnibus GitLab CE instance (11.2.3) from Ubuntu 16.04.5 LTS to 18.04.1 LTS (via do-release-upgrade) and enabling GitLab Container Registry (via external_registry_url).

I’m not sure which of these steps caused the issues, but basically

  1. our GitLab Runner (11.2.0) failed to pick up new jobs and journalctl -f included errors like these:
    gitlab-runner[12647]: time="2018-09-18T16:32:54+02:00" level=warning msg="Checking for jobs... failed"    runner=924beeca status="500 Internal Server Error"
  2. we couldn’t access the “CI / CD” page of projects using private variables and our /var/log/gitlab/gitlab-rails/production_json.log included errors like these:
    {"method":"GET","path":"/***/***/settings/ci_cd","format":"html","controller":"Projects::Settings::CiCdController","action":"show","status":500,"error":"ActionView::Template::Error: bad decrypt","duration":522.81,"view":0.0,"db":38.22,"time":"2018-09-18T14:08:29.651Z","params":[{"key":"namespace_id","value":"***"},{"key":"project_id","value":"***"}],"remote_ip":"***","user_id":***,"username":"***","gitaly_calls":1}`
1 Like
2

Eventually I found out that the gitlab-secrets.json had somehow been reset and recovering it to a version from before the update solved the issues.

1 Like