Updating gitlab 14.5.1-ce to gitlab 14.5.2-ce gives me 500 code error

How to Use GitLab
,
1

Hi,

I installed gitlab 14.5.1-ce on Ubuntu 20.04.3 LTS by :

$ curl -s     https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh     | sudo bash 
$ sudo         EXTERNAL_URL="http://192.168.0.5" apt install gitlab-ce

Everything worked fine, I could sign in as root. After upgrading using

$ sudo apt update
$ sudo apt full-upgrade -y

Now, as soon as I sign in, I have 500 code error :

$ sudo tail -f /var/log/gitlab/nginx/gitlab_access.log
192.168.0.5 - - [08/Dec/2021:14:26:50 +0100] "POST /users/sign_in HTTP/1.1" 500 2926 "http://192.168.0.5/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -
192.168.0.5 - - [08/Dec/2021:14:26:51 +0100] "GET /favicon.ico HTTP/1.1" 301 168 "http://192.168.0.5/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" -

I reconfigured and restarted with better results :

$ sudo gitlab-ctl reconfigure

Here are some debug logs :

$ sudo gitlab-rake gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... 
yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ... 
GitLab Instance / Monitoring ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git version >= 2.33.0 ? ... yes (2.33.1)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Any idea how to debug that problem before downgrading ?

Thanks in advance for your help.

Gilles

2

Hi again,

I tried to investigate more my problem :

$ sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1
I, [2021-12-08T15:13:22.455943 #234566]  INFO -- : Checking encrypted values in the database
I, [2021-12-08T15:14:26.826850 #234566]  INFO -- : - Ci::InstanceVariable failures: 0
I, [2021-12-08T15:14:27.371532 #234566]  INFO -- : - Ci::PipelineScheduleVariable failures: 0
I, [2021-12-08T15:14:27.885163 #234566]  INFO -- : - Ci::Variable failures: 0
I, [2021-12-08T15:14:28.232276 #234566]  INFO -- : - Ci::GroupVariable failures: 0
I, [2021-12-08T15:14:28.282697 #234566]  INFO -- : - Ci::PipelineVariable failures: 0
I, [2021-12-08T15:14:28.318736 #234566]  INFO -- : - Ci::JobVariable failures: 0
I, [2021-12-08T15:14:29.548913 #234566]  INFO -- : - ApplicationSetting failures: 1
D, [2021-12-08T15:14:29.551612 #234566] DEBUG -- :   - ApplicationSetting[1]: ci_jwt_signing_key, runners_registration_token
I, [2021-12-08T15:14:29.724543 #234566]  INFO -- : - User failures: 0
I, [2021-12-08T15:14:29.803086 #234566]  INFO -- : - Clusters::Platforms::Kubernetes failures: 0
I, [2021-12-08T15:14:29.839940 #234566]  INFO -- : - Snippet failures: 0
I, [2021-12-08T15:14:29.843391 #234566]  INFO -- : - PersonalSnippet failures: 0
I, [2021-12-08T15:14:29.847132 #234566]  INFO -- : - ProjectSnippet failures: 0
I, [2021-12-08T15:14:29.865640 #234566]  INFO -- : - Clusters::Applications::Helm failures: 0
I, [2021-12-08T15:14:29.890382 #234566]  INFO -- : - Clusters::Applications::Prometheus failures: 0
I, [2021-12-08T15:14:29.913456 #234566]  INFO -- : - AlertManagement::HttpIntegration failures: 0
I, [2021-12-08T15:14:29.942278 #234566]  INFO -- : - PagesDomainAcmeOrder failures: 0
I, [2021-12-08T15:14:29.961614 #234566]  INFO -- : - PagesDomain failures: 0
I, [2021-12-08T15:14:29.986726 #234566]  INFO -- : - RemoteMirror failures: 0
I, [2021-12-08T15:14:30.009940 #234566]  INFO -- : - ProjectImportData failures: 0
I, [2021-12-08T15:14:30.042589 #234566]  INFO -- : - JiraConnectInstallation failures: 0
I, [2021-12-08T15:14:30.067691 #234566]  INFO -- : - GrafanaIntegration failures: 0
I, [2021-12-08T15:14:30.102281 #234566]  INFO -- : - WebHook failures: 0
I, [2021-12-08T15:14:30.109120 #234566]  INFO -- : - ProjectHook failures: 0
I, [2021-12-08T15:14:30.112309 #234566]  INFO -- : - ServiceHook failures: 0
I, [2021-12-08T15:14:30.116062 #234566]  INFO -- : - SystemHook failures: 0
I, [2021-12-08T15:14:30.147384 #234566]  INFO -- : - ErrorTracking::ProjectErrorTrackingSetting failures: 0
I, [2021-12-08T15:14:30.183828 #234566]  INFO -- : - Serverless::DomainCluster failures: 0
I, [2021-12-08T15:14:30.207398 #234566]  INFO -- : - Alerting::ProjectAlertingSetting failures: 0
I, [2021-12-08T15:14:30.228044 #234566]  INFO -- : - IncidentManagement::ProjectIncidentManagementSetting failures: 0
I, [2021-12-08T15:14:30.250941 #234566]  INFO -- : - Integrations::ZentaoTrackerData failures: 0
I, [2021-12-08T15:14:30.282146 #234566]  INFO -- : - Integrations::IssueTrackerData failures: 0
I, [2021-12-08T15:14:30.309925 #234566]  INFO -- : - Integrations::JiraTrackerData failures: 0
I, [2021-12-08T15:14:30.335117 #234566]  INFO -- : - BulkImports::Configuration failures: 0
I, [2021-12-08T15:14:30.367210 #234566]  INFO -- : - Atlassian::Identity failures: 0
I, [2021-12-08T15:14:30.409342 #234566]  INFO -- : - Clusters::KubernetesNamespace failures: 0
I, [2021-12-08T15:14:30.419498 #234566]  INFO -- : - Clusters::Integrations::Prometheus failures: 0
I, [2021-12-08T15:14:30.454998 #234566]  INFO -- : - Packages::Debian::GroupDistributionKey failures: 0
I, [2021-12-08T15:14:30.482292 #234566]  INFO -- : - Packages::Debian::ProjectDistributionKey failures: 0
I, [2021-12-08T15:14:30.514736 #234566]  INFO -- : - Clusters::Providers::Aws failures: 0
I, [2021-12-08T15:14:30.536757 #234566]  INFO -- : - Clusters::Providers::Gcp failures: 0
I, [2021-12-08T15:14:30.541077 #234566]  INFO -- : - Gitlab::BackgroundMigration::BackfillJiraTrackerDeploymentType2::JiraTrackerDataTemp failures: 0
I, [2021-12-08T15:14:30.578240 #234566]  INFO -- : - Ci::Runner failures: 0
I, [2021-12-08T15:14:30.606801 #234566]  INFO -- : - Ci::Build failures: 0
I, [2021-12-08T15:14:33.156546 #234566]  INFO -- : - Group failures: 0
I, [2021-12-08T15:14:33.249202 #234566]  INFO -- : - Project failures: 1
D, [2021-12-08T15:14:33.249278 #234566] DEBUG -- :   - Project[1]: runners_token
I, [2021-12-08T15:14:33.306698 #234566]  INFO -- : - DeployToken failures: 0
I, [2021-12-08T15:14:33.333164 #234566]  INFO -- : - Clusters::AgentToken failures: 0
I, [2021-12-08T15:14:33.358177 #234566]  INFO -- : - Operations::FeatureFlagsClient failures: 0
I, [2021-12-08T15:14:33.358261 #234566]  INFO -- : Total: 2 row(s) affected
I, [2021-12-08T15:14:33.358284 #234566]  INFO -- : Done!

It seems that there is a problem with :

I, [2021-12-08T15:14:29.548913 #234566]  INFO -- : - ApplicationSetting failures: 1
D, [2021-12-08T15:14:29.551612 #234566] DEBUG -- :   - ApplicationSetting[1]: ci_jwt_signing_key, runners_registration_token
I, [2021-12-08T15:14:33.249202 #234566]  INFO -- : - Project failures: 1
D, [2021-12-08T15:14:33.249278 #234566] DEBUG -- :   - Project[1]: runners_token

Any idea how to solve that problem ?

Thanks in advance for your help.

Gilles

3

Hello,

Sorry to ask it again but I think that my previous post may be considered as an answer to my problem but it is not, it is just some debug information. Have you an idea how can I progress to solve my problem please ?

Thanks in advance for your help.

Gilles

4

Hello everybody,

Since I have no answer to my questions in this forum, where do you suggest to get any help to debug my problem ?

Thanks in advance for your help.

Gilles