Omnibus upgrade from 11.11.3->13.1.11: problem with secrets

How to Use GitLab
,
1

We’re going to update our inherited GitLab (Oomnibus 11.11.3/ubuntu 16.04).
Path:

  1. 11.11.3 → 11.11.8
  2. 11.11.8 → 12.0.12
  3. 12.0.12 → 12.1.17
  4. 12.1.17 → 12.10.14
  5. 12.10.14 → 13.0.14
  6. 13.0.14 → 13.1.11
  7. 13.1.11 → 13.8.8
  8. 13.8.8 → latest 13.12.Z(13.12.10)
    moving to server with Ubuntu 20.04
  9. 13.12.10 → latest 14.0.Z (14.0.9)
  10. 14.0.9(latest 14.0.Z) → 14.1.3
  11. 14.1.3 → latest 14.Y.Z (14.2.1)

We didn’t have any issues until 6 step (after the upgrading to 13.1.11) or just weren’t aware of some. We tried the new check: gitlab-rake gitlab:doctor:secrets VERBOSE=1 and got multiple errors:
ProjectImportData failures: 5
JiraTrackerData failures: 5
Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData failures: 5

output with errors

sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1

I, [2021-08-31T22:44:21.453594 #1716182] INFO – : Checking encrypted values in the database
I, [2021-08-31T22:44:28.116229 #1716182] INFO – : - ApplicationSetting failures: 0
I, [2021-08-31T22:44:28.229109 #1716182] INFO – : - User failures: 0
I, [2021-08-31T22:44:28.242724 #1716182] INFO – : - Ci::InstanceVariable failures: 0
I, [2021-08-31T22:44:28.265432 #1716182] INFO – : - Clusters::Platforms::Kubernetes failures: 0
I, [2021-08-31T22:44:28.315314 #1716182] INFO – : - Snippet failures: 0
I, [2021-08-31T22:44:28.316713 #1716182] INFO – : - PersonalSnippet failures: 0
I, [2021-08-31T22:44:28.318136 #1716182] INFO – : - ProjectSnippet failures: 0
I, [2021-08-31T22:44:28.330316 #1716182] INFO – : - Ci::PipelineScheduleVariable failures: 0
I, [2021-08-31T22:44:28.351533 #1716182] INFO – : - Ci::GroupVariable failures: 0
I, [2021-08-31T22:44:28.370344 #1716182] INFO – : - Clusters::Applications::Helm failures: 0
I, [2021-08-31T22:44:28.382289 #1716182] INFO – : - Clusters::Applications::Prometheus failures: 0
I, [2021-08-31T22:44:28.858086 #1716182] INFO – : - Ci::Variable failures: 0
I, [2021-08-31T22:44:28.872165 #1716182] INFO – : - AlertManagement::HttpIntegration failures: 0
I, [2021-08-31T22:44:28.883974 #1716182] INFO – : - Alerting::ProjectAlertingSetting failures: 0
I, [2021-08-31T22:44:28.895921 #1716182] INFO – : - Atlassian::Identity failures: 0
I, [2021-08-31T22:44:28.907171 #1716182] INFO – : - BulkImports::Configuration failures: 0
I, [2021-08-31T22:44:28.918955 #1716182] INFO – : - Ci::JobVariable failures: 0
I, [2021-08-31T22:44:30.766660 #1716182] INFO – : - Ci::PipelineVariable failures: 0
I, [2021-08-31T22:44:30.768681 #1716182] INFO – : - Clusters::Integrations::Prometheus failures: 0
I, [2021-08-31T22:44:30.792710 #1716182] INFO – : - Clusters::KubernetesNamespace failures: 0
I, [2021-08-31T22:44:30.806290 #1716182] INFO – : - Clusters::Providers::Aws failures: 0
I, [2021-08-31T22:44:30.819298 #1716182] INFO – : - Clusters::Providers::Gcp failures: 0
I, [2021-08-31T22:44:30.835377 #1716182] INFO – : - ErrorTracking::ProjectErrorTrackingSetting failures: 0
I, [2021-08-31T22:44:30.848005 #1716182] INFO – : - GrafanaIntegration failures: 0
I, [2021-08-31T22:44:30.898085 #1716182] INFO – : - WebHook failures: 0
I, [2021-08-31T22:44:30.900558 #1716182] INFO – : - ProjectHook failures: 0
I, [2021-08-31T22:44:30.901943 #1716182] INFO – : - ServiceHook failures: 0
I, [2021-08-31T22:44:30.903345 #1716182] INFO – : - SystemHook failures: 0
I, [2021-08-31T22:44:30.914828 #1716182] INFO – : - IncidentManagement::ProjectIncidentManagementSetting failures: 0
I, [2021-08-31T22:44:30.926919 #1716182] INFO – : - JiraConnectInstallation failures: 0
I, [2021-08-31T22:44:30.940297 #1716182] INFO – : - Packages::Debian::GroupDistribution failures: 0
I, [2021-08-31T22:44:30.954553 #1716182] INFO – : - Packages::Debian::ProjectDistribution failures: 0
I, [2021-08-31T22:44:30.973528 #1716182] INFO – : - PagesDomain failures: 0
I, [2021-08-31T22:44:30.985800 #1716182] INFO – : - PagesDomainAcmeOrder failures: 0
I, [2021-08-31T22:44:31.016589 #1716182] INFO – : - ProjectImportData failures: 5
D, [2021-08-31T22:44:31.016631 #1716182] DEBUG – : - ProjectImportData[2]: credentials
D, [2021-08-31T22:44:31.016651 #1716182] DEBUG – : - ProjectImportData[4]: credentials
D, [2021-08-31T22:44:31.016668 #1716182] DEBUG – : - ProjectImportData[6]: credentials
D, [2021-08-31T22:44:31.016686 #1716182] DEBUG – : - ProjectImportData[8]: credentials
D, [2021-08-31T22:44:31.016719 #1716182] DEBUG – : - ProjectImportData[11]: credentials
I, [2021-08-31T22:44:31.027989 #1716182] INFO – : - IssueTrackerData failures: 0
D, [2021-08-31T22:44:31.051738 #1716182] DEBUG – : > Something went wrong for JiraTrackerData[2].username: must specify an iv
D, [2021-08-31T22:44:31.051940 #1716182] DEBUG – : > Something went wrong for JiraTrackerData[3].username: must specify an iv
D, [2021-08-31T22:44:31.052124 #1716182] DEBUG – : > Something went wrong for JiraTrackerData[4].username: must specify an iv
D, [2021-08-31T22:44:31.052288 #1716182] DEBUG – : > Something went wrong for JiraTrackerData[5].username: must specify an iv
D, [2021-08-31T22:44:31.052443 #1716182] DEBUG – : > Something went wrong for JiraTrackerData[6].username: must specify an iv
I, [2021-08-31T22:44:31.052501 #1716182] INFO – : - JiraTrackerData failures: 5
D, [2021-08-31T22:44:31.052523 #1716182] DEBUG – : - JiraTrackerData[2]: username
D, [2021-08-31T22:44:31.052541 #1716182] DEBUG – : - JiraTrackerData[3]: username
D, [2021-08-31T22:44:31.052558 #1716182] DEBUG – : - JiraTrackerData[4]: username
D, [2021-08-31T22:44:31.052574 #1716182] DEBUG – : - JiraTrackerData[5]: username
D, [2021-08-31T22:44:31.052590 #1716182] DEBUG – : - JiraTrackerData[6]: username
I, [2021-08-31T22:44:31.065093 #1716182] INFO – : - OpenProjectTrackerData failures: 0
I, [2021-08-31T22:44:31.094932 #1716182] INFO – : - RemoteMirror failures: 0
I, [2021-08-31T22:44:31.108840 #1716182] INFO – : - Serverless::DomainCluster failures: 0
I, [2021-08-31T22:44:31.120609 #1716182] INFO – : - Gitlab::BackgroundMigration::BackfillJiraTrackerDeploymentType2::JiraTrackerDataTemp failures: 0
I, [2021-08-31T22:44:31.121594 #1716182] INFO – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::IssueTrackerData failures: 0
D, [2021-08-31T22:44:31.134218 #1716182] DEBUG – : > Something went wrong for Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[2].username: must specify an iv
D, [2021-08-31T22:44:31.134400 #1716182] DEBUG – : > Something went wrong for Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[3].username: must specify an iv
D, [2021-08-31T22:44:31.134598 #1716182] DEBUG – : > Something went wrong for Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[4].username: must specify an iv
D, [2021-08-31T22:44:31.134756 #1716182] DEBUG – : > Something went wrong for Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[5].username: must specify an iv
D, [2021-08-31T22:44:31.134911 #1716182] DEBUG – : > Something went wrong for Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[6].username: must specify an iv
I, [2021-08-31T22:44:31.134969 #1716182] INFO – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData failures: 5
D, [2021-08-31T22:44:31.134993 #1716182] DEBUG – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[2]: username
D, [2021-08-31T22:44:31.135011 #1716182] DEBUG – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[3]: username
D, [2021-08-31T22:44:31.135029 #1716182] DEBUG – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[4]: username
D, [2021-08-31T22:44:31.135046 #1716182] DEBUG – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[5]: username
D, [2021-08-31T22:44:31.135063 #1716182] DEBUG – : - Gitlab::BackgroundMigration::MigrateIssueTrackersSensitiveData::JiraTrackerData[6]: username
I, [2021-08-31T22:44:31.210257 #1716182] INFO – : - Group failures: 0
I, [2021-08-31T22:44:31.327472 #1716182] INFO – : - Project failures: 0
I, [2021-08-31T22:44:31.357811 #1716182] INFO – : - Ci::Runner failures: 0
I, [2021-08-31T22:44:37.620483 #1716182] INFO – : - Ci::Build failures: 0
I, [2021-08-31T22:44:37.644259 #1716182] INFO – : - DeployToken failures: 0
I, [2021-08-31T22:44:37.659213 #1716182] INFO – : - Clusters::AgentToken failures: 0
I, [2021-08-31T22:44:37.671818 #1716182] INFO – : - Operations::FeatureFlagsClient failures: 0
I, [2021-08-31T22:44:37.671898 #1716182] INFO – : Total: 15 row(s) affected
I, [2021-08-31T22:44:37.671911 #1716182] INFO – : Done!

Screenshot with check results

image
image1765×947 325 KB

What can we do wrong? What can be affected by these errors? How determinate which projects/repositories were affected (we have more than 500 projects)?

2

You might want to check this issue out, as other people have experienced similar. It’s more to do with your Jira integration it seems.

you might want to add your error messages in that issue to get a response on how to fix it.

3

@iwalker
Thank you! It resolved errors that related to “JiraTrackerData”.
Now I have 5 errors with “ProjectImportData”:

image
image1344×342 101 KB

Regarding adding my errors to the issue Error in ProcessCommitWorker: JIRA ArgumentError: must specify an iv (#198030) · Issues · GitLab.org / GitLab · GitLab : is it okay that I will post messages in a closed issue? Or will be better to create a new one?