17.7 Upgrade breaks Gitlab-Runner jobs with token not set error

Enable.Services · December 20, 2024, 12:36pm

Please help fill in this template with all the details to help others help you more efficiently. Use formatting blocks for code, config, logs and ensure to remove sensitive data.

Problem to solve

CI Jobs fail to run or cancel. Logs show an issue with the job token signing key:

==> /var/log/gitlab/gitlab-rails/production.log <==
  
RuntimeError (CI job token signing key is not set):
  
lib/ci/job_token/jwt.rb:66:in `key'
lib/ci/job_token/jwt.rb:41:in `decode'
app/finders/ci/auth_job_finder.rb:33:in `find_job_by_token'
app/finders/ci/auth_job_finder.rb:14:in `execute!'
app/finders/ci/auth_job_finder.rb:24:in `execute'
lib/gitlab/auth/auth_finders.rb:485:in `find_valid_running_job_by_token!'
lib/gitlab/auth/auth_finders.rb:373:in `find_user_from_job_token_query_params_or_header'
lib/gitlab/auth/auth_finders.rb:88:in `find_user_from_job_token'
ee/lib/ee/gitlab/auth/auth_finders.rb:61:in `find_user_from_job_token'
lib/gitlab/auth/request_authenticator.rb:90:in `find_user_from_any_authentication_method'
lib/gitlab/auth/request_authenticator.rb:41:in `find_sessionless_user'
lib/gitlab/auth/request_authenticator.rb:22:in `block in user'
lib/gitlab/auth/request_authenticator.rb:21:in `each'
lib/gitlab/auth/request_authenticator.rb:21:in `user'
lib/gitlab/auth/request_authenticator.rb:17:in `find_authenticated_requester'
...

journalctl -u gitlab-runner -n 20 -f 

...
Dec 20 11:33:51 indra gitlab-runner[4672]: WARNING: Appending trace to coordinator... failed   code=500 job=167317 job-log= job-status= runner=qYyH73aH sent-log=0-1133 status=500 Internal Server Error update-interval=0s

Steps to reproduce

Upgrade to version 17.7 using gitlab-runners

Configuration

Debian Bookworm
4 Gitlab runners, all running shell executor
Redis enabled

Versions

Please add an x whether options apply, and add the version information.

[ x] Self-managed
GitLab.com SaaS
Dedicated

Versions

GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info): 17.7

Helpful resources

Check the FAQ for helpful documentation, issues/bugs/feature proposals, and troubleshooting tips.
Before opening a new topic, make sure to search for keywords in the forum search
Check the GitLab project for existing issues. If you encounter a bug, please create a bug report issue.
Review existing troubleshooting docs.

Thanks for taking the time to be thorough in your request, it really helps!

fr3nch13 · December 20, 2024, 9:20pm

I’m having the exact same issue. I opened an issue with the gitlab-runner project at: All runners failing after gitlab-fips package updated to 17.7.0 (500 err on server with "CI job token signing key is not set") (#38397) · Issues · GitLab.org / gitlab-runner · GitLab

fr3nch13 · December 21, 2024, 12:43am

This fix worked for me at least: All runners failing after gitlab-fips package updated to 17.7.0 (500 err on server with "CI job token signing key is not set") (#38397) · Issues · GitLab.org / gitlab-runner · GitLab

gitlab-rails db:migrate:down VERSION=20241017160504
gitlab-rails db:migrate:up VERSION=20241017160504
gitlab-ctl restart

imker25 · December 21, 2024, 8:57am

I had the same issue after installing the latest update. To fix the problem I needed to do the following steps:

Update the gitlab-runner-helper-images.deb package on each runner
Update the gitlab-runner_amd64.deb package on each runner
Restart the gitlab-runner.service on each runner

Do the DB migration as shown by @fr3nch13 on the GitLab main server

gitlab-rails db:migrate:down VERSION=20241017160504
gitlab-rails db:migrate:up VERSION=20241017160504
gitlab-ctl restart

Now runners are stable again, and CI Jobs are executed without problems.

stanhu · December 23, 2024, 8:18am

Thanks for sharing the workaround.

For those of you who are running into this, can you run the following before you initiate the workaround?

gitlab-rake db:migrate:status
gitlab-rails console and then:
Gitlab::CurrentSettings.ci_job_token_signing_key.present?
ApplicationSetting.last.valid?

Genva · December 23, 2024, 8:33am

gitlab-rake db:migrate:status

   up     20241017160107  regular  17.7         AddCiJobTokenSigningKeyToApplicationSettings
   up     20241017160504  regular  17.7         GenerateCiJobTokenSigningKey
   up     20241029130136  regular  17.7         AddWithHighestRolePlannerToUsersStatistics
   up     20241030031829  regular  17.7         AddResourceUsageLimitsToApplicationSettings
   up     20241104130731  regular  17.7         AddRoleApproversToApprovalMergeRequestRules
   up     20241105134416  regular  17.7         AddUsernamePasswordToVirtualRegistriesPackagesMavenUpstream
   up     20241105181558  regular  17.7         AddUniqIndexPackageReference
   up     20241106114853  regular  17.7         CreateSystemAccessGroupMicrosoftApplications
   up     20241106115015  regular  17.7         CreateSystemAccessGroupMicrosoftGraphAccessTokens
   up     20241106183051  regular  17.7         CreateAiConversationThreadsAndMessages
   up     20241108172443  regular  17.7         AddMigrateJenkinsBannerApplicationSetting
   up     20241111084209  regular  17.7         AddTemporaryIndicesForGitlabSubscriptionsCleanup
   up     20241111092153  regular  17.7         CleanTrialGitlabSubscriptionsDateAttributes
   up     20241111104411  regular  17.7         RemoveTemporaryIndicesForGitlabSubscriptionsCleanup
   up     20241111143504  regular  17.7         AddFkToAiConversationThreadsAndMessages
   up     20241111162911  regular  17.7         AddLegacyDestinationIdToExternalStreamingDestinations
   up     20241112055215  regular  17.7         AddPipelineVariablesDefaultRoleToNamespaceSettings
   up     20241112123436  regular  17.7         UpdateSeatControlInApplicationSettings
   up     20241113070714  regular  17.7         AddNamespaceSettingTokenExpiryInherited
   up     20241114113844  regular  17.7         AddLfkTriggersToNotes
   up     20241114232506  regular  17.7         MoveAllReadyZoektIndicesToPending
   up     20241115075017  regular  17.7         AddMemberRoleIdToProjectGroupLinks
   up     20241115121334  regular  17.7         AddDeletePipelinesInSecondsToProjectCiCdSettings
   up     20241117164341  regular  17.7         RemoveSecurityPoliciesDescriptionTextLimit
   up     20241119101703  regular  17.7         AddPostgresTableSizesView
   up     20241119103103  regular  17.7         AddPagesDefaultDomainRedirectToProjectSettings
   up     20241120200334  regular  17.7         DeleteRemoteDevelopmentAgentConfigs
   up     20241120210519  regular  17.7         AddAmazonQToAiSettings
   up     20241121033259  regular  17.7         AddFkToMemberRolesOnProjectGroupLinks
   up     20241121033758  regular  17.7         AddIndexToProjectGroupLinksOnMemberRoleId
   up     20241121071110  regular  17.7         AddDescriptionToPersonalAccessTokens
   up     20241121191751  regular  17.7         AddCreatedAtColumnToProjectComplianceFrameworkSettings
   up     20241121191754  regular  17.7         AddIndexToProjectComplianceFrameworkSettingsCreatedAt
   up     20241121192044  regular  17.7         AddIndexesForExternalStreamingDestinationUniqueNameScope
   up     20241122093316  regular  17.7         AddForeginKeyGroupMicrosoftGraphAccessTokens
   up     20241122093341  regular  17.7         AddForeginKeyApplicationMicrosoftGraphAccessTokens
   up     20241122121328  regular  17.7         AddApprovalPolicyActionIdxToApprovalProjectRules
   up     20241122121350  regular  17.7         AddApprovalPolicyActionIdxToApprovalMergeRequestRules
   up     20241122121652  regular  17.7         AddActionIdxToScanResultPolicies
   up     20241125133011  regular  17.7         AddGroupIdToGroupWikiRepositoryStates
   up     20241125140958  regular  17.7         AddMetadataToZoektEnabledNamespaces
   up     20241125145005  regular  17.7         AddNamespaceIdToIssuableSeverities
   up     20241125155301  regular  17.7         AddInstanceIntegrationIdColumnToIssueTrackerData
   up     20241125155621  regular  17.7         AddInstanceIntegrationIdColumnToZentaoTrackerData
   up     20241125155711  regular  17.7         AddInstanceIntegrationIdColumnToJiraTrackerData
   up     20241125173317  regular  17.7         AddProjectIdToCiBuildTraceChunks
   up     20241125222645  regular  17.7         AddStopSettingToEnvironments
   up     20241126103055  regular  17.7         AddMultipleNotNullConstraintToIssueTrackerData
   up     20241126104129  regular  17.7         AddMultipleNotNullConstraintToZentaoTrackerData
   up     20241126104222  regular  17.7         AddMultipleNotNullConstraintToJiraTrackerData
   up     20241126104505  regular  17.7         RemoveNotNullConstraintFromIssueTrackerData
   up     20241126104606  regular  17.7         RemoveNotNullConstraintFromZentaoTrackerData
   up     20241126104707  regular  17.7         RemoveNotNullConstraintFromJiraTrackerData
   up     20241126105922  regular  17.7         AddInstanceIntegrationFkToIssueTrackerData
   up     20241126110640  regular  17.7         AddInstanceIntegrationFkToZentaoTrackerData
   up     20241126110806  regular  17.7         AddInstanceIntegrationFkToJiraTrackerData
   up     20241126131337  regular  17.7         AddDefaultPermissionsToGroupScopeLinks
   up     20241126131359  regular  17.7         AddDefaultPermissionsToProjectScopeLinks
   up     20241126151227  regular  17.7         AddProjectIdToCiBuildPendingStates
   up     20241126152143  regular  17.7         AddIndexOnInstanceIntegrationIdToIssueTrackerData
   up     20241126162408  regular  17.7         AddIndexOnInstanceIntegrationIdToZentaoTrackerData
   up     20241126170110  regular  17.7         AddIndexOnInstanceIntegrationIdToJiraTrackerData
   up     20241127015108  regular  17.7         AddShardingKeyIdToUploads
   up     20241127092714  regular  17.7         CreateContainerRegistryProtectionTagRules
   up     20241127135043  regular  17.7         AddCompositeIdentityEnforcedToUsers
   up     20241127142223  regular  17.7         AddAssignDuoSeatsToSamlGroupLinks
   up     20241127161525  regular  17.7         AddDesignsAndDevelopmentWidgetsToTicketWorkItemType
   up     20241127164647  regular  17.7         TrackAiConversationThredRecordChanges
   up     20241128103042  regular  17.7         AddIndexSeatAssignmentNamespaceLastActivityOn
   up     20241128113732  regular  17.7         AddOrganizationToBulkImportEntities
   up     20241128141733  regular  17.7         CreateProjectsBranchRulesSquashOptions
   up     20241129135625  regular  17.7         AddFkForProjectsBranchRulesSquashOptionsProtectedBranch
   up     20241129135646  regular  17.7         AddFkForProjectsBranchRulesSquashOptionsProject
   up     20241129142800  regular  17.7         AddCheckConstraintToConanFileMetadata
   up     20241129160936  regular  17.7         AddStreamingDestinationRefToExternalAuditEventDestinations
   up     20241129161024  regular  17.7         AddStreamingDestinationRefToInstanceAuditEventDestination
   up     20241129161110  regular  17.7         AddStreamingDestinationRefToGoogleCloudLoggingConfiguration
   up     20241129161203  regular  17.7         AddStreamingDestinationRefToInstanceGoogleCloudLoggingConfiguration
   up     20241129161205  regular  17.7         ReplaceUniqueIndexOnScanResultPolicies
   up     20241129161213  regular  17.7         AddStreamingDestinationRefToAmazonS3Configuration
   up     20241129161223  regular  17.7         AddStreamingDestinationRefToInstanceAmazonS3Configuration
   up     20241201092435  regular  17.7         AddIndexToBuildSources
   up     20241201162318  regular  17.7         AddCustomRolesToScanResultPolicies
   up     20241202054640  regular  17.7         AddVulnerabilityEventsToWebHooks
   up     20241202140049  regular  17.7         AddNamespaceIdToIssueUserMentions
   up     20241202141407  regular  17.7         AddNamespaceIdToResourceLinkEvents
   up     20241202142250  regular  17.7         AddNamespaceIdToResourceWeightEvents
   up     20241202143619  regular  17.7         AddNamespaceIdToWorkItemProgresses
   up     20241202145035  regular  17.7         AddNamespaceIdToIncidentManagementIssuableEscalationStatuses
   up     20241202151532  regular  17.7         AddNamespaceIdToIssuableResourceLinks
   up     20241203073521  regular  17.7         AddNamespaceIdToIssueAssignmentEvents
   up     20241203074400  regular  17.7         AddNamespaceIdToIssueMetrics
   up     20241203075014  regular  17.7         AddNamespaceIdToIssueCustomerRelationsContacts
   up     20241203075616  regular  17.7         AddNamespaceIdToIssueEmailParticipants
   up     20241203080305  regular  17.7         AddNamespaceIdToIssueEmails
   up     20241203081752  regular  17.7         AddNamespaceIdToIssuableMetricImages
   up     20241203144833  regular  17.7         AddNamespaceIdToSentryIssues
   up     20241203213010  regular  17.7         DropIssuesCorrectWorkItemTypeIdDefault
   up     20241204035453  regular  17.7         AddPipelinesTriggerId
   up     20241204130221  regular  17.7         AddProtectedBranchProjectIdToProtectedBranchMergeAccessLevels
   up     20241204130226  regular  17.7         AddProtectedBranchNamespaceIdToProtectedBranchMergeAccessLevels
   up     20241204180053  regular  17.7         RemoveSoftwareLicenseExistenceConstraintToSoftwareLicensePolicies
   up     20241204215259  regular  17.7         AddBidirectionalSyncTriggerOnIssuesWorkItemTypeId
   up     20241204221832  regular  17.7         AddProjectIdToCiBuildTraceMetadata
   up     20241205143056  regular  17.7         AddNamespaceIdToStatusPagePublishedIncidents
   up     20241205191938  regular  17.7         AddAutopopulatedToCiJobTokenProjectScopeLink
   up     20241205191957  regular  17.7         AddAutopopulatedToCiJobTokenGroupScopeLink
   up     20241206150204  regular  17.7         AddCustomRolesConstraintToScanResultPolicies
   up     20241211124617  regular  17.7         AddZoektIndexOverprovisionedReadyHighWatermarkExceeded
   up     20241023084019  post     17.7         RemoveVirtualRegistriesPackagesMavenCachedResponsesDownloadsCountColumn
   up     20241024201454  post     17.7         ValidateProjectIdNotNullConstraintOnCiStages
   up     20241027152700  post     17.7         RequeueBackfillMilestoneReleasesProjectId
   up     20241107064635  post     17.7         QueueBackfillCiRunnerMachinesPartitionedTable
   up     20241108133603  post     17.7         QueueBackfillSeatAssignmentsTable
   up     20241111220006  post     17.7         UpdateUniqueIndexOnPackagesNugetSymbol
   up     20241112084921  post     17.7         RemoveMembersLastActivityOnColumn
   up     20241112163029  post     17.7         QueueDeleteOrphanedGroups
   up     20241113144244  post     17.7         QueueBackfillVulnerabilityNamespaceHistoricalStatistics
   up     20241113230840  post     17.7         DropBrokenFkForPCiBuildsUpstreamPipelineId
   up     20241114202257  post     17.7         QueueBackfillComplianceViolationNullTargetProjectIds
   up     20241114225103  post     17.7         AddIndexApprovalProjectRulesOnProjectIdRuleType
   up     20241114225309  post     17.7         AddIndexApprovalProjectRulesOnProjectIdConfigIdAndId
   up     20241114225508  post     17.7         AddIndexApprovalMrRulesOnProjectIdPolicyRuleIdAndId
   up     20241114225528  post     17.7         AddIndexScanResultPolicyViolationsOnProjectIdPolicyRuleIdAndId
   up     20241114231251  post     17.7         DropIndexApprovalMrRulesOnConfigIdAndPolicyRuleId
   up     20241114231606  post     17.7         DropIndexApprovalProjectRulesOnProjectId
   up     20241114231724  post     17.7         DropIndexScanResultPolicyViolationsOnProjectId
   up     20241115121337  post     17.7         AddIndexOnProjectSettingsForOldPipelines
   up     20241115122644  post     17.7         FinalizeDeleteOrphanedBuildRecords
   up     20241115133502  post     17.7         FinalizeCopyTaggingsToPCiBuildTags
   up     20241115140940  post     17.7         SyncPartitionFkForPCiBuildsExecutionConfigsAndPCiPipelines
   up     20241115211552  post     17.7         QueueBackfillSoftwareLicenseSpdxIdentifierForSoftwareLicensePolicies
   up     20241118121418  post     17.7         AddSyncIndexToSbomOccurrencesForSeverityAggregations
   up     20241118124301  post     17.7         RequeueBackfillSecurityPolicies
   up     20241118145604  post     17.7         QueueCreateMissingTerraformModuleMetadata
   up     20241119085925  post     17.7         RemoveNamespaceLimitsTemporaryStorageIncreaseEndsOnColumn
   up     20241119105148  post     17.7         RemoveFileChecksumFromXrayReports
   up     20241119143004  post     17.7         AsyncValidationForBuildsPipelinesFk
   up     20241120134610  post     17.7         SetNamespacesOrganizationIdNotNull
   up     20241120232540  post     17.7         FinalizeBackfillProjectRelationExportsProjectId
   up     20241121090148  post     17.7         RemoveNamespacePendingDeleteColumn
   up     20241121115756  post     17.7         PrepareNamespacesOrganizationIdNotNullValidation
   up     20241121140458  post     17.7         AddIndexToVulnerabilityIdentifiersOnProjectIdAndName
   up     20241122031025  post     17.7         ChangeOrganizationIdDefaultOauth
   up     20241125072626  post     17.7         FinalizeBackfillVulnerabilityFindingsRemediationsProjectId
   up     20241125085625  post     17.7         AddFkBetweenCiBuildsAndCiPipelines
   up     20241125125332  post     17.7         QueueBackfillFreeSharedRunnersMinutesLimit
   up     20241125133120  post     17.7         IndexGroupWikiRepositoryStatesOnGroupId
   up     20241125133216  post     17.7         AddGroupWikiRepositoryStatesGroupIdFk
   up     20241125133312  post     17.7         AddGroupWikiRepositoryStatesGroupIdTrigger
   up     20241125133627  post     17.7         QueueBackfillGroupWikiRepositoryStatesGroupId
   up     20241125145006  post     17.7         IndexIssuableSeveritiesOnNamespaceId
   up     20241125145007  post     17.7         AddIssuableSeveritiesNamespaceIdFk
   up     20241125145008  post     17.7         AddIssuableSeveritiesNamespaceIdTrigger
   up     20241125145009  post     17.7         QueueBackfillIssuableSeveritiesNamespaceId
   up     20241125173511  post     17.7         IndexCiBuildTraceChunksOnProjectId
   up     20241125174339  post     17.7         QueueBackfillCiBuildTraceChunksProjectId
   up     20241125232348  post     17.7         FinalizeDisableAllowRunnerRegistrationOnNamespaceSettingsForGitlabCom
   up     20241126151232  post     17.7         IndexCiBuildsPendingStatesOnProjectId
   up     20241126151234  post     17.7         QueueBackfillCiBuildPendingStatesProjectId
   up     20241126182756  post     17.7         DropWorkspacesDesiredConfigGeneratorVersionDefault
   up     20241127102904  post     17.7         ValidateNamespacesOrganizationIdNotNullConstraint
   up     20241127140830  post     17.7         AddDomainIdxToAlertManagementAlerts
   up     20241127151803  post     17.7         QueueMigrateSoftwareLicenseWithoutSpdxIdentifierToCustomLicenses
   up     20241127210044  post     17.7         BackfillUserDetails
   up     20241128071834  post     17.7         RemovePushTablesForcePushDeleteBranchRegex
   up     20241128102335  post     17.7         AsyncValidationForCiPipelinesFks
   up     20241128102925  post     17.7         RemoveBrokenFkBetweenCiPipelinesBuildConfigs
   up     20241128114137  post     17.7         SetOrganizationIdForBulkImportEntities
   up     20241128114319  post     17.7         AddMultiNotNullConstraintToBulkImportEntities
   up     20241129084249  post     17.7         AddIndexMemberRoleOnBaseAccessLevel
   up     20241129104415  post     17.7         CreateSubscriptionsIndexOnNoteableSynchronously
   up     20241129104531  post     17.7         CreateSentNotificationsIndexOnNoteableSynchronously
   up     20241129110758  post     17.7         DropSentNotificationsIndexOnNoteableId
   up     20241202112738  post     17.7         RevertOauthOrganizationDefault
   up     20241202140050  post     17.7         IndexIssueUserMentionsOnNamespaceId
   up     20241202140051  post     17.7         AddIssueUserMentionsNamespaceIdFk
   up     20241202140052  post     17.7         AddIssueUserMentionsNamespaceIdTrigger
   up     20241202140053  post     17.7         QueueBackfillIssueUserMentionsNamespaceId
   up     20241202141408  post     17.7         IndexResourceLinkEventsOnNamespaceId
   up     20241202141409  post     17.7         AddResourceLinkEventsNamespaceIdFk
   up     20241202141410  post     17.7         AddResourceLinkEventsNamespaceIdTrigger
   up     20241202141411  post     17.7         QueueBackfillResourceLinkEventsNamespaceId
   up     20241202142251  post     17.7         IndexResourceWeightEventsOnNamespaceId
   up     20241202142252  post     17.7         AddResourceWeightEventsNamespaceIdFk
   up     20241202142253  post     17.7         AddResourceWeightEventsNamespaceIdTrigger
   up     20241202142254  post     17.7         QueueBackfillResourceWeightEventsNamespaceId
   up     20241202143620  post     17.7         IndexWorkItemProgressesOnNamespaceId
   up     20241202143621  post     17.7         AddWorkItemProgressesNamespaceIdFk
   up     20241202143622  post     17.7         AddWorkItemProgressesNamespaceIdTrigger
   up     20241202143623  post     17.7         QueueBackfillWorkItemProgressesNamespaceId
   up     20241202145036  post     17.7         IndexIncidentManagementIssuableEscalationStatusesOnNamespaceId
   up     20241202145037  post     17.7         AddIncidentManagementIssuableEscalationStatusesNamespaceIdFk
   up     20241202145038  post     17.7         AddIncidentManagementIssuableEscalationStatusesNamespaceIdTrigger
   up     20241202145039  post     17.7         QueueBackfillIncidentManagementIssuableEscalationStatusesNamespaceId
   up     20241202151533  post     17.7         IndexIssuableResourceLinksOnNamespaceId
   up     20241202151534  post     17.7         AddIssuableResourceLinksNamespaceIdFk
   up     20241202151535  post     17.7         AddIssuableResourceLinksNamespaceIdTrigger
   up     20241202151536  post     17.7         QueueBackfillIssuableResourceLinksNamespaceId
   up     20241202170820  post     17.7         QueueBackfillIssueLinkIdOnRelatedEpicLinks
   up     20241203042054  post     17.7         AddProjectOrganizationIdNotNullConstraint
   up     20241203042134  post     17.7         PrepareProjectOrganizationIdNotNullValidation
   up     20241203073522  post     17.7         IndexIssueAssignmentEventsOnNamespaceId
   up     20241203073523  post     17.7         AddIssueAssignmentEventsNamespaceIdFk
   up     20241203073524  post     17.7         AddIssueAssignmentEventsNamespaceIdTrigger
   up     20241203073525  post     17.7         QueueBackfillIssueAssignmentEventsNamespaceId
   up     20241203074401  post     17.7         IndexIssueMetricsOnNamespaceId
   up     20241203074402  post     17.7         AddIssueMetricsNamespaceIdFk
   up     20241203074403  post     17.7         AddIssueMetricsNamespaceIdTrigger
   up     20241203074404  post     17.7         QueueBackfillIssueMetricsNamespaceId
   up     20241203075015  post     17.7         IndexIssueCustomerRelationsContactsOnNamespaceId
   up     20241203075016  post     17.7         AddIssueCustomerRelationsContactsNamespaceIdFk
   up     20241203075017  post     17.7         AddIssueCustomerRelationsContactsNamespaceIdTrigger
   up     20241203075018  post     17.7         QueueBackfillIssueCustomerRelationsContactsNamespaceId
   up     20241203075617  post     17.7         IndexIssueEmailParticipantsOnNamespaceId
   up     20241203075618  post     17.7         AddIssueEmailParticipantsNamespaceIdFk
   up     20241203075619  post     17.7         AddIssueEmailParticipantsNamespaceIdTrigger
   up     20241203075620  post     17.7         QueueBackfillIssueEmailParticipantsNamespaceId
   up     20241203080306  post     17.7         IndexIssueEmailsOnNamespaceId
   up     20241203080307  post     17.7         AddIssueEmailsNamespaceIdFk
   up     20241203080308  post     17.7         AddIssueEmailsNamespaceIdTrigger
   up     20241203080309  post     17.7         QueueBackfillIssueEmailsNamespaceId
   up     20241203081753  post     17.7         IndexIssuableMetricImagesOnNamespaceId
   up     20241203081754  post     17.7         AddIssuableMetricImagesNamespaceIdFk
   up     20241203081755  post     17.7         AddIssuableMetricImagesNamespaceIdTrigger
   up     20241203081756  post     17.7         QueueBackfillIssuableMetricImagesNamespaceId
   up     20241203144834  post     17.7         IndexSentryIssuesOnNamespaceId
   up     20241203144835  post     17.7         AddSentryIssuesNamespaceIdFk
   up     20241203144836  post     17.7         AddSentryIssuesNamespaceIdTrigger
   up     20241203144837  post     17.7         QueueBackfillSentryIssuesNamespaceId
   up     20241203172717  post     17.7         QueueBackfillSubscriptionAddOnPurchasesStartedAt
   up     20241204045404  post     17.7         PrepareAsyncIndexForPCiPipelinesTriggerId
   up     20241204050830  post     17.7         QueueBackfillPCiPipelinesTriggerId
   up     20241204130222  post     17.7         IdxProtectedBranchMergeAccessLevelsOnProtectedBranchProjectId
   up     20241204130223  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchProjectIdFk
   up     20241204130224  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchProjectIdTrigger
   up     20241204130225  post     17.7         QueueBackfillProtectedBranchMergeAccessLevelsProjectId
   up     20241204130227  post     17.7         IdxProtectedBranchMergeAccessLevelsOnProtectedBranchNamespaceId
   up     20241204130228  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchNamespaceIdFk
   up     20241204130229  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchNamespaceIdTrigger
   up     20241204130230  post     17.7         QueueBackfillProtectedBranchMergeAccessLevelsNamespaceId
   up     20241204133318  post     17.7         IndexProjectCiCdSettingsForPipelineRemoval
   up     20241204201518  post     17.7         QueueBackfillGroupWikiActivityEvents
   up     20241204223047  post     17.7         IndexCiBuildTraceMetadataOnProjectId
   up     20241204223441  post     17.7         QueueBackfillCiBuildTraceMetadataProjectId
   up     20241205120931  post     17.7         RemoveRolledupDatesWidgetFromWorkItemTypes
   up     20241205143057  post     17.7         IndexStatusPagePublishedIncidentsOnNamespaceId
   up     20241205143058  post     17.7         AddStatusPagePublishedIncidentsNamespaceIdFk
   up     20241205143059  post     17.7         AddStatusPagePublishedIncidentsNamespaceIdTrigger
   up     20241205143060  post     17.7         QueueBackfillStatusPagePublishedIncidentsNamespaceId
   up     20241205143233  post     17.7         AddIndexContainerRegistryProtectionTagRulesOnMinAccessLevel
   up     20241205211347  post     17.7         NewFkForPCiBuildsExecutionConfigsAndPCiPipelines
   up     20241205232318  post     17.7         FinalizeBackfillResourceLinkEvents
   up     20241206154945  post     17.7         QueueRequeueDeleteOrphanedGroups
   up     20241209121009  post     17.7         RemoveOrphanedPCiBuildsExecutionConfigs
   up     20241211042515  post     17.7         RequeueBackfillFreeSharedRunnersMinutesLimit
   up     20241211080728  post     17.7         AsyncValidateStagesPipelinesFk
   up     20241212111446  post     17.7         RemoveMigrateMembershipsColumn
   up     20241204074201  post     17.8         AddForeignKeyToNamespacesOnOrganizationId
   up     20241211072300  post     17.8         RetryAddFkFromPartitionedCiRunnerManagersToPartitionedCiRunners
   up     20241211202308  post     17.8         QueueBackfillOnboardingStatusRole

gitlab-rails console:

Gitlab::CurrentSettings.ci_job_token_signing_key.present?
ApplicationSetting.last.valid?

--------------------------------------------------------------------------------
 Ruby:         ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
 GitLab:       17.7.0-ee (1fd574ee571) EE
 GitLab Shell: 14.39.0
 PostgreSQL:   14.11
------------------------------------------------------------[ booted in 30.31s ]
WARNING: Active Record does not support composite primary key.

security_findings has composite primary key. Composite primary key is ignored.
Loading production environment (Rails 7.0.8.6)
irb(main):001:0> Gitlab::CurrentSettings.ci_job_token_signing_key.present?
=> false
irb(main):002:0> ApplicationSetting.last.valid?
=> true
irb(main):003:0>
irb(main):004:0*
=> nil
irb(main):005:0>

stanhu · December 23, 2024, 8:36am

Thanks for the info! It looks like 20241017160504 is listed there, but for some reason the migration didn’t generate a key.

stanhu · December 23, 2024, 8:37am

From which version were you upgrading? Were you all upgrading from 17.6.x?

Genva · December 23, 2024, 8:42am

Latest 17.6 version. For me GitLab updates are installed daily if available.

aaronjcopley · December 31, 2024, 5:42pm

I’ve also encountered this issue. Unfortunately, I performed the workaround before seeing @stanhu post requesting the 2 steps be done first.

gbschenkel · January 2, 2025, 8:23pm

gitlab-rake db:migrate:status

database: gitlabhq_production

 Status   Migration ID    Type     Milestone    Name
--------------------------------------------------
   up     20241017160107  regular  17.7         AddCiJobTokenSigningKeyToApplicationSettings
   up     20241017160504  regular  17.7         GenerateCiJobTokenSigningKey
   up     20241029130136  regular  17.7         AddWithHighestRolePlannerToUsersStatistics
   up     20241030031829  regular  17.7         AddResourceUsageLimitsToApplicationSettings
   up     20241104130731  regular  17.7         AddRoleApproversToApprovalMergeRequestRules
   up     20241105134416  regular  17.7         AddUsernamePasswordToVirtualRegistriesPackagesMavenUpstream
   up     20241105181558  regular  17.7         AddUniqIndexPackageReference
   up     20241106114853  regular  17.7         CreateSystemAccessGroupMicrosoftApplications
   up     20241106115015  regular  17.7         CreateSystemAccessGroupMicrosoftGraphAccessTokens
   up     20241106183051  regular  17.7         CreateAiConversationThreadsAndMessages
   up     20241108172443  regular  17.7         AddMigrateJenkinsBannerApplicationSetting
   up     20241111084209  regular  17.7         AddTemporaryIndicesForGitlabSubscriptionsCleanup
   up     20241111092153  regular  17.7         CleanTrialGitlabSubscriptionsDateAttributes
   up     20241111104411  regular  17.7         RemoveTemporaryIndicesForGitlabSubscriptionsCleanup
   up     20241111143504  regular  17.7         AddFkToAiConversationThreadsAndMessages
   up     20241111162911  regular  17.7         AddLegacyDestinationIdToExternalStreamingDestinations
   up     20241112055215  regular  17.7         AddPipelineVariablesDefaultRoleToNamespaceSettings
   up     20241112123436  regular  17.7         UpdateSeatControlInApplicationSettings
   up     20241113070714  regular  17.7         AddNamespaceSettingTokenExpiryInherited
   up     20241114113844  regular  17.7         AddLfkTriggersToNotes
   up     20241114232506  regular  17.7         MoveAllReadyZoektIndicesToPending
   up     20241115075017  regular  17.7         AddMemberRoleIdToProjectGroupLinks
   up     20241115121334  regular  17.7         AddDeletePipelinesInSecondsToProjectCiCdSettings
   up     20241117164341  regular  17.7         RemoveSecurityPoliciesDescriptionTextLimit
   up     20241119101703  regular  17.7         AddPostgresTableSizesView
   up     20241119103103  regular  17.7         AddPagesDefaultDomainRedirectToProjectSettings
   up     20241120200334  regular  17.7         DeleteRemoteDevelopmentAgentConfigs
   up     20241120210519  regular  17.7         AddAmazonQToAiSettings
   up     20241121033259  regular  17.7         AddFkToMemberRolesOnProjectGroupLinks
   up     20241121033758  regular  17.7         AddIndexToProjectGroupLinksOnMemberRoleId
   up     20241121071110  regular  17.7         AddDescriptionToPersonalAccessTokens
   up     20241121191751  regular  17.7         AddCreatedAtColumnToProjectComplianceFrameworkSettings
   up     20241121191754  regular  17.7         AddIndexToProjectComplianceFrameworkSettingsCreatedAt
   up     20241121192044  regular  17.7         AddIndexesForExternalStreamingDestinationUniqueNameScope
   up     20241122093316  regular  17.7         AddForeginKeyGroupMicrosoftGraphAccessTokens
   up     20241122093341  regular  17.7         AddForeginKeyApplicationMicrosoftGraphAccessTokens
   up     20241122121328  regular  17.7         AddApprovalPolicyActionIdxToApprovalProjectRules
   up     20241122121350  regular  17.7         AddApprovalPolicyActionIdxToApprovalMergeRequestRules
   up     20241122121652  regular  17.7         AddActionIdxToScanResultPolicies
   up     20241125133011  regular  17.7         AddGroupIdToGroupWikiRepositoryStates
   up     20241125140958  regular  17.7         AddMetadataToZoektEnabledNamespaces
   up     20241125145005  regular  17.7         AddNamespaceIdToIssuableSeverities
   up     20241125155301  regular  17.7         AddInstanceIntegrationIdColumnToIssueTrackerData
   up     20241125155621  regular  17.7         AddInstanceIntegrationIdColumnToZentaoTrackerData
   up     20241125155711  regular  17.7         AddInstanceIntegrationIdColumnToJiraTrackerData
   up     20241125173317  regular  17.7         AddProjectIdToCiBuildTraceChunks
   up     20241125222645  regular  17.7         AddStopSettingToEnvironments
   up     20241126103055  regular  17.7         AddMultipleNotNullConstraintToIssueTrackerData
   up     20241126104129  regular  17.7         AddMultipleNotNullConstraintToZentaoTrackerData
   up     20241126104222  regular  17.7         AddMultipleNotNullConstraintToJiraTrackerData
   up     20241126104505  regular  17.7         RemoveNotNullConstraintFromIssueTrackerData
   up     20241126104606  regular  17.7         RemoveNotNullConstraintFromZentaoTrackerData
   up     20241126104707  regular  17.7         RemoveNotNullConstraintFromJiraTrackerData
   up     20241126105922  regular  17.7         AddInstanceIntegrationFkToIssueTrackerData
   up     20241126110640  regular  17.7         AddInstanceIntegrationFkToZentaoTrackerData
   up     20241126110806  regular  17.7         AddInstanceIntegrationFkToJiraTrackerData
   up     20241126131337  regular  17.7         AddDefaultPermissionsToGroupScopeLinks
   up     20241126131359  regular  17.7         AddDefaultPermissionsToProjectScopeLinks
   up     20241126151227  regular  17.7         AddProjectIdToCiBuildPendingStates
   up     20241126152143  regular  17.7         AddIndexOnInstanceIntegrationIdToIssueTrackerData
   up     20241126162408  regular  17.7         AddIndexOnInstanceIntegrationIdToZentaoTrackerData
   up     20241126170110  regular  17.7         AddIndexOnInstanceIntegrationIdToJiraTrackerData
   up     20241127015108  regular  17.7         AddShardingKeyIdToUploads
   up     20241127092714  regular  17.7         CreateContainerRegistryProtectionTagRules
   up     20241127135043  regular  17.7         AddCompositeIdentityEnforcedToUsers
   up     20241127142223  regular  17.7         AddAssignDuoSeatsToSamlGroupLinks
   up     20241127161525  regular  17.7         AddDesignsAndDevelopmentWidgetsToTicketWorkItemType
   up     20241127164647  regular  17.7         TrackAiConversationThredRecordChanges
   up     20241128103042  regular  17.7         AddIndexSeatAssignmentNamespaceLastActivityOn
   up     20241128113732  regular  17.7         AddOrganizationToBulkImportEntities
   up     20241128141733  regular  17.7         CreateProjectsBranchRulesSquashOptions
   up     20241129135625  regular  17.7         AddFkForProjectsBranchRulesSquashOptionsProtectedBranch
   up     20241129135646  regular  17.7         AddFkForProjectsBranchRulesSquashOptionsProject
   up     20241129142800  regular  17.7         AddCheckConstraintToConanFileMetadata
   up     20241129160936  regular  17.7         AddStreamingDestinationRefToExternalAuditEventDestinations
   up     20241129161024  regular  17.7         AddStreamingDestinationRefToInstanceAuditEventDestination
   up     20241129161110  regular  17.7         AddStreamingDestinationRefToGoogleCloudLoggingConfiguration
   up     20241129161203  regular  17.7         AddStreamingDestinationRefToInstanceGoogleCloudLoggingConfiguration
   up     20241129161205  regular  17.7         ReplaceUniqueIndexOnScanResultPolicies
   up     20241129161213  regular  17.7         AddStreamingDestinationRefToAmazonS3Configuration
   up     20241129161223  regular  17.7         AddStreamingDestinationRefToInstanceAmazonS3Configuration
   up     20241201092435  regular  17.7         AddIndexToBuildSources
   up     20241201162318  regular  17.7         AddCustomRolesToScanResultPolicies
   up     20241202054640  regular  17.7         AddVulnerabilityEventsToWebHooks
   up     20241202140049  regular  17.7         AddNamespaceIdToIssueUserMentions
   up     20241202141407  regular  17.7         AddNamespaceIdToResourceLinkEvents
   up     20241202142250  regular  17.7         AddNamespaceIdToResourceWeightEvents
   up     20241202143619  regular  17.7         AddNamespaceIdToWorkItemProgresses
   up     20241202145035  regular  17.7         AddNamespaceIdToIncidentManagementIssuableEscalationStatuses
   up     20241202151532  regular  17.7         AddNamespaceIdToIssuableResourceLinks
   up     20241203073521  regular  17.7         AddNamespaceIdToIssueAssignmentEvents
   up     20241203074400  regular  17.7         AddNamespaceIdToIssueMetrics
   up     20241203075014  regular  17.7         AddNamespaceIdToIssueCustomerRelationsContacts
   up     20241203075616  regular  17.7         AddNamespaceIdToIssueEmailParticipants
   up     20241203080305  regular  17.7         AddNamespaceIdToIssueEmails
   up     20241203081752  regular  17.7         AddNamespaceIdToIssuableMetricImages
   up     20241203144833  regular  17.7         AddNamespaceIdToSentryIssues
   up     20241203213010  regular  17.7         DropIssuesCorrectWorkItemTypeIdDefault
   up     20241204035453  regular  17.7         AddPipelinesTriggerId
   up     20241204130221  regular  17.7         AddProtectedBranchProjectIdToProtectedBranchMergeAccessLevels
   up     20241204130226  regular  17.7         AddProtectedBranchNamespaceIdToProtectedBranchMergeAccessLevels
   up     20241204180053  regular  17.7         RemoveSoftwareLicenseExistenceConstraintToSoftwareLicensePolicies
   up     20241204215259  regular  17.7         AddBidirectionalSyncTriggerOnIssuesWorkItemTypeId
   up     20241204221832  regular  17.7         AddProjectIdToCiBuildTraceMetadata
   up     20241205143056  regular  17.7         AddNamespaceIdToStatusPagePublishedIncidents
   up     20241205191938  regular  17.7         AddAutopopulatedToCiJobTokenProjectScopeLink
   up     20241205191957  regular  17.7         AddAutopopulatedToCiJobTokenGroupScopeLink
   up     20241206150204  regular  17.7         AddCustomRolesConstraintToScanResultPolicies
   up     20241211124617  regular  17.7         AddZoektIndexOverprovisionedReadyHighWatermarkExceeded
   up     20241023084019  post     17.7         RemoveVirtualRegistriesPackagesMavenCachedResponsesDownloadsCountColumn
   up     20241024201454  post     17.7         ValidateProjectIdNotNullConstraintOnCiStages
   up     20241027152700  post     17.7         RequeueBackfillMilestoneReleasesProjectId
   up     20241107064635  post     17.7         QueueBackfillCiRunnerMachinesPartitionedTable
   up     20241108133603  post     17.7         QueueBackfillSeatAssignmentsTable
   up     20241111220006  post     17.7         UpdateUniqueIndexOnPackagesNugetSymbol
   up     20241112084921  post     17.7         RemoveMembersLastActivityOnColumn
   up     20241112163029  post     17.7         QueueDeleteOrphanedGroups
   up     20241113144244  post     17.7         QueueBackfillVulnerabilityNamespaceHistoricalStatistics
   up     20241113230840  post     17.7         DropBrokenFkForPCiBuildsUpstreamPipelineId
   up     20241114202257  post     17.7         QueueBackfillComplianceViolationNullTargetProjectIds
   up     20241114225103  post     17.7         AddIndexApprovalProjectRulesOnProjectIdRuleType
   up     20241114225309  post     17.7         AddIndexApprovalProjectRulesOnProjectIdConfigIdAndId
   up     20241114225508  post     17.7         AddIndexApprovalMrRulesOnProjectIdPolicyRuleIdAndId
   up     20241114225528  post     17.7         AddIndexScanResultPolicyViolationsOnProjectIdPolicyRuleIdAndId
   up     20241114231251  post     17.7         DropIndexApprovalMrRulesOnConfigIdAndPolicyRuleId
   up     20241114231606  post     17.7         DropIndexApprovalProjectRulesOnProjectId
   up     20241114231724  post     17.7         DropIndexScanResultPolicyViolationsOnProjectId
   up     20241115121337  post     17.7         AddIndexOnProjectSettingsForOldPipelines
   up     20241115122644  post     17.7         FinalizeDeleteOrphanedBuildRecords
   up     20241115133502  post     17.7         FinalizeCopyTaggingsToPCiBuildTags
   up     20241115140940  post     17.7         SyncPartitionFkForPCiBuildsExecutionConfigsAndPCiPipelines
   up     20241115211552  post     17.7         QueueBackfillSoftwareLicenseSpdxIdentifierForSoftwareLicensePolicies
   up     20241118121418  post     17.7         AddSyncIndexToSbomOccurrencesForSeverityAggregations
   up     20241118124301  post     17.7         RequeueBackfillSecurityPolicies
   up     20241118145604  post     17.7         QueueCreateMissingTerraformModuleMetadata
   up     20241119085925  post     17.7         RemoveNamespaceLimitsTemporaryStorageIncreaseEndsOnColumn
   up     20241119105148  post     17.7         RemoveFileChecksumFromXrayReports
   up     20241119143004  post     17.7         AsyncValidationForBuildsPipelinesFk
   up     20241120134610  post     17.7         SetNamespacesOrganizationIdNotNull
   up     20241120232540  post     17.7         FinalizeBackfillProjectRelationExportsProjectId
   up     20241121090148  post     17.7         RemoveNamespacePendingDeleteColumn
   up     20241121115756  post     17.7         PrepareNamespacesOrganizationIdNotNullValidation
   up     20241121140458  post     17.7         AddIndexToVulnerabilityIdentifiersOnProjectIdAndName
   up     20241122031025  post     17.7         ChangeOrganizationIdDefaultOauth
   up     20241125072626  post     17.7         FinalizeBackfillVulnerabilityFindingsRemediationsProjectId
   up     20241125085625  post     17.7         AddFkBetweenCiBuildsAndCiPipelines
   up     20241125125332  post     17.7         QueueBackfillFreeSharedRunnersMinutesLimit
   up     20241125133120  post     17.7         IndexGroupWikiRepositoryStatesOnGroupId
   up     20241125133216  post     17.7         AddGroupWikiRepositoryStatesGroupIdFk
   up     20241125133312  post     17.7         AddGroupWikiRepositoryStatesGroupIdTrigger
   up     20241125133627  post     17.7         QueueBackfillGroupWikiRepositoryStatesGroupId
   up     20241125145006  post     17.7         IndexIssuableSeveritiesOnNamespaceId
   up     20241125145007  post     17.7         AddIssuableSeveritiesNamespaceIdFk
   up     20241125145008  post     17.7         AddIssuableSeveritiesNamespaceIdTrigger
   up     20241125145009  post     17.7         QueueBackfillIssuableSeveritiesNamespaceId
   up     20241125173511  post     17.7         IndexCiBuildTraceChunksOnProjectId
   up     20241125174339  post     17.7         QueueBackfillCiBuildTraceChunksProjectId
   up     20241125232348  post     17.7         FinalizeDisableAllowRunnerRegistrationOnNamespaceSettingsForGitlabCom
   up     20241126151232  post     17.7         IndexCiBuildsPendingStatesOnProjectId
   up     20241126151234  post     17.7         QueueBackfillCiBuildPendingStatesProjectId
   up     20241126182756  post     17.7         DropWorkspacesDesiredConfigGeneratorVersionDefault
   up     20241127102904  post     17.7         ValidateNamespacesOrganizationIdNotNullConstraint
   up     20241127140830  post     17.7         AddDomainIdxToAlertManagementAlerts
   up     20241127151803  post     17.7         QueueMigrateSoftwareLicenseWithoutSpdxIdentifierToCustomLicenses
   up     20241127210044  post     17.7         BackfillUserDetails
   up     20241128071834  post     17.7         RemovePushTablesForcePushDeleteBranchRegex
   up     20241128102335  post     17.7         AsyncValidationForCiPipelinesFks
   up     20241128102925  post     17.7         RemoveBrokenFkBetweenCiPipelinesBuildConfigs
   up     20241128114137  post     17.7         SetOrganizationIdForBulkImportEntities
   up     20241128114319  post     17.7         AddMultiNotNullConstraintToBulkImportEntities
   up     20241129084249  post     17.7         AddIndexMemberRoleOnBaseAccessLevel
   up     20241129104415  post     17.7         CreateSubscriptionsIndexOnNoteableSynchronously
   up     20241129104531  post     17.7         CreateSentNotificationsIndexOnNoteableSynchronously
   up     20241129110758  post     17.7         DropSentNotificationsIndexOnNoteableId
   up     20241202112738  post     17.7         RevertOauthOrganizationDefault
   up     20241202140050  post     17.7         IndexIssueUserMentionsOnNamespaceId
   up     20241202140051  post     17.7         AddIssueUserMentionsNamespaceIdFk
   up     20241202140052  post     17.7         AddIssueUserMentionsNamespaceIdTrigger
   up     20241202140053  post     17.7         QueueBackfillIssueUserMentionsNamespaceId
   up     20241202141408  post     17.7         IndexResourceLinkEventsOnNamespaceId
   up     20241202141409  post     17.7         AddResourceLinkEventsNamespaceIdFk
   up     20241202141410  post     17.7         AddResourceLinkEventsNamespaceIdTrigger
   up     20241202141411  post     17.7         QueueBackfillResourceLinkEventsNamespaceId
   up     20241202142251  post     17.7         IndexResourceWeightEventsOnNamespaceId
   up     20241202142252  post     17.7         AddResourceWeightEventsNamespaceIdFk
   up     20241202142253  post     17.7         AddResourceWeightEventsNamespaceIdTrigger
   up     20241202142254  post     17.7         QueueBackfillResourceWeightEventsNamespaceId
   up     20241202143620  post     17.7         IndexWorkItemProgressesOnNamespaceId
   up     20241202143621  post     17.7         AddWorkItemProgressesNamespaceIdFk
   up     20241202143622  post     17.7         AddWorkItemProgressesNamespaceIdTrigger
   up     20241202143623  post     17.7         QueueBackfillWorkItemProgressesNamespaceId
   up     20241202145036  post     17.7         IndexIncidentManagementIssuableEscalationStatusesOnNamespaceId
   up     20241202145037  post     17.7         AddIncidentManagementIssuableEscalationStatusesNamespaceIdFk
   up     20241202145038  post     17.7         AddIncidentManagementIssuableEscalationStatusesNamespaceIdTrigger
   up     20241202145039  post     17.7         QueueBackfillIncidentManagementIssuableEscalationStatusesNamespaceId
   up     20241202151533  post     17.7         IndexIssuableResourceLinksOnNamespaceId
   up     20241202151534  post     17.7         AddIssuableResourceLinksNamespaceIdFk
   up     20241202151535  post     17.7         AddIssuableResourceLinksNamespaceIdTrigger
   up     20241202151536  post     17.7         QueueBackfillIssuableResourceLinksNamespaceId
   up     20241202170820  post     17.7         QueueBackfillIssueLinkIdOnRelatedEpicLinks
   up     20241203042054  post     17.7         AddProjectOrganizationIdNotNullConstraint
   up     20241203042134  post     17.7         PrepareProjectOrganizationIdNotNullValidation
   up     20241203073522  post     17.7         IndexIssueAssignmentEventsOnNamespaceId
   up     20241203073523  post     17.7         AddIssueAssignmentEventsNamespaceIdFk
   up     20241203073524  post     17.7         AddIssueAssignmentEventsNamespaceIdTrigger
   up     20241203073525  post     17.7         QueueBackfillIssueAssignmentEventsNamespaceId
   up     20241203074401  post     17.7         IndexIssueMetricsOnNamespaceId
   up     20241203074402  post     17.7         AddIssueMetricsNamespaceIdFk
   up     20241203074403  post     17.7         AddIssueMetricsNamespaceIdTrigger
   up     20241203074404  post     17.7         QueueBackfillIssueMetricsNamespaceId
   up     20241203075015  post     17.7         IndexIssueCustomerRelationsContactsOnNamespaceId
   up     20241203075016  post     17.7         AddIssueCustomerRelationsContactsNamespaceIdFk
   up     20241203075017  post     17.7         AddIssueCustomerRelationsContactsNamespaceIdTrigger
   up     20241203075018  post     17.7         QueueBackfillIssueCustomerRelationsContactsNamespaceId
   up     20241203075617  post     17.7         IndexIssueEmailParticipantsOnNamespaceId
   up     20241203075618  post     17.7         AddIssueEmailParticipantsNamespaceIdFk
   up     20241203075619  post     17.7         AddIssueEmailParticipantsNamespaceIdTrigger
   up     20241203075620  post     17.7         QueueBackfillIssueEmailParticipantsNamespaceId
   up     20241203080306  post     17.7         IndexIssueEmailsOnNamespaceId
   up     20241203080307  post     17.7         AddIssueEmailsNamespaceIdFk
   up     20241203080308  post     17.7         AddIssueEmailsNamespaceIdTrigger
   up     20241203080309  post     17.7         QueueBackfillIssueEmailsNamespaceId
   up     20241203081753  post     17.7         IndexIssuableMetricImagesOnNamespaceId
   up     20241203081754  post     17.7         AddIssuableMetricImagesNamespaceIdFk
   up     20241203081755  post     17.7         AddIssuableMetricImagesNamespaceIdTrigger
   up     20241203081756  post     17.7         QueueBackfillIssuableMetricImagesNamespaceId
   up     20241203144835  post     17.7         AddSentryIssuesNamespaceIdFk
   up     20241203144836  post     17.7         AddSentryIssuesNamespaceIdTrigger
   up     20241203144837  post     17.7         QueueBackfillSentryIssuesNamespaceId
   up     20241203172717  post     17.7         QueueBackfillSubscriptionAddOnPurchasesStartedAt
   up     20241204045404  post     17.7         PrepareAsyncIndexForPCiPipelinesTriggerId
   up     20241204050830  post     17.7         QueueBackfillPCiPipelinesTriggerId
   up     20241204130222  post     17.7         IdxProtectedBranchMergeAccessLevelsOnProtectedBranchProjectId
   up     20241204130223  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchProjectIdFk
   up     20241204130224  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchProjectIdTrigger
   up     20241204130225  post     17.7         QueueBackfillProtectedBranchMergeAccessLevelsProjectId
   up     20241204130227  post     17.7         IdxProtectedBranchMergeAccessLevelsOnProtectedBranchNamespaceId
   up     20241204130228  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchNamespaceIdFk
   up     20241204130229  post     17.7         AddProtectedBranchMergeAccessLevelsProtectedBranchNamespaceIdTrigger
   up     20241204130230  post     17.7         QueueBackfillProtectedBranchMergeAccessLevelsNamespaceId
   up     20241204133318  post     17.7         IndexProjectCiCdSettingsForPipelineRemoval
   up     20241204223047  post     17.7         IndexCiBuildTraceMetadataOnProjectId
   up     20241205120931  post     17.7         RemoveRolledupDatesWidgetFromWorkItemTypes
   up     20241205143058  post     17.7         AddStatusPagePublishedIncidentsNamespaceIdFk
   up     20241205143060  post     17.7         QueueBackfillStatusPagePublishedIncidentsNamespaceId
   up     20241205143233  post     17.7         AddIndexContainerRegistryProtectionTagRulesOnMinAccessLevel
   up     20241205211347  post     17.7         NewFkForPCiBuildsExecutionConfigsAndPCiPipelines
   up     20241205232318  post     17.7         FinalizeBackfillResourceLinkEvents
   up     20241206090205  post     17.7         RemoveRevisionColumnsFromPackagesConanFileMetadata
   up     20241206154945  post     17.7         QueueRequeueDeleteOrphanedGroups
   up     20241209120605  post     17.7         NewFkForPCiBuildsExecutionConfigsAndPCiPipelines2
   up     20241209121009  post     17.7         RemoveOrphanedPCiBuildsExecutionConfigs
   up     20241209121731  post     17.7         NewFkForPCiBuildsExecutionConfigsAndPCiPipelines3
   up     20241210202511  post     17.7         AddIndexGitlabSubscriptionHistoriesOnChangeTypeAndHostedPlanId
   up     20241211042515  post     17.7         RequeueBackfillFreeSharedRunnersMinutesLimit
   up     20241211080728  post     17.7         AsyncValidateStagesPipelinesFk
   up     20241212111446  post     17.7         RemoveMigrateMembershipsColumn
   up     20241204074201  post     17.8         AddForeignKeyToNamespacesOnOrganizationId
   up     20241211072300  post     17.8         RetryAddFkFromPartitionedCiRunnerManagersToPartitionedCiRunners
   up     20241211202308  post     17.8         QueueBackfillOnboardingStatusRole

gitlab-rails console:

--------------------------------------------------------------------------------
 Ruby:         ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
 GitLab:       17.7.0-ee (1fd574ee571) EE
 GitLab Shell: 14.39.0
 PostgreSQL:   14.11
------------------------------------------------------------[ booted in 43.84s ]
WARNING: Active Record does not support composite primary key.

security_findings has composite primary key. Composite primary key is ignored.
Loading production environment (Rails 7.0.8.6)
irb(main):003:0> Gitlab::CurrentSettings.ci_job_token_signing_key.present?
=> false
irb(main):004:0> ApplicationSetting.last.valid?
=> true

gbschenkel · January 2, 2025, 8:27pm

I migrated from v17.6.2 to v17.7.0, and also did after the db:migrade:down/up for get runners running again.
Using Oracle Linux v9.3 as host and running GitLab Omnibus on Podman v5.2.2.

stanhu · January 3, 2025, 3:56am

Thanks, everyone.

Please go ahead with the workaround, or wait for GitLab v17.7.1 to be released, which is targeted for Wednesday, January 8, 2025.

All runners failing after gitlab-fips package updated to 17.7.0 (500 err on server with "CI job token signing key is not set") (#38397) · Issues · GitLab.org / gitlab-runner · GitLab explains why this problem occurs, and Fix CI job token signing key not always generated (!176483) · Merge requests · GitLab.org / GitLab · GitLab should fix this in v17.7.1.

Automatically clear schema cache with DDL changes in migrations (!176524) · Merge requests · GitLab.org / GitLab · GitLab is also under review, and I think that should prevent similar issues from happening in the future.

beat · January 7, 2025, 6:59pm

Here, as requested, my results for this (upgraded from 17.6.latest to 17.7.0) before applying workaround:

...
   up     20241209121009  Remove orphaned p ci builds execution configs
   up     20241209121731  New fk for p ci builds execution configs and p ci pipelines3
   up     20241210202511  Add index gitlab subscription histories on change type and hosted plan
   up     20241211042515  Requeue backfill free shared runners minutes limit
   up     20241211072300  Retry add fk from partitioned ci runner managers to partitioned ci runners
   up     20241211080728  Async validate stages pipelines fk
   up     20241211124617  Add zoekt index overprovisioned ready high watermark exceeded
   up     20241211202308  Queue backfill onboarding status role
   up     20241212111446  Remove migrate memberships column



~# gitlab-rails console
--------------------------------------------------------------------------------
 Ruby:         ruby 3.2.5 (2024-07-26 revision 31d0f1a2e7) [x86_64-linux]
 GitLab:       17.7.0 (eedc7c560c9) FOSS
 GitLab Shell: 14.39.0
 PostgreSQL:   14.11
------------------------------------------------------------[ booted in 58.56s ]
Loading production environment (Rails 7.0.8.6)
irb(main):001:0> Gitlab::CurrentSettings.ci_job_token_signing_key.present?
=> true
irb(main):002:0> ApplicationSetting.last.valid?
=> true
irb(main):003:0>

stanhu · January 8, 2025, 6:23pm

GitLab 17.7.1 was released today to address this problem, as well as other security fixes and bugs: GitLab Patch Release: 17.7.1, 17.6.3, 17.5.5 | GitLab

I’ve confirmed that it fixes this issue automatically: All runners failing after gitlab-fips package updated to 17.7.0 (500 err on server with "CI job token signing key is not set") (#38397) · Issues · GitLab.org / gitlab-runner · GitLab

Thanks, everyone, for your help here.

herczegm · January 14, 2025, 8:21am

@stanhu : Can you confirm that I can upgrade to 17.7.1 from 17.6.1 directly? Is this upgrade history is valid? Upgrade Path

Thank you for your answer in advance!

stanhu · January 14, 2025, 8:37am

Yes, you can.

herczegm · January 15, 2025, 8:23am

Thanks! Successfully upgraded from 17.6.1 to 17.7.1 ce. Additional info in case of Runner upgrade, (if Gitlab repo is not available which is in my case) you need to separetly download and install the Gitlab Runner Helper Image and after the the actual GitlabRunner package.

stanhu · January 15, 2025, 4:32pm

Note that 17.7.2 was released today to address an issue with merge requests: GitLab Patch Release: 17.7.2 | GitLab

Topic		Replies	Views
After updating from version Gitlab 11.8.10 breaks runners CI Self-managed ci , runner	2	1773	September 25, 2019
CI_JOB_TOKEN sometime gives project not found GitLab CI/CD	1	821	August 20, 2020
"Can't verify CSRF token authenticity" on Gitlab Runner How to Use GitLab	0	1279	November 3, 2019
Runners not picking up jobs after upgrade from 15.6.0 to 15.8.1 Upgrade runner , upgrade	3	2017	March 17, 2023
CI Job Token Redirecting to Sign In Page GitLab CI/CD	0	17	August 7, 2024

17.7 Upgrade breaks Gitlab-Runner jobs with token not set error

Problem to solve

Steps to reproduce

Configuration

Versions

Helpful resources

Related topics