Suddenly appeared: This job failed because the necessary resources were not successfully created

Problem to solve

When attempting to run my deploy CI task, I am seeing the above error message. I have not changed anything in my gitlab-ci.yml since the last successful deploy, so I’m unsure whats going on here. I am running a local gitlab and used the kubernetes next-click-finish install.

It runs the following stages correctly:

build, package when it tries to deploy to my kubernetes cluster it suddenly fails.
I have changed nothing in my gitlab-ci.yml or on my cluster end. So why it fails baffles me.

It returns the above error almost instantly after starting the job.

Thoughts?
Describe your question in as much detail as possible:

Steps to reproduce

Whenever I run the script it fails. There is no indication what is going wrong.

Configuration

prod-deployment:
#  before_script:
#    - apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
#    - apt-get update && apt-get --only-upgrade install google-cloud-sdk-kpt google-cloud-sdk-anthos-auth google-cloud-sdk-firestore-emulator google-cloud-sdk-spanner-emulator google-cloud-sdk-package-go-module google-cloud-sdk-nomos google-cloud-sdk-skaffold google-cloud-sdk-config-connector google-cloud-sdk-app-engine-java google-cloud-sdk google-cloud-sdk-kubectl-oidc google-cloud-sdk-minikube google-cloud-sdk-local-extract google-cloud-sdk-app-engine-python google-cloud-sdk-cbt google-cloud-sdk-cloud-run-proxy google-cloud-sdk-pubsub-emulator google-cloud-sdk-app-engine-python-extras google-cloud-sdk-cloud-build-local google-cloud-sdk-terraform-tools google-cloud-sdk-harbourbridge google-cloud-sdk-bigtable-emulator google-cloud-sdk-datastore-emulator google-cloud-sdk-datalab kubectl google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-app-engine-go google-cloud-sdk-app-engine-grpc
  stage: deploy
  image: gcr.io/google.com/cloudsdktool/google-cloud-cli:latest
  script:
    - export DOCKER_HOST="tcp://localhost:2375"
    - echo "$GCP_SERVICE_KEY" > key.json
    - export USE_GKE_GCLOUD_AUTH_PLUGIN=True
    - gcloud auth activate-service-account --key-file=key.json
    - gcloud config set project bokkz-development-cluster
    - gcloud config set container/cluster ****
    - gcloud config set compute/zone europe-west4-c
    - gcloud container clusters get-credentials **** --zone europe-west4-c
    - export OTAP=***.***.***
    - export LABEL=prod
    - |
      if test -f "deployment.yaml"; then
          echo "Local Deployment exist"
          echo "Using Local configuration"
          sed -i "s/__CI_BUILD_REF_SLUG__/${CI_ENVIRONMENT_SLUG}/" deployment.yaml
          sed -i "s/__VERSION__/${CI_COMMIT_REF_NAME}/" deployment.yaml
          sed -i "s/__CI_PROJECT_NAME__/${CI_PROJECT_NAME}/" deployment.yaml
          sed -i "s/__OTAP__/${OTAP}/" deployment.yaml
          sed -i "s/__LABEL__/${LABEL}/" deployment.yaml      
          cat deployment.yaml
          if kubectl apply -f deployment.yaml || grep -q unchanged || grep -q configured; then
                echo "=> Patching deployment to force image update."
                kubectl patch -f deployment.yaml -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ci-last-updated\":\"$(date +'%s')\"}}}}}"
                kubectl rollout status -f deployment.yaml
            else
                echo "=> Deployment apply has changed the object, no need to force image update."
          fi
        else
          echo "Local Deployment does not exist"
          echo "Using global configuration"
          sed -i "s/__CI_BUILD_REF_SLUG__/${CI_ENVIRONMENT_SLUG-$CI_PROJECT_NAME}/" $SCRIPTS_DIR/manifest/deployment.yaml
          sed -i "s/__VERSION__/${CI_COMMIT_REF_NAME}/" $SCRIPTS_DIR/manifest/deployment.yaml
          sed -i "s/__CI_PROJECT_NAME__/${CI_PROJECT_NAME}/" $SCRIPTS_DIR/manifest/deployment.yaml
          sed -i "s/__OTAP__/${OTAP}/" $SCRIPTS_DIR/manifest/deployment.yaml
          sed -i "s/__LABEL__/${LABEL}/" deployment.yaml      
          cat $SCRIPTS_DIR/manifest/deployment.yaml
          if kubectl apply -f $SCRIPTS_DIR/manifest/deployment.yaml | grep -q unchanged; then
                echo "=> Patching deployment to force image update."
                kubectl patch -f $SCRIPTS_DIR/manifest/deployment.yaml -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ci-last-updated\":\"$(date +'%s')\"}}}}}"
            else
                echo "=> Deployment apply has changed the object, no need to force image update."
          fi
        kubectl rollout status -f $SCRIPTS_DIR/manifest/deployment.yaml
      fi
  only:
    - master
  environment:
    name: prod
    url: http://prod.***.**

Versions

Please select whether options apply, and add the version information.

  • [ X] Self-managed
  • GitLab.com SaaS
  • [X ] Self-hosted Runners

Versions

  • GitLab [14.1.6-ee]
  • GitLab Runner: 15.2.1

Maybe a race condition with multiple pipelines triggering the prod-deployment job. Can you show the full CI/CD job log to better understand the configuration if-then-else path? The script section checks for a deployment.yml and it is not clear to me where it originates from, and which errors are shown in the logs additionally.

GitLab 14.x is fairly old, too. I’d suggest upgrading to 17.x and consider using the agent for Kubernetes for deployment workflows.

The deployment is either part of the git repository. If it finds one it will use the local deployment. If it doesn’t find one it will revert to the ‘default’ one.

For example:

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  labels:
    my-super-app: "prod-algemeen-****-*****"
  name: "*****-*****-****-****"
  namespace: "alg"
spec:
  replicas: 1
  selector:
    matchLabels:
      my-super-app: "prod-****-****-****"
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: "RollingUpdate"
  template:
    metadata:
      labels:
        my-super-app: "prod-****-****-****"
    spec:
      containers:
        image: "gitlab.***.***:***/****/****-****-*****:master"
        imagePullPolicy: "Always"
        name: "prod-****-****-*****"
        ports:
        - containerPort: 8080
      imagePullSecrets:
      - name: "****"
      nodeSelector:
        workload-type: "prod"

It is kinda hard to get the logs as it says there is No job log.

Thank you for your help so far.

Thanks for the details. Maybe the Runner logs can help shed some light at the timestamp of the executed job.

It gets there during the stage build, the following logs from the runner:

Checking for jobs... received                       job=33995 repo_url=https://gitlab.****.****/BOK/****-****-****.git runner=1P4bQxnz
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
Job succeeded                                       duration=56.854775794s job=33995 project=186 runner=1P4bQxnz

The package stage:

Checking for jobs... received                       job=33996 repo_url=https://gitlab.****.****/BOK/****-****-*****.git runner=1P4bQxnz
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
ERROR: Could not create cache adapter               error=cache factory not found: factory for cache adapter "" was not registered
Job succeeded                                       duration=25.169341257s job=33996 project=186 runner=1P4bQxnz

But it seems it never ‘receives’ the deployment stage and it goes straight to error. The URL states it should be job 33997 but the runner never seems to pick it up or receive it.

A quick search for this error message led me to ERROR: Could not create cache adapter (#3802) · Issues · GitLab.org / gitlab-runner · GitLab where it says it might be a wrong message/cache configuration.

I’m not sure if it relates to your problem (or hides it). The fix was in 16.2, so a few hops in your upgrade path to 17.x. I’m not sure how much time you want to spend troubleshooting an unsupported version though - upgrading to 17.x will help eliminate potential bugs which is harder to say for 14/15.

I’m curious though, before finally saying to upgrade - how does the Runner config.toml look like (redact sensitive details like tokens). Maybe there is a cache setting that leads us somewhere.

Hi dnsmichi,

Those runners were installed from the gitlab instance using the default installation options provided by Gitlab.

As I didn’t see any way out. I’ve decided (as I had no option left anymore, and eventually you have to upgrade) to upgrade. So I did the long upgrade path to version 15.

And than I tried to run the pipeline again, and than it started to work again.

I’ve should have upgraded first. So that should be marked as the answer.

I know I should upgrade asap to 17. But I really really do not like the expiration of the tokens because that means I have to redesign our gitlab ci pipeline so I do not have to change the token every year when it expires. But for now I will upgrade to the latest 15 version.

Thank you for your help and cooperation, it is and was greatly appreciated.
Kind regards,
Arjan

1 Like

@bokkz you can extend the token expiry using this script from the Gitlab docs: GitLab token overview | GitLab