Merge pipeline runs before commit pipeline finishes yet explicit need breaks

GitLab CI/CD
1

I have the following stages:

stages:
  - lint
  - build
  - test
  - release
  - deploy
  - cleanup

where the release and deploy runs only after merge-request, aka merge-request-pipeline:

  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

The problem is that if the merge-request pipeline fails, and I push a new commit to fix it, the merge request pipeline starts running before the build has finished in the commit pipeline (see figure below), as is release runs before test.

However, when I tried to explicitly tell gitlab, test is needed it tells me:

‘release’ job needs ‘test’ job, but ‘test’ is not in any previous stage
When I used in release the following:

needs:
   - test

image
image748×235 22 KB

What is the solution?

Enclosed the full pipeline

script 
default:
  image: docker:24.0.5
  services:
    - docker:24.0.5-dind

stages:
  - lint
  - build
  - test
  - release
  - deploy
  - cleanup

lint:
  image: python:3.10-slim
  stage: lint
  script:
    - apt-get update && apt-get install -y git
    - echo "Linting..."
    - pip install pre-commit
    - pre-commit run --all-files

build:
  stage: build
  script:
    - echo -n $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - echo "Building Docker Image..."
    - docker compose pull $CI_REGISTRY_IMAGE:latest || true
    - docker compose build builder
    - echo "Pushing Docker Image to Registry..."
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  needs:
    - lint
test:
  stage: test
  script:
    - echo -n $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - echo "Running Tests..."
    - docker compose run test
  needs:
    - build

cleanup:
  stage: cleanup
  script:
    - echo -n $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - echo "Cleaning up Docker Image from Registry (if tests failed)..."
    - |
      echo "Test results: $CI_JOB_STATUS"
      if [ "$CI_JOB_STATUS" == "failed" ]; then
        docker rmi $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA || true
      fi

release:
  stage: release
  script:
    - echo CI_REGISTRY_IMAGE:CI_COMMIT_SHORT_SHA = $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - echo CI_MERGE_REQUEST_SOURCE_BRANCH_SHA	 = $CI_MERGE_REQUEST_SOURCE_BRANCH_SHA
    - echo -n $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
    - echo "Tagging Image as Latest..."
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  needs:
    - test

deploy:
  stage: deploy
  script:
    - echo "Deploying to Production..."
    - echo "DEPLOY_USER = $DEPLOY_USER"
    - echo "DEPLOY_SERVER = $DEPLOY_SERVER"
    - mkdir -p ~/.ssh  # Create the .ssh directory if it doesn't exist
    - echo "$DEPLOY_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - ssh-keyscan $DEPLOY_SERVER >> ~/.ssh/known_hosts
    - echo Materialize docker-compose.yml
    - apk add gettext
    - envsubst < docker-compose.yml > docker-compose-materialized.yml
    - echo "Copying Docker Compose file to remote server..."
    - scp -i ~/.ssh/id_rsa ./docker-compose-materialized.yml $DEPLOY_USER@$DEPLOY_SERVER:~/docker-compose.yml
    - echo "Logging in to Docker Registry $CI_REGISTRY on Remote Server..."
    - ssh -i ~/.ssh/id_rsa $DEPLOY_USER@$DEPLOY_SERVER "echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY"
    - echo "Pulling Docker Image..."
    - ssh -i ~/.ssh/id_rsa $DEPLOY_USER@$DEPLOY_SERVER "export VAR=value && docker compose pull"
    - echo "Starting new Docker container..."
    - ssh -i ~/.ssh/id_rsa $DEPLOY_USER@$DEPLOY_SERVER "docker compose up -d app"
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
2

You may be using the wrong syntax keyword needed. I have not found any reference to that word. It should probably be needs:.

Below are some examples from the documentation.

linux:build:
  stage: build
  script: echo "Building linux..."

linux:rspec:
  stage: test
  needs: ["linux:build"]
  script: echo "Running rspec on linux..."

  needs:
    - project: namespace/group/project-name
      job: build-1
      ref: main
      artifacts: true
3

yeah, I have used needs, sorry for the confusion

4

I think that changing the needs reference to the explicit list form might help even though it should work the way you have it. I would also recommend adding “-job” to the job names that match their stage to prevent confusion over if you are talking about a job or a stage.

needs: ["test-job"]