Results are not getting generated if test runs for long hour and result file is huge in CI/CD

Infrastructure as Code & Cloud Native
1

Issue :-

Whenever the automation scripts runs for more hours in CI/CD (example: more then 10 hrs) the reports are not getting generated from the GitLab.
Whereas same works fine when test runs for just couple of hours.

.gitlab-ci.yml

variables:
BRANCH: prime.stabilization
TEST_SUITE_NAME: MMR
REGION: us-west-2
DB_PREFIX: mmr-
RDS_SECRET_SQL_SERVER: arn:aws:secretsmanager:us-east-2:039721755773:secret:SQLDBSecret4ADF657C-JJPDzxFqWWgs-b9RFug
S3_DB_BACKUPS: scmvpc-v2-rdsgoldimages93f77924-7z2dtg1tg9hl
IL_USER: new.info9f
IL_PASSWORD: ids123
ZELENIUM_URL: zalmmr.devcloud2.idsgrp.com
ENV_NAME: mmrtest
BUILD_NUMBER: latest

stages:

  • restore_db
  • deploy_env
  • test
  • cleanup

restore_db:
stage: restore_db
image: srahouski/dockerawscli
tags:

  • scmvpc2
    services:
  • docker:18.09.7-dind
    variables:
    GOLD_IMAGE: il-util-sqlgold
    DOCKER_HOST: tcp://localhost:2375
    script:
  • SCHEMA=$(echo MMR_${TEST_SUITE_NAME}_${BRANCH##*.} | tr ‘[:lower:]’ ‘[:upper:]’)
  • echo Restoring test database $SCHEMA from backup [$DB_PREFIX]
  • eval $(/root/.local/bin/aws ecr get-login --region $REGION --no-include-email | sed ‘s|https://||’)
  • ECR_REPO=$(aws --region $REGION ecr describe-repositories --query “repositories[?repositoryName==’$GOLD_IMAGE’].repositoryUri” --output text)
  • docker pull $ECR_REPO
  • docker run --network=host --rm $ECR_REPO:latest ${RDS_SECRET_SQL_SERVER} ${SCHEMA} ${DB_PREFIX} ${S3_DB_BACKUPS} DROP
  • echo “SUCCESS” > .job_status
    only:
    • master

deploy_env:
stage: deploy_env
image: srahouski/dockerawscli
tags:

  • scmvpc2
    script:
  • echo Deploying environment
  • SCHEMA=$(echo MMR_${TEST_SUITE_NAME}_${BRANCH##*.} | tr ‘[:lower:]’ ‘[:upper:]’)
  • |
    POST_DATA="{
    “branch”: “${BRANCH}”,
    “buildNumber”: “${BUILD_NUMBER}”,
    “dbSchema”: “${SCHEMA}”,
    “dbBackup”: “${DB_PREFIX}”,
    “selectDbServer”: “${RDS_SECRET_SQL_SERVER}”,
    “environmentName”: “${ENV_NAME}”,
    “owner”: “GitLab-${CI_PROJECT_NAME}” }"
  • wget -q -O- idsgrp.com --post-data “$POST_DATA” --header=‘Content-Type:application/json’
  • |
    ENV_URL=“http://${ENV_NAME}.devcloud.idsgrp.com”
    echo “Waiting for the service to start ${ENV_URL}”
    attempt_counter=0
    max_attempts=80
    while [ “$(wget -q -O- --no-cache ${ENV_URL}/status?appServer=il)” != “OK” ]; do
    if [ $attempt_counter -eq $max_attempts ];then
    echo “Max attempts reached”
    exit 1
    fi
    printf ‘.’
    attempt_counter=$(($attempt_counter+1))
    sleep 10
    done
  • echo “SUCCESS” > .job_status
    only:
    • master

test:
tags:

  • scmvpc2
    cache:
    paths:
    • .m2/repository
      artifacts:
      when: always
      name: cucumber-reports
      expire_in: 6 day
      paths:
    • target/cucumber-html-reports/**
    • mmr-job.log
      variables:
      MAVEN_OPTS: “-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository”
      stage: test
      image: maven:3-jdk-11
      script:
  • sh run.sh http://${ENV_NAME}.devcloud.idsgrp.com ${IL_USER} ${IL_PASSWORD} ${ZELENIUM_URL} | tee -a mmr-job.log
  • echo “CLEANUP” > .job_status
    only:
    • master

before_script:

  • echo “FAIL” > .job_status

cleanup:
stage: cleanup
image: srahouski/dockerawscli
tags:

  • scmvpc2
    services:
  • docker:18.09.7-dind
    variables:
    GOLD_IMAGE: il-util-dropdb
    DOCKER_HOST: tcp://localhost:2375
    script:
  • echo “Shutting down environment ${ENV_NAME}”
  • wget -q -O- idsgrp.com --post-data env=${ENV_NAME}
  • SCHEMA=$(echo MMR_${TEST_SUITE_NAME}_${BRANCH##*.} | tr ‘[:lower:]’ ‘[:upper:]’)
  • echo “Droping test database $SCHEMA”
  • eval $(/root/.local/bin/aws ecr get-login --region $REGION --no-include-email | sed ‘s|https://||’)
  • ECR_REPO=$(aws --region $REGION ecr describe-repositories --query “repositories[?repositoryName==’$GOLD_IMAGE’].repositoryUri” --output text)
  • docker pull $ECR_REPO
  • docker run --network=host --rm $ECR_REPO:latest ${RDS_SECRET_SQL_SERVER} ${SCHEMA} DROP
  • echo “SUCCESS” > .job_status
    when: always
    only:
    • master
2

Hi @lkumar1
do you mean Job output?