Job succeeded with exit code 1 in CI/CD

hacker619 · August 7, 2023, 10:03am

Hi,

I’m new to gitlab CI/CD.

Recently, I installed gitlab runner to enable CI/CD.

But the results from gitlab are always “Job succeeded” even though I put code “exit 1” in .gitlab-ci.yml

Could somebody help me to solve this issue? ToT?

I had too many trial-and-errors but failed.

Here is the detailed information of this issue.

Gitlab version : 12.1.3
Gitlab runner version : 12.2.0

.gitlab-ci.yml

gitlab-ci_yml414×502 7.64 KB

Actually, the above capture image is wrong. This issue was reproduced even when I don’t have “allow_failure” field in .gitlab-ci.yml file
gitlab-runner config.toml
gitlab CI/CD result

ci_cd_job_succeeded_with_exit_code_11234×822 45.8 KB

According to the upper image, you can see “Job succeeded” even though it executed “exit 1”

Thanks in advance.

dnsmichi · August 7, 2023, 5:15pm

allow_failure: true allows jobs to fail but succeed so that follow-up jobs in the stages can continue. This is useful to for example add jobs into the pipeline that would scan a Docker image. If not, they fail but the overall pipeline can still run ok (the container scanning template does exactly that).

For future questions, please attach the configuration as code formatting block too. That makes copy-paste easier in answers. Thanks!

hacker619 · August 7, 2023, 11:04pm

Hi dnsmichi

Thanks for your quick answer.

Just like you said, I removed allow_failure property but the symptom is still reproduced.
(I left an additional description for this below the screenshot 1 (.gitlab-ci.yml))

Here is what I’ve done until now.

Remove whole content in the script of build_linux except for “exit 1” => reproduced
Tried to use after script just like this => reproduced

build_linux:
    script:
        - exit 1
    after_script:
        - exit 1

Insert invalid phrase in .gitlab-ci.yml => failed(with “yaml invalid” tag in gitlab page)

Thanks.

hacker619 · August 8, 2023, 12:23am

Hi dnsmichi.

There is one more thing I want to share with you.

Actually, I successfully received error code from gitlab-runner for windows just like below.

Condition : Windows gitlab runner version : 16.0.1
.gitlab-ci.yaml(Windows)

stages:
  - build
  - test

variables:
  VCPKG_FOLDER: "d:\\GitLab-Runner\\vcpkg"
  VCPKG_TOOLCHAIN: "d:\\GitLab-Runner\\vcpkg\\scripts\\buildsystems\\vcpkg.cmake"
  INTEL_VARS: "c:\\Program Files (x86)\\Intel\\oneAPI\\setvars.bat"
  BUILD_ARGS: >
    -DCMAKE_TOOLCHAIN_FILE=$VCPKG_TOOLCHAIN
    -DVCPKG_TARGET_TRIPLET=x64-windows-static
    -G Ninja
    -A x64
    -DCMAKE_C_COMPILER=icx
    -DCMAKE_CXX_COMPILER=icx
    -DUSE_MSVC_CRTDBG=OFF
    -DDUMP_MUREX_JSON_LOG=OFF
    -DBUILD_FICC=ON
    -DBUILD_QMR=ON
    -DBUILD_EQ=ON
    -DUSE_ICX_WIN32=ON

build_windows:
  stage: build
  tags:
    - winrunner
  before_script:
    - chcp 437
    - set LC_ALL=en_US.UTF-8
#    - git config core.longpaths true
#    - cmake --version
#    - python setup.py develop


  script:
    - echo "Building Windows"
    - set VS2019INSTALLDIR="c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools"
    - '& $INTEL_VARS'
    - mkdir cmake-build-release
    - cd cmake-build-release
    - echo "cmake .. -B . $BUILD_ARGS"
    - '& cmake .. -B . $BUILD_ARGS'
    - '& cmake --build . -j 20 --verbose --config Release'

test_windows:
  before_script:
    - chcp 65001
  stage: test
  dependencies:
    - build_windows
  tags:
    - winrunner
  script:
    - bin/unittest.exe

build log

Running with gitlab-runner 16.0.1 (79704081)
  on HQIT-ELS14 6x4VzYVs, system ID: s_c6a13299c6c3
Preparing the "shell" executor
Using Shell (powershell) executor...
Preparing environment
Running on HQIT-ELS14...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in D:/GitLab-Runner/builds/6x4VzYVs/0/model/kbp/.git/
Checking out 717e4ffe as detached HEAD (ref is 232-gitlab-runner-client-setting)...
Removing cmake-build-release/
git-lfs/2.12.1 (GitHub; windows amd64; go 1.14.10; git 85b28e06)

Skipping Git submodules setup
Executing "step_script" stage of the job script
$ chcp 437
Active code page: 437
$ set LC_ALL=en_US.UTF-8
$ echo "Building Windows"
Building Windows
$ set VS2019INSTALLDIR="c:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools"
$ & $INTEL_VARS
:: initializing oneAPI environment...
   initializing Visual Studio command-line environment...
:: ERROR: Visual Studio not found in "C:\Program Files (x86)\Microsoft Visual Studio\<Year>\<Edition>"
   Set VS2017INSTALLDIR or VS2019INSTALLDIR env vars to point to your installation and try again.
   Visual Studio environment was not configured.
:  compiler -- latest

ERROR: Visual Studio 2017 or 2019 is not found in "C:\Program Files (x86)\Microsoft Visual Studio\<2017 or 2019>\<Edition>", please set VS2017INSTALLDIR or VS2019INSTALLDIR
:  debugger -- latest
:  dev-utilities -- latest
:  dpl -- latest
:  mkl -- latest
:  tbb -- latest
:: oneAPI environment initialized ::
$ mkdir cmake-build-release


    ????: D:\GitLab-Runner\builds\6x4VzYVs\0\model\kbp


Mode                LastWriteTime         Length Name                          
----                -------------         ------ ----                          
d-----       2023-07-28   ?? 2:40                cmake-build-release           
$ cd cmake-build-release
$ echo "cmake .. -B . $BUILD_ARGS"
cmake .. -B . -DCMAKE_TOOLCHAIN_FILE=d:\GitLab-Runner\vcpkg\scripts\buildsystems\vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows-static -G Ninja -A x64  -DCMAKE_C_COMPILER=icx  -DCMAKE_CXX_COMPILER=icx  -DUSE_MSVC_CRTDBG=OFF -DDUMP_MUREX_JSON_LOG=OFF -DBUILD_FICC=ON -DBUILD_QMR=ON -DBUILD_EQ=ON -DUSE_ICX_WIN32=ON

$ & cmake .. -B . $BUILD_ARGS
-- Building for: Visual Studio 16 2019
CMake Error at cmake/modules/color_message.cmake:17 (_message):
  Could not find toolchain file:
  d:\GitLab-Runner\vcpkg\scripts\buildsystems\vcpkg.cmake
  -DVCPKG_TARGET_TRIPLET=x64-windows-static -G Ninja -A x64
  -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icx -DUSE_MSVC_CRTDBG=OFF
  -DDUMP_MUREX_JSON_LOG=OFF -DBUILD_FICC=ON -DBUILD_QMR=ON -DBUILD_EQ=ON
  -DUSE_ICX_WIN32=ON

  
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.26/Modules/CMakeDetermineSystem.cmake:154 (message)
  CMakeLists.txt:11 (project)


-- Configuring incomplete, errors occurred!
CMake Warning:
  Value of CMAKE_TOOLCHAIN_FILE contained a newline; truncating


ERROR: Job failed: exit status 1

There are some differences between linux log and windows log when I check the logs in gitlab webpage.
In linux log, the error logs are not highlighted.(you can check it in the screenshot 3 of main content.)
But in windows log, the error log (“Could not find toolchain file: …” ) is highlighted in red color.

I’m not sure how gitlab recognize this and make the difference.

But I share the information with you for your better understanding.

Thanks.

balonik · August 8, 2023, 6:24am

I’d say the coloring comes from the shell itself.

But I was not able to reproduce what you are reporting. I have created job

build_linux:
    script:
        - exit 1

and it fails as expected

Running with gitlab-runner 16.1.0~beta.59.g83c66823 (83c66823)
  on blue-5.saas-linux-small-amd64.runners-manager.gitlab.com/default -AzERasQ, system ID: s_4cb09cee29e2
  feature flags: FF_USE_IMPROVED_URL_MASKING:true
Resolving secrets
00:00
Preparing the "docker+machine" executor
00:17
Using Docker executor with image ruby:3.1 ...
Pulling docker image ruby:3.1 ...
Using docker image sha256:11309a2e746029e332998bea5ac94af2325662becce19c744027f49ebad51673 for ruby:3.1 with digest ruby@sha256:1d1309bd0f3dd0d7c8bd50aaeb9f10608002cf99eea493a7670000a1b71164c0 ...
Preparing environment
00:05
Running on runner--azerasq-project-38655043-concurrent-0 via runner-azerasq-s-l-s-amd64-1691475479-e81ae7d4...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/_redacted_/.git/
Created fresh repository.
Checking out f333f2f3 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:01
Using docker image sha256:11309a2e746029e332998bea5ac94af2325662becce19c744027f49ebad51673 for ruby:3.1 with digest ruby@sha256:1d1309bd0f3dd0d7c8bd50aaeb9f10608002cf99eea493a7670000a1b71164c0 ...
$ exit 1
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

hacker619 · August 8, 2023, 6:38am

Hi balonik.

Yeap, I don’t understand too.

To check the issue further, I installed gitlab and gitlab-runner on another machine and confirmed it works well.
Maybe I need to try something more.

Thanks.

dnsmichi · August 8, 2023, 6:34pm

Error highlighting depends on the called scripts, and how they use terminal colors to format error messages. Linux and windows support different shells where the called scripts need to be compatible with. You can also change the message color formatting.

I have not used PowerShell in while … lets look what the command does in an isolated way.

$ & cmake .. -B . $BUILD_ARGS
-- Building for: Visual Studio 16 2019
CMake Error at cmake/modules/color_message.cmake:17 (_message):
  Could not find toolchain file:
  d:\GitLab-Runner\vcpkg\scripts\buildsystems\vcpkg.cmake
  -DVCPKG_TARGET_TRIPLET=x64-windows-static -G Ninja -A x64
  -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icx -DUSE_MSVC_CRTDBG=OFF
  -DDUMP_MUREX_JSON_LOG=OFF -DBUILD_FICC=ON -DBUILD_QMR=ON -DBUILD_EQ=ON
  -DUSE_ICX_WIN32=ON

  
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.26/Modules/CMakeDetermineSystem.cmake:154 (message)
  CMakeLists.txt:11 (project)

The ampersand character & looks suspicous in the beginning. On Linux, you would use an ampersand at the end to send a script command into the background. On Windows with Powershell this has a different meaning. Googled for powershell ampersand and learned that this is the call operator, so calling a function. What does the & symbol in powershell mean? - Stack Overflow

IIRC calling a function hides the returned error codes, and even when there is an error, Powershell will continue. Found How to automatically stop powershell if ampersand command fails. example: & $cmd # now it fails - Stack Overflow which explains that you need to add manual error handling.

Something like this should capture the error in CI/CD. Or you remove the ampersand in the beginning (not sure why it is there in the first place).

& cmake .. -B . $BUILD_ARGS
if ($? -eq $false) {
    write-host -background DarkBlue -foreground Red "<Error Exit>"
    exit 1 
}

DerekGao · August 15, 2023, 12:24pm

Hi. you use dependencies in job test_linux. But you don’d define artifacts in job bulid_linux. try to define it.
dependencies

Topic		Replies	Views
Job getting succeeded even when there is an error in gitlab-ci GitLab CI/CD ci , pipelines	3	4201	July 14, 2023
GitLab exit codes GitLab CI/CD ci	1	11375	December 5, 2023
My gitlab-ci.yml job failed (ERROR: Job failed: exit status 1) without enough debug information GitLab CI/CD ci , archlinux , docker , logs	1	10539	July 11, 2023
Gitlab Runner jobs fails with exit code 1 on Ubuntu 20.04 GitLab CI/CD runner , ubuntu	0	2452	March 26, 2021
[Resolved] Job exit code 1 everytime at the end of script GitLab CI/CD	2	6224	July 12, 2018

Job succeeded with exit code 1 in CI/CD

Related topics