The output is too lengthy, causing the GitLab Runner to mark the pipeline as failed

GitLab CI/CD
1

Problem to solve

Hello,

The output is too lengthy, causing the GitLab Runner to mark the pipeline as failed, even though I have already set the output_limit in the GitLab Runner configuration file to 1,024,000 (1 GB):

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ci-u22-x64-13"
  url = "http://gitlab.tpt.com"
  id = 0
  token = "QjpLR-BRMRMUXUxvVjuu"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  builds_dir = "/gitlab-runner/builds"
  output_limit = 1024000
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

The pipeline I’m running is for compiling LLVM code, which is a substantial task that generates a large amount of output throughout the process. I’ve tried redirecting the compile output to a file, which ends up being a 200MB log file with 1.8 million lines upon completion. However, using cat to output this file in the pipeline also causes the pipeline to fail, even though I’ve already set the output_limit to 1GB.

$ ls -alh build.log 
-rw-rw-r-- 1 gitlab-runner gitlab-runner 215M Oct 25 12:25 build.log
$ cat build.log  | wc -l
1804934

I know that I could export the log file using artifacts, but since the compilation process takes several hours, redirecting the output entirely to a log file means I won’t be able to see the progress during those hours—for instance, which file is currently being compiled.

Steps to reproduce

Which troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?

You can generate a large file with the following bash script and print it using cat command in pipeline:

#!/bin/bash

# Output file name
output_file="large_file.txt"

# Number of lines to generate
line_count=1800000

# Clear the output file
> "$output_file"

# Generate lines
for i in $(seq 1 $line_count); do
  echo "This is line $i" >> "$output_file"
done

echo "File $output_file has been generated with $line_count lines."

Configuration

.gitlab-ci.yml:

stages:          # List of stages for jobs, and their order of execution
  - build

build-job:       # This job runs in the build stage, which runs first.
  stage: build
  script:
    - cat build.log
  tags:
    -  ubuntu22-x64-old
  artifacts:
    expire_in: 2 days
    paths:
      - build.log

/etc/gitlab-runner/config.toml:

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ci-u22-x64-13"
  url = "http://gitlab.tpt.com"
  id = 0
  token = "QjpLR-BRMRMUXUxvVjuu"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "shell"
  builds_dir = "/gitlab-runner/builds"
  output_limit = 1024000
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Versions

Please select whether options apply, and add the version information.

  • Self-managed
  • GitLab.com SaaS
  • Dedicated
  • Self-hosted Runners

Versions

  • GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info): JiHu GitLab v17.4.0-jh
  • GitLab Runner, if self-hosted (Web /admin/runners or CLI gitlab-runner --version):
# gitlab-runner --version
Version:      17.5.2
Git revision: c6eae8d7
Git branch:   17-5-stable
GO version:   go1.22.7
Built:        2024-10-22T14:57:46+0000
OS/Arch:      linux/amd64

Really appreciate!

2

Do you have to print the whole log? Wouldn’t the last x lines be enough for you?

You can then offer the complete log output via artifacts.

3

Yes, I need to see the compilation in detail to be able to trace the cause when an error is reported, but this often triggers log restrictions on the job such as:

UNSUPPORTED: LLVM :: CodeGen/AMDGPU/GlobalISel/combine-fabs-fneg.mir (6437 of 52212)
UNSUPPORTED: LLVM :: CodeGen/AMDGPU/GlobalISel/combine-fcanonicalize.mir (6438 of 52212)
UNSUPPORTED: LLVM :: CodeGen/AMDGPU/GlobalISel/combine-fdiv-sqrt-to-rsq.mir (6439 of 52212)
UNSUPPORTED: LLVM :: CodeGen/AMDGPU/GlobalISel/combine-fma-add-ext-fma.ll (6440 of 52212)
UNSUPPORTED: LLVM :: CodeGen/AMDGPU/GlobalISel/combine-f
e[33;1mJob's log exceeded limit of 4194304 bytes.
Job execution will continue but no more output will be collected.e[0;m
4

Is there a parameter to change the verbosity level?

As far, I understand you, you only want to see how far the LLVM is and where it might crash.
tbh I would sacrifice the option to see how far it is and just take the option to see the error with tail.

I found the following issue on gitlab: How to handle very large build logs (#1362) · Issues · GitLab.org / gitlab-runner · GitLab
I think there is no direct solution without reducing the amount of output logs