GitLab CE - Docker Setup - Problem with CI/CD Artifacts

GitLab CI/CD
, , ,
1

Problem to solve

CI/CD Pipeline fails while uploading artifacts:
My ci/cd pipeline seems to work.
But when the job which should upload artifacts ends there is following log:
Runner log (Gitlab)

app-web/node_modules/: found 97754 matching artifact files and directories 
app-web/.next/: found 4332 matching artifact files and directories 
untracked: found 93477 files                       
ERROR: Uploading artifacts as "archive" to coordinator... error  error=couldn't execute POST against http://gitlab-ce/api/v4/jobs/865/artifacts?artifact_format=zip&artifact_type=archive: Post "http://gitlab-ce/api/v4/jobs/865/artifacts?artifact_format=zip&artifact_type=archive": dial tcp: lookup gitlab-ce on 127.0.0.53:53: server misbehaving id=865 token=glcbt-64
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
ERROR: Uploading artifacts as "archive" to coordinator... error  error=couldn't execute POST against http://gitlab-ce/api/v4/jobs/865/artifacts?artifact_format=zip&artifact_type=archive: Post "http://gitlab-ce/api/v4/jobs/865/artifacts?artifact_format=zip&artifact_type=archive": dial tcp: lookup gitlab-ce on 127.0.0.53:53: server misbehaving id=865 token=glcbt-64
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
ERROR: Uploading artifacts as "archive" to coordinator... error  error=couldn't execute POST against http://gitlab-ce/api/v4/jobs/865/artifacts?artifact_format=zip&artifact_type=archive: Post "http://gitlab-ce/api/v4/jobs/865/artifacts?artifact_format=zip&artifact_type=archive": dial tcp: lookup gitlab-ce on 127.0.0.53:53: server misbehaving id=865 token=glcbt-64
FATAL: invalid argument                            
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Runner log (Docker)

Appending trace to coordinator...ok                 code=202 job=865 job-log=0-101716 job-status=running runner=PzqZxs6pk sent-log=101643-101715 status=202 Accepted update-interval=3s
Appending trace to coordinator...ok                 code=202 job=865 job-log=0-101744 job-status=running runner=PzqZxs6pk sent-log=101716-101743 status=202 Accepted update-interval=3s
Appending trace to coordinator...ok                 code=202 job=865 job-log=0-101988 job-status=running runner=PzqZxs6pk sent-log=101744-101987 status=202 Accepted update-interval=3s
Updating job...                                     bytesize=101988 checksum=crc32:27c03c48 job=865 runner=PzqZxs6pk
Submitting job to coordinator...ok                  bytesize=101988 checksum=crc32:27c03c48 code=200 job=865 job-status=running runner=PzqZxs6pk update-interval=0s
Appending trace to coordinator...ok                 code=202 job=865 job-log=0-103678 job-status=running runner=PzqZxs6pk sent-log=101988-103677 status=202 Accepted update-interval=3s
Appending trace to coordinator...ok                 code=202 job=865 job-log=0-104445 job-status=running runner=PzqZxs6pk sent-log=103678-104444 status=202 Accepted update-interval=3s
Appending trace to coordinator...ok                 code=202 job=865 job-log=0-105708 job-status=running runner=PzqZxs6pk sent-log=104445-105707 status=202 Accepted update-interval=3s
WARNING: Job failed: exit code 1
                   duration_s=288.744818242 job=865 project=1 runner=PzqZxs6pk
Appending trace to coordinator...ok                 code=202 job=865 job-log=0-105752 job-status=running runner=PzqZxs6pk sent-log=105708-105751 status=202 Accepted update-interval=3s
Updating job...                                     bytesize=105752 checksum=crc32:11e777ee job=865 runner=PzqZxs6pk
WARNING: Submitting job to coordinator... job failed  bytesize=105752 checksum=crc32:11e777ee code=200 job=865 job-status=failed runner=PzqZxs6pk status=200 OK update-interval=0s
Removed job from processing list                    builds=0 job=865 max_builds=1 project=1 repo_url=https://public-url.de/dev/app.git time_in_queue_seconds=1

Is there a way to solute it?

T

PS: Has somebody a hint how I could test the pipeline? Currently I always push a new commit and tag to main which triggers the pipeline execution. But this is very annoying, because I have to remove these test commits, test tags and pipeline runs after that.
At the end the first pipeline run never works and I have to adjust some stuff even if the formatting is correct.

Configuration

CI/CD job:

build-frontend-nextjs:
  stage: build
  image: $FRONTEND_BASE_IMAGE
  retry: 2
  before_script:
    - apt-get update && apt-get install -y build-essential libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev curl
    - corepack enable
    - cd ./app-web/
    - rm -fr node_modules
    - export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
  script:
    - npm ci
    - npm run build
  rules:
    - if: "$CI_COMMIT_TAG"
      changes:
        - ./app-web/**
  artifacts:
    untracked: true
    paths:
      - app-web/node_modules/
      - app-web/.next/

Versions

  • [ x] Self-hosted Runners

Versions

  • GitLab (Web: /help or self-managed system information): 17.4
  • GitLab Runner, if self-hosted (Web /admin/runners or CLI gitlab-runner --version): Version 17.4.0

Edit:
One further information:
The docker runner is in network mode “host”.
This is because my pipeline should run tests which starts docker containers. The test further bind a port to that docker container and the whole test execution assumes http://localhost:<rng-binded-port> works.
This does not work in network_mode bridge. Because of that I chose network mode host which fixed that problem.
But maybe this makes problem while gitlab-runner talks with gitlab-ce container ?

Runner config:

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "node"
  url = "http://gitlab-ce"
  id = 1
  token = "token"
  token_obtained_at = 2024-02-07T13:36:13Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  clone_url = "https://git.mydomain.de"
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = true
    image = "node:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0

[[runners]]
  name = "Docker Image Builder"
  url = "http://gitlab-ce"
  id = 3
  token = "token"
  token_obtained_at = 2024-08-31T22:28:22Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = true
    image = "docker:stable"
    privileged = true
    network_mode="host"
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/tmp/:/tmp/"]
    shm_size = 0
    network_mtu = 0
5

Okay if somebody else run into this problem, I could provide a solution which worked for me, but I’m still not 100% sure why I had that issue.

Because my gitlab runner is in “host” network_mode I thought this could be the reason.
But the container can still reach the internal docker ip address of the gitlab instance. Even from my host system I can reach it, so this is not the problem.

Further I thought just to mount port 80 of gitlab to localhost:8080 could solute the problem.
But surprisingly I get following error message:

gitlab-runner  | WARNING: Checking for jobs... failed                runner=PzqZxs6pk status=couldn't execute POST against http://localhost:8080/api/v4/jobs/request: Post "http://localhost:8080/api/v4/jobs/request": dial tcp [::1]:8080: connect: connection refused

This behavior with localhost and the behavior of the docker domain name (gitlab-ce) is strange.
So I tried to add the public gitlab address (https://mydomain.de) to my runners “url” property in the config.toml.
That did not work, because my docker-compose file had the host_name defined, so the internal dns received the docker ip for that public domain and it tried to reach <docker-ip>:443 which is not open, because my gitlab is behind a reverse proxy which does the https requests.
So this did not work.

BUT:
After I removed hostname: mydomain.def rom the docker compose file it routed correctly to my gitlab instance and it worked again. Its sad that a application which is in the same network needs to use the reverse proxy and in the worst case go out the system and comes in again, but what should I say…
At least it works.