GitLab K8s Runner fails for get_sources

Infrastructure as Code & Cloud Native
,
1

Hi all,
we are trying to move gitlab-runners from standard CentOS VMs to kebernetes.
But after setup and registration, pipeline fails with unknown error:

Running with gitlab-runner 15.7.0 (259d2fd4)
  on Kubernetes-local JXRw3mH1
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image gitlab-test.domain:5005/image:latest ...
Using attach strategy to execute scripts...
Preparing environment
00:04
Waiting for pod gitlab-runner/runner-jxrw3mh1-project-290-concurrent-0dpd88 to be running, status is Pending
Running on runner-jxrw3mh1-project-290-concurrent-0dpd88 via gitlab-runner-d7df6c548-hsgxg...
Getting source from Git repository
00:00
error: could not lock config file /root/.gitconfig: Read-only file system
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Inside the log of the job pod we found:

 helper Running on runner-jxrw3mh1-project-290-concurrent-0dpd88 via gitlab-runner-d7df6c548-hsgxg...                                                      
 helper                                                                                                     
 helper {"command_exit_code": 0, "script": "/scripts-290-207166/prepare_script"}                                                                 
 helper error: could not lock config file /root/.gitconfig: Read-only file system                                                                
 helper                                                                                                     
 helper {"command_exit_code": 1, "script": "/scripts-290-207166/get_sources"}                                                                  
 helper                                                                                                     
 helper {"command_exit_code": 0, "script": "/scripts-290-207166/cleanup_file_variables"}

Inside the log of the gitlab-runner pod we found:

 Starting in container "helper" the command ["gitlab-runner-build" "<<<" "/scripts-290-207167/get_sources" "2>&1 | tee -a /logs-290-207167/output.log"] with script: #!/usr/bin/env bash
 if set -o | grep pipefail > /dev/null; then set -o pipefail; fi; set -o errexit
 set +o noclobber
 : | eval $'export FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=$\'false\'\nexport FF_NETWORK_PER_BUILD=$\'false\'\nexport FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=$\'false\'\nexport FF_USE_DIRECT_DOWNLOAD
 exit 0
   job=207167 project=290 runner=JXRw3mH1
 Remote process exited with the status: CommandExitCode: 1, Script: /scripts-290-207167/get_sources  job=207167 project=290 runner=JXRw3mH1
 Container "helper" exited with error: command terminated with exit code 1  job=207167 project=290 runner=JXRw3mH1

notes:

  • the error “error: could not lock config file /root/.gitconfig: Read-only file system” is due to the current user inside container is different by root
  • the file /logs-290-207167/output.log contains the log of the job pod

Inside job pod shell we also tested some git commands and perform successfully fetch and clone using our personal credentials (the same user that perform the run of the pipeline from gitlab gui).

We think the problem can be related on gitlab-ci-token, but we have finished our investigation… :frowning:

Someone had the same problem?