Gitlab-runner doesn't run as service

crispus · March 3, 2020, 12:18pm

Hello,

After migration from Debian 9 to Debian 10 the gitlab-runner stopped working (the service is up and running, but all the runners/jobs/pipelines are failing). It seems to fail at prepare_script/executor, doesn’t get to run any of the pipeline’s script:

Running with gitlab-runner 12.8.0 (1b659122)
on Crawler 7de7ff65
section_start:1583183047:prepare_executor
Using Shell executor…
section_end:1583183047:prepare_executor
section_start:1583183047:prepare_script
Running on home-server…
section_end:1583183047:prepare_script
section_start:1583183047:upload_artifacts_on_failure
section_end:1583183047:upload_artifacts_on_failure
ERROR: Job failed: exit status 1

I checked other threads as Dind gitlab-runner service doesn't start up suddenly etc
My issue is closest to Gitlab-runner works standalone, but not running as a systemd service but is not related to Registering runners - status=couldn't execute POST against.... (#2351) · Issues · GitLab.org / gitlab-runner · GitLab

My system is a physical PC in a local network with:
Debian 10
Gitlab-CE 12.8.0
gitlab-runner 12.8.0

Running from command line (with or without --debug) works just fine.
From service when failing (changed with --debug option) the logs are:

Using Shell executor… job=3195 project=11 runner=d4091cf4
Waiting for signals… job=3195 project=11 runner=d4091cf4
No referees configured job=3195 project=11 runner=d4091cf4
Executing build stage build_stage=prepare_script job=3195 project=11 runner=d4091cf4
(to gitlab-runner) root on none
pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
pam_unix(su:session): session closed for user gitlab-runner
Executing build stage build_stage=upload_artifacts_on_failure job=3195 project=11 runner=d4091cf4
su[29994]: (to gitlab-runner) root on none
su[29994]: pam_unix(su:session): session opened for user gitlab-runner by (uid=0)
su[29994]: pam_unix(su:session): session closed for user gitlab-runner
Skipping referees execution job=3195 project=11 runner=d4091cf4
WARNING: Job failed: exit status 1 duration=95.019831ms job=3195 project=11 runner=d4091cf4
WARNING: Job failed: exit status 1 duration=95.019831ms job=3195 project=11 runner=d4091cf4
Appending trace to coordinator… ok code=202 job=3195 job-log=0-484 job-status=running runner=d4091cf4 sent-log=0-483 status=202 Accepted update-interval=3s
Submitting job to coordinator… ok code=200 job=3195 job-status= runner=d4091cf4
WARNING: Failed to process runner builds=0 error=exit status 1 executor=shell runner=d4091cf4
WARNING: Failed to process runner builds=0 error=exit status 1 executor=shell runner=d4091cf4

While the success (from command line) looks like this:

Using Shell executor… job=3196 project=11 runner=d4091cf4
Waiting for signals… job=3196 project=11 runner=d4091cf4
No referees configured job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=prepare_script job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=get_sources job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=restore_cache job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=download_artifacts job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=build_script job=3196 project=11 runner=d4091cf4
Appending trace to coordinator… ok code=202 job=3196 job-log=0-1597 job-status=running runner=d4091cf4 sent-log=0-1596 status=202 Accepted update-interval=3s
Appending trace to coordinator… ok code=202 job=3196 job-log=0-155670 job-status=running runner=d4091cf4 sent-log=1597-155669 status=202 Accepted update-interval=3s
Appending trace to coordinator… ok code=202 job=3196 job-log=0-157136 job-status=running runner=d4091cf4 sent-log=155670-157135 status=202 Accepted update-interval=3s
Appending trace to coordinator… ok code=202 job=3196 job-log=0-168186 job-status=running runner=d4091cf4 sent-log=157136-168185 status=202 Accepted update-interval=3s
Appending trace to coordinator… ok code=202 job=3196 job-log=0-319972 job-status=running runner=d4091cf4 sent-log=168186-319971 status=202 Accepted update-interval=3s
Appending trace to coordinator… ok code=202 job=3196 job-log=0-323627 job-status=running runner=d4091cf4 sent-log=319972-323626 status=202 Accepted update-interval=3s
Executing build stage build_stage=after_script job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=archive_cache job=3196 project=11 runner=d4091cf4
Executing build stage build_stage=upload_artifacts_on_success job=3196 project=11 runner=d4091cf4
Skipping referees execution job=3196 project=11 runner=d4091cf4
Job succeeded duration=18.631915138s job=3196 project=11 runner=d4091cf4

The only difference I see is some related to running as user gitlab-runner…

The .gitlab-ci.yml is very simple (just calls a shell script):

build:
stage: build
script:
- ./build.sh

Any idea?

Thanks.

crispus · March 9, 2020, 2:04pm

Same issue with 12.9.0~beta.2440.ge2317539 (e2317539) …

crispus · March 23, 2020, 5:59pm

The same with 12.9.0.
Any idea how to debug this?

Thanks

gitlab-greg · March 23, 2020, 6:27pm

It sounds like you’re running gitlab-runner for stretch but your Debian version is now buster. With an operating system upgrade, repository sources are removed for the previous release and installed packages should be updated to run the build packaged for your new OS version.

https://docs.gitlab.com/runner/install/linux-repository.html

If upgrading to Debian 9 (Stretch) to Debian 10 (Buster), you’ll need to add repository sources for GitLab Runner buster, and then update the gitlab-runner to use the package for buster (not stretch) with sudo apt-get upgrade.

I hope you find this helpful. Let us know if this fixes your issue or if you have any questions!

crispus · March 23, 2020, 6:50pm

Thanks for the reply.
I think I already upgraded it:

# apt policy gitlab-runner      
gitlab-runner:
  Installed: 12.9.0
  Candidate: 12.9.0
  Version table:
 *** 12.9.0 500
        500 https://packages.gitlab.com/runner/gitlab-runner/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

I can’t understand why same command
/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"
in systemd doesn’t work, but it works on command line…

crispus · March 26, 2020, 9:14pm

I finnaly found the solution: Gitlab-runner fails all jobs when running with systemd
just comment the clear_console part from the .bash_logout


# when leaving the console clear the screen to increase privacy

#if [ "$SHLVL" = 1 ]; then
#    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
#fi