Hi,
I’ve been experimenting with CI/CD in GitLab on a docker instance and only having 500 errors and falsely reported failed jobs.
I have a very simple setup, 1 runner and 1 pipeline in GitLab. When I push to my GitLab repo, it will trigger the job and communicate with the runner, but it will get “stuck”. It will show some progress in the job window, but once there’s high-output it will stop working. The Runner will finish successfully based on traces in the runner docker instance inside /tmp/trace#####, however it cannot communicate back to GitLab. The error the runner has is:
WARNING: Checking for jobs... failed
runner=RUNNERID status=couldn't execute POST against https://mydomain.com/api/v4/jobs/request: Post https://mydomain.com/api/v4/jobs/request: EOF
After some time passes, a new log entry:
WARNING: Appending trace to coordinator... failed code=500 job=77 job-log= job-status= runner=RUNNERID sent-log=1894-506296 status=500 Internal Server Error update-interval=0s
The trace will remain on the runner instance inside /tmp, and the GitLab job will timeout. All following jobs will remain in “Pending” because “This job is stuck because the project doesn’t have any runners online assigned to it.”… however the runner is online, I must restart the runner docker instance.
Versions are:
GitLab CE: 14.5.0
GitLab Runner: 14.5
This is incredibly frustrating. My options are:
- Reduce log output or pipe all output to a file and then upload to an s3/gcps bucket
- All “failed” jobs are actually successful, and I must restart runner docker instance.
I cannot find any documentation or reported bug for this issue. I assume this is not expected behavior?