Hello everyone,
I’m here to see if anyone else is having the same problem as me and if you have any solutions. When I upgraded to Gitlab Runner 17.11.0 and then 17.11.1, I noticed that the gitlab_runner_job_duration_seconds metric was no longer working correctly. In fact, the sum of job execution times is giving me a negative result. Moreover, it decreases over time (see screenshot).
Example Metrics
# HELP gitlab_runner_job_duration_seconds Histogram of job durations
# TYPE gitlab_runner_job_duration_seconds histogram
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="30"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="60"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="300"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="600"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="1800"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="3600"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="7200"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="10800"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="18000"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="36000"} 2
gitlab_runner_job_duration_seconds_bucket{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru",le="+Inf"} 2
gitlab_runner_job_duration_seconds_sum{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru"} -2e-09
gitlab_runner_job_duration_seconds_count{runner="pCpHUNbYF",system_id="r_y5h2G6CFH6ru"} 2
Log API
{"code":202,"job":10017260613,"job-log":"0-2221","job-status":"running","level":"info","msg":"Appending trace to coordinator...ok","runner":"pCpHUNbYF","sent-log":"806-2220","status":"202 Accepted","time":"2025-05-13T09:24:01Z","update-interval":60000000000}
{"duration_s":-1e-9,"job":10017260613,"level":"info","msg":"Job succeeded","project":69189285,"runner":"pCpHUNbYF","time":"2025-05-13T09:24:10Z"}
{"code":202,"job":10017260613,"job-log":"0-2466","job-status":"running","level":"info","msg":"Appending trace to coordinator...ok","runner":"pCpHUNbYF","sent-log":"2221-2465","status":"202 Accepted","time":"2025-05-13T09:24:10Z","update-interval":60000000000}
{"bytesize":2466,"checksum":"crc32:e41c3eaf","job":10017260613,"level":"info","msg":"Updating job...","runner":"pCpHUNbYF","time":"2025-05-13T09:24:10Z"}
{"bytesize":2466,"checksum":"crc32:e41c3eaf","code":202,"job":10017260613,"job-status":"running","level":"info","msg":"Submitting job to coordinator...accepted, but not yet completed","runner":"pCpHUNbYF","time":"2025-05-13T09:24:11Z","update-interval":1000000000}
{"bytesize":2466,"checksum":"crc32:e41c3eaf","job":10017260613,"level":"info","msg":"Updating job...","runner":"pCpHUNbYF","time":"2025-05-13T09:24:12Z"}
{"bytesize":2466,"checksum":"crc32:e41c3eaf","code":200,"job":10017260613,"job-status":"success","level":"info","msg":"Submitting job to coordinator...ok","runner":"pCpHUNbYF","time":"2025-05-13T09:24:12Z","update-interval":0}
{"builds":0,"job":10017260613,"level":"info","max_builds":50,"msg":"Removed job from processing list","project":69189285,"repo_url":"https://gitlab.com/myrepo.git","time":"2025-05-13T09:24:12Z","time_in_queue_seconds":2}
{"job":10017260620,"level":"info","msg":"Checking for jobs... received","repo_url":"https://gitlab.com/myrepo.git","runner":"pCpHUNbYF","time":"2025-05-13T09:24:15Z"}
{"builds":1,"job":10017260620,"level":"info","max_builds":50,"msg":"Added job to processing list","project":69189285,"repo_url":"https://gitlab.com/myrepo.git","time":"2025-05-13T09:24:15Z","time_in_queue_seconds":1}
Thank you in advance for your help.