GitLab Pipeline randomly failing using Docker Runners - suspect caching issue

GitLab CI/CD
, ,
8

@ricardoamarilla After more searching, I think my issue is related to the issue described in this GitLab Forum post.

I tried the solution described there of adding cache parameters to my docker executor GitLab Runner config.toml files and I restarted the docker containers. (see config.toml below)

The pipeline now completes IF I have only ONE shared runner enabled. In other words, when I have two shared runners available (one running on one EC2 instance and another running on another EC2 instance), when the runner’s start executing in parallel (during the static-analysis and test jobs), one of the runners always fails. During 4 pipeline runs, shared-runner-1 always failed, regardless of which job it ran (static-analysis or test). I’m guessing that’s just a coincidence because running with shared-runner-1 by itself is able to successfully complete the pipeline.

— UPDATE —
After I paused and then restarted each of the GitLab Runners in turn (and ran my pipeline which completed successfully with only one shared-runner available), I then enabled both shared runners again and ran a couple more tests. AND THIS TIME THE Pipeline completed successfully with both shared-runer-1 and 2 participating in the pipeline! Why was I not able to use both gitlab runners in the pipeline until I paused and restarted the shared runners? Should I have cleared the “Runner Caches”?
— end UPDATE —

Is there a gitlab-runner command I can execute on the running gitlab-runner to see what configuration it thinks it has? I tried a couple variations of the following command, but failed to be able to display the gitlab-runner config:

$ docker exec -it gitlab-runner gitlab-runner verify
Runtime platform                                    arch=amd64 os=linux pid=61 revision=58ba2b95 version=14.2.0
Running in system-mode.

Verifying runner... is alive                        runner=EwF7xhTi

Here’s the config.toml on the failing shared runner - docker executor (the working shared runner config.toml is the same):

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "shared-runner-1"
  url = <hidden URL>
  token = "<hidden token>"
  executor = "docker"
  cache_dir = "/cache"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "centos:7.9"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    cache_dir = ""
    volumes = ["/cache"]
    shm_size = 0