Windows docker executor deletes cache.zip before cache is restored

maxcode · August 22, 2019, 12:55pm

Hey,

I wanted to make a bug report but before I do that I wanted to ask here, to ensure I have not miss understood how it should work.

It seems that the gitlab-runner helper is removing cache.zip files when it shouldn’t.

My set up is as follows:
I have a gitlab runner (12.1.0) on a windows machine with “Docker desktop community 2.1.0.1”. The runner has the following config.toml:

    [[runners]]
      name = "builder-docker"
      url = "https://git.***********.net/"
      token = "**********************"
      executor = "docker"
      [runners.custom_build_dir]
      [runners.docker]
        tls_verify = false
        image = "alpine:3.9"
        privileged = false
        disable_entrypoint_overwrite = false
        oom_kill_disable = false
        disable_cache = false
        cache_dir = "/cache"
        volumes = ["/cache"]
        shm_size = 0
      [runners.cache]
        [runners.cache.s3]
        [runners.cache.gcs]

My project has the following .gitlab-ci.yml:

    cache:
      key: "sbt-cache"
      paths:
        - "sbt-cache/"    
    variables:
        SBT_OPTS: "-Dsbt.global.base=sbt-cache/.sbtboot -Dsbt.boot.directory=sbt-cache/.boot -Dsbt.ivy.home=sbt-cache/.ivy2 -Dsbt.log.noformat=true"
    build-publish:
      image: openjdk:8
      script:    
        - ls -al
        - ./activator $SBT_OPTS compile
      tags:
        - docker

All seems fine but when I run my job I see the following lines:

    Running with gitlab-runner 12.1.0 (de7731dd)
      on builder-docker **********
    Using Docker executor with image openjdk:8 ...
    Pulling docker image openjdk:8 ...
    Using docker image sha256:08ded5f856cc9c3d9ef07d81d06646c35849b9f0359d0a52a8fab6d43cc2cc5d for openjdk:8 ...
    Running on runner-********-project-60-concurrent-0 via IQIQ-7N9DN3S9UR...
    Fetching changes...
    Reinitialized existing Git repository in /builds/web/api-services/.git/
    Checking out f38f9381 as testing...
    Removing "..\\..\\..\\cache\\web\\api-services\\sbt-cache\\cache.zip"
    Removing .settings/
    Removing modules/common/target/
    Skipping Git submodules setupChecking cache for sbt-cache...
    No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.

The line Removing "..\\..\\..\\cache\\web\\api-services\\sbt-cache\\cache.zip" seems wrong. Also my test line - ls -al shows the project has no sbt-cache folder.

Anyone have any ideas about this. Am I missing something?

Best,
Jonathan

bageren · November 8, 2020, 8:43pm

Hi, I am facing the exact same issue. Did you ever get it solved?

maikschulze · November 9, 2020, 8:48am

Hi,

I’ve run into this problem as well and implemented a fix. I haven’t issued a merge request. If you find my code change helpful, feel free to do that.

I have modified the gitlab-runner which inserts the commands into the helper in the following way:
https://gitlab.com/maikschulze/gitlab-runner/-/compare/40ec2241...master
The important bit is fixing the backslash-containing path with

file = filepath.ToSlash(file)

Over the last two months I have used the modified gitlab-runner quite a lot and successfully. I could not find any side effects when using the patched gitlab-runner.exe with Windows clients as well. The forward slashes are handled gracefully there as well.

The job output would look the following:

Restoring cache
00:01
BUGFIX file before …....\cache\my-group\my-project\build-linux-on-windows-master-1\cache.zip…
BUGFIX file after …/…/…/cache/my-group/my-project/build-linux-on-windows-master-1/cache.zip…
Checking cache for build-linux-on-windows-master-1…
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
…
Saving cache
00:01
BUGFIX file before …....\cache\my-group\my-project\build-linux-on-windows-master-1\cache.zip…
BUGFIX file after …/…/…/cache/my-group/my-project/build-linux-on-windows-master-1/cache.zip…
Creating cache build-linux-on-windows-master-1…
_build/: found 81 matching files and directories
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
Created cache

For convenience, I have just rerun the binaries windows/386 windows/amd64 job and constructed the gitlab-runner artifact for download here:
https://gitlab.com/maikschulze/gitlab-runner/-/jobs/837633767

maikschulze · November 9, 2020, 8:56am

One side note on usage:

When using this unofficial gitlab-runner you will have to explicitly specify a helper image in your config.toml file, e.g.

[runners.docker]
helper_image = “gitlab/gitlab-runner-helper:x86_64-738bbe5a”
…
helper_image = “gitlab/gitlab-runner-helper:x86_64-738bbe5a-servercore1909”

Also see:

justheath · November 23, 2020, 3:39pm

Thanks for finding and fixing this. I’m new to using GitLab pipelines and couldn’t figure out why my cache was never restored.

Sadly I dismissed your fix when I first found your post. After re-reading an hour later and having a better understanding of what was happening in my pipeline, I returned and sure enough, this was the problem. I originally failed to make the connection between my cache failure and path separators but another issue I read demonstrated how the cache path was being treated as a filename, so the zip was never actually saved to the cache. A little slow, but I finally made the connection.

I got a patched runner installed and declared a helper_image and for the first time cache was restored properly and my pipeline succeeded!

chrycheng · May 19, 2022, 5:04pm

Have you gotten around to issuing that merge request? If not, I’d be happy to do it. Only, I can’t seem to find the fix you mentioned (file = filepath.ToSlash(file)) in the diff you linked to. Are you sure you’re comparing the right diffs in the right order?

maikschulze · May 26, 2022, 11:18am

Hi Christian,

I’ve finally gotten around issuing a merge request: fix caching in Linux containers on Windows hosts by avoiding insertion of '\'... (!3464) · Merge requests · GitLab.org / gitlab-runner · GitLab
I’ve identified this problem to be related to issue Caching using Docker executor under Docker for Windows doesn't work. (#4091) · Issues · GitLab.org / gitlab-runner · GitLab

Best regards
Maik

chrycheng · May 26, 2022, 2:40pm

Outstanding! I’ll follow the MR for updates. Thank you very much

tweijie · May 8, 2023, 11:12am

Update as of 2023 on how to apply Maik’s patch to your gitlab runner.

uninstall the existing runner (apparently the new binary will have the same name under Services so you can only choose one).
download the binary (as created by the GitLab-org’s CI, I arrived at this job via the pipeline #548732500 as shown in the merge request #3464.
install and register the modified gitlab-runner as usual (gitlab’s docs)
set the variable helper_image in config.toml as mentioned by Maik
```
[runners.docker]
helper_image = "gitlab/gitlab-runner-helper:x86_64-436955cb"
```
- This is necessary as the custom gitlab-runner provided by Maik is not in the main release of Gitlab yet. So there won’t be a gitlab-runner-helper of the same commit (which is not available on docker hub).
- For the image tag, check that it is available on docker hub
- I use 436955cb by looking at the available docker images in my pc.
restart the gitlab-runner

Hope it helps!

vinivasq · June 25, 2023, 5:58pm

Thanks a lot!
That was the only way i could get the runner to cache propperly on windows.