Artifacts fail to upload - storage.googleapis.com 500

Hi,

Under my .gitlab-ci.yaml I have:

cache:
  untracked: true
  policy: pull-push

Couple weeks ago I was able to cache untracked artifacts and pull/push them automatically; this is not happening anymore. Additional info: the pipeline didn’t change. I am using shared runners. Currently, I have a tiny untracked text file only. Silver plan.

Last week:

Restoring cache
00:02
 Checking cache for default...
 Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/xxx/default 
 Successfully extracted cache
Downloading artifacts
(...)
Saving cache
00:02
 Creating cache default...
 untracked: found 1 files                           
 Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/xxx/default 
 Created cache
Uploading artifacts for successful job
00:01
 Job succeeded

Now I see this:

Restoring cache
 Checking cache for default...
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 FATAL: received: 500 Internal Server Error         
 Failed to extract cache
Downloading artifacts
(...)
Saving cache
00:05
 Creating cache default...
 untracked: found 1 files                           
 Uploading cache.zip to https://storage.googleapis.com/gitlab-org-ci-runners-cache/project/xxx/default 
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 Uploading cache.zip to https://storage.googleapis.com/gitlab-org-ci-runners-cache/project/xxx/default 
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 Uploading cache.zip to https://storage.googleapis.com/gitlab-org-ci-runners-cache/project/xxx/default 
 FATAL: received: 500 Internal Server Error         
 Failed to create cache
Uploading artifacts for successful job
 Job succeeded

Any tip how to debug/solve this? Thank you.

Cheers,
F

I’m having a very similar problem, though it’s first-time use (as of yesterday). My configuration specifies a global cache for package metadata:

...
cache:
  key: global
  paths:
    - "var_cache_dnf"
...

When a job runs on both a MR or on a branch, I get:

 Checking cache for global-1...
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 FATAL: received: 500 Internal Server Error         
 Failed to extract cache
...

My job continues running, then caching fails again at the end:

...
Saving cache
00:04
 Creating cache global-1...
 var_cache_dnf: found 2 matching files              
 Uploading cache.zip to https://storage.googleapis.com/gitlab-org-ci-runners-cache/project/19009784/global-1 
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 Uploading cache.zip to https://storage.googleapis.com/gitlab-org-ci-runners-cache/project/19009784/global-1 
 WARNING: Retrying...                                error=received: 500 Internal Server Error
 Uploading cache.zip to https://storage.googleapis.com/gitlab-org-ci-runners-cache/project/19009784/global-1 
 FATAL: received: 500 Internal Server Error         
 Failed to create cache
...

I tried setting the ‘cache’ block in each job, and using different values for ‘key’ (including no key). Always the same result

Hi @r4z0r7o3 did you manage to get this working? My pipeline still fails. Looks like something is failing on storage google cloud but even after enabling the debug variable I don’t get more info.

https://docs.gitlab.com/ee/ci/variables/#debug-logging

I could not get it to work, and I agree the error makes it appear like a GCS problem. However, I found that saving artifacts (not cache) DOES work. So I’m planning to use that like cache, just curl’ing down the latest artifact containing my cache, and starting with that. Though having this handled automatically would be nicer

Thanks @r4z0r7o3 I will follow this artifacts workaround too.