Getting Auth Failed (code: zone_not_found) with S3 cache

We are using hosted gitlab.com and we host our own runners with Amazon S3 for caching.

We recently created a new project with a single CI job configuration. The cache configuration for the job looks like this:

 10  cache:
  9     key: $CACHE_KEY
  8     paths:
  7       - $PIP_CACHE_DIR
  6     policy: pull

Existing projects currently work just fine but this project’s job is returning fatal errors for pulling and pushing to s3 for cache.

FATAL: Get https://s3-REGION.amazonaws.com/BUCKET/project/PROJECT_ID/CACHE_KEY?X-Amz-Algorithm=....&X-Amz-Signature=[FILTERED] Auth Failed (code: zone_not_found)

FATAL: Put https://s3…?X-Amz-Algorithm=…X-Amz-Signature=[FILTERED]
Auth Failed (code: zone_not_found)
Failed to create cache

I’m not seeing any cache issues with existing projects. I’ve played around with different cache_key to make sure that’s not the problem. I can’t seem to find anything around Auth failed (code: zone_not_found). Is it referring to AWS availability zones/missing iam permissions or some internal gitlab issue? Any pointers on where to debug next?

Edit: I’ve noticed there are issues re: uploading artifacts from CI which we are also experiencing. I’m not sure if this might be related?

Case closed. A mistake on our end! We had a CI variable called http_proxy :man_facepalming: