Gitlab Container Registry Image Pull results in 403 with We're sorry, but this service is not available in your location

I have a K8s cluster on Linode that pulls images from registry.gitlab.com
It’s configured properly with image pull secret and has been working perfectly fine for months up until now.
All of a sudden I get image pull errors, specifically seeing this response on image pull:

Failed to pull image “registry.gitlab.com/townlogic/api:275fc6144c2e5e37a27af7e7d8fbc5734ac9a966”: rpc error: code = Unknown desc = error pulling image configuration: error parsing HTTP 403 response body: invalid character ‘<’ looking for beginning of value: “<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>We're sorry, but this service is not available in your location</Details></Error>

This part of the error message I thought was quite strange: “We’re sorry, but this service is not available in your location”

Anyone have any idea on what happened? This is happening to more than one repository on the same Gitlab project, from the same k8s cluster.
Both were working completely okay for months until this happened. I haven’t changed any configurations in k8s.

Thanks

We’ve got this issue too, the cluster is based in AU. Pulling the image to my local environment works fine. This has been working for over a year and suddenly broke without touching any configuration.

@dkoleary88 which hosting are you with?

Linode.
Cluster region is Sydney, AU…
Gitlab stopped supporting this region??

@erfanimani_pacvac
How can we get Gitlab support to take a look at this? Seems like an issue on their side.

@dkoleary88 We’re running Linode in Sydney too.

We’ve created a support ticket on Linode’s end, I’ll place an update here if there’s any progress.

1 Like

Thanks @erfanimani_pacvac
Please do let me know what you find.

I would do a whois on the IP address of the servers to find out a bit more of which region/country it’s assigned to. Sometimes can happen, if the IP range is assigned to a country that is on the Gitlab unsupported region list. This could be for many reasons. Maybe the IP range is assigned to that unsupported region, or the range has been taken over/released and still has old data for it’s region assignment. Or neither, and it’s just being flagged incorrectly.

Thanks @iwalker. Is there a way for someone at Gitlab to look at this? Whois and GeoIP looks OK.

We’ve been unable to run builds for a few days now, and have been forced to use a different service.

@dkoleary88 Linode reckons no issues on their end.

Thanks for getting back to me @erfanimani_pacvac
What other service are you using for container image hosting?

Hi @dkoleary88, we’ve switched to Docker Hub as it has got the best cost/value. The rest (Github, Digital Ocean, etc.) all have pretty low caps on their storage from what I can gather.

Thanks for the update @erfanimani_pacvac
Pity that Gitlab broke.

@erfanimani_pacvac
Looks like Gitlab have fixed the issue now and images are pulling as expected now… :grimacing:

1 Like