I have a K8s cluster on Linode that pulls images from registry.gitlab.com
It’s configured properly with image pull secret and has been working perfectly fine for months up until now.
All of a sudden I get image pull errors, specifically seeing this response on image pull:
Failed to pull image “registry.gitlab.com/townlogic/api:275fc6144c2e5e37a27af7e7d8fbc5734ac9a966”: rpc error: code = Unknown desc = error pulling image configuration: error parsing HTTP 403 response body: invalid character ‘<’ looking for beginning of value: “<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>We're sorry, but this service is not available in your location</Details></Error>”
This part of the error message I thought was quite strange: “We’re sorry, but this service is not available in your location”
Anyone have any idea on what happened? This is happening to more than one repository on the same Gitlab project, from the same k8s cluster.
Both were working completely okay for months until this happened. I haven’t changed any configurations in k8s.
Thanks
We’ve got this issue too, the cluster is based in AU. Pulling the image to my local environment works fine. This has been working for over a year and suddenly broke without touching any configuration.
@dkoleary88 which hosting are you with?
Linode.
Cluster region is Sydney, AU…
Gitlab stopped supporting this region??
@erfanimani_pacvac
How can we get Gitlab support to take a look at this? Seems like an issue on their side.
@dkoleary88 We’re running Linode in Sydney too.
We’ve created a support ticket on Linode’s end, I’ll place an update here if there’s any progress.
1 Like
Thanks @erfanimani_pacvac
Please do let me know what you find.
I would do a whois on the IP address of the servers to find out a bit more of which region/country it’s assigned to. Sometimes can happen, if the IP range is assigned to a country that is on the Gitlab unsupported region list. This could be for many reasons. Maybe the IP range is assigned to that unsupported region, or the range has been taken over/released and still has old data for it’s region assignment. Or neither, and it’s just being flagged incorrectly.
Thanks @iwalker. Is there a way for someone at Gitlab to look at this? Whois and GeoIP looks OK.
We’ve been unable to run builds for a few days now, and have been forced to use a different service.
@dkoleary88 Linode reckons no issues on their end.
Thanks for getting back to me @erfanimani_pacvac
What other service are you using for container image hosting?
Hi @dkoleary88, we’ve switched to Docker Hub as it has got the best cost/value. The rest (Github, Digital Ocean, etc.) all have pretty low caps on their storage from what I can gather.
Thanks for the update @erfanimani_pacvac
Pity that Gitlab broke.
@erfanimani_pacvac
Looks like Gitlab have fixed the issue now and images are pulling as expected now… 
1 Like
Hello there,
got exactly the same issue this afternoon, in the morning it was ok.
May I know how did you fix it, please?
Regards
Have same issue, still face same problem.
Had create a ticket to GitLab support, have auto email answer, should wait 24 hours, but have Premium account -_-
Linode DE + GitLab Registry
1 Like
I’m on a Frankfurt Linode as well, can you please keep me updated with the response from the support?
Thank you and regards
2 Likes
Sure, still waiting response from second request. But get response from first one:
The cause appears to be that parts of Google Cloud Services have deny-listed Linode’s Frankfurt IPs.
Since the registry.gitlab.com access redirects clients to Google Storage API URLs for image layer blob downloads/uploads, the direct access made from Linode hosts to their services end up getting blocked.
Our infrastructure teams are looking into this and are in touch with Google Cloud support.
Unfortunately there’s no workaround at this time other than using another region in Linode (such as Amsterdam, which was reported working) to go around Google Cloud’s current blocking.
At this time there is no ETA from Google Cloud support on the resolution but I will keep you posted with any material updates or information requests I receive from our infrastructure team.
1 Like
Same problem here…
Our production server is on Linode (Frankfurt) and has not been able to pull an image from registry.gitlab.com for 4 days now. 
1 Like
I was able to temporarily work around the issue by creating a cloud firewall and block outgoing requests to Google’s IPv6-Subnet. It will fall back to IPv4 to pull the images.
You saved my day!!
Blocking outgoing traffic on the suggested subnet worked flawlessly, btw I’m on Ubuntu and find more pratical to do:
sudo ufw deny out to 2a00:1450::/32
1 Like
