Registry sending back 403 for one of my servers only

Hey all, first post here.

I have a 3 node docker swarm and one of my servers cannot seem to pull images. I have copied and pasted the ~/.docker/config.json from the working servers so it is using the same exact token.

Here is the error I am getting on the docker server:

Error response from daemon:

Get https://example.com:4567/v2/project/repo/manifests/tag: error parsing HTTP 403 response body: invalid character 'F' looking for beginning of value: "Forbidden\n"

It’s version:

docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

On the gitlab server this is what I am getting:

I have tried:

  • Updating gitlab
  • logging out the docker, removing the token and logging in again but that gave a slightly different 403 error.
  • restarting the docker services on the server with issues

Thanks

The reason could be that your server IP got on the Black List, there is more detailed description.