Gitlab CE docker push to registry hosted in s3 fails with "denied: access forbidden"

GitLab CI/CD
1

In gitlab CE docker push to registry hosted in s3 fails with “denied: access forbidden”

Here is a sample of my gitlab-ci.yml file:

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN gitlab.domain.net (successful)

Build:
  stage: build
  script:
    - docker build --pull -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME .  (successful)
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME    (FAILS)

Result:
Fails during the push.
The push refers to repository [gitlab.domain.net:6565/sfloyd/myapp]
7f44c19d438d: Preparing
26472437084c: Preparing
2de391e51d73: Preparing
d73dd9e65295: Preparing
686245e78935: Preparing
d7ff1dc646ba: Preparing
644879075e24: Preparing
d7ff1dc646ba: Waiting
644879075e24: Waiting
denied: access forbidden
ERROR: Job failed: exit code 1

I have tried not using S3 and having the registry store the images on the gitlab server itself and that fails with the same result. I even set the S3 bucket open to the world for read and write as a test and that also failed.
The CI_JOB_TOKEN was given “read_registry” access just like the Gitlab documentation states…although I feel like it should have write as well but its not an option.
I have swapped out the username to log in with, with the username of the token and it fails with the same result.
I created an access token for my admin user giving it API access and that fails.
Interestingly if i use my admin user and the access token with API access it works LOCALLY but not when I put those same values in the gitlab-ci.yml file.

$ docker images
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
gitlab.domain.net:6565/sfloyd/myapp   master              b03b47a5987d        14 seconds ago      211MB
ubuntu                                16.04               5e8b97a2a082        2 days ago          114MB

$ docker push gitlab.domain.net:6565/sfloyd/myapp:master
The push refers to repository [gitlab.domain.net:6565/sfloyd/myapp]
6852ea579a31: Pushed
5f7a44809386: Pushed
2de391e51d73: Layer already exists
d73dd9e65295: Layer already exists
686245e78935: Layer already exists
d7ff1dc646ba: Layer already exists
644879075e24: Layer already exists
master: digest: sha256:0f9f6c3d6ebc7f23f52a5b7be06457359de1c24959336de5b5e9e0a208a5b8c8 size: 1781

Ive pretty much exhausted everything I can think of. Anyone have a suggestion?

GitLab 10.8.1
Gitlab Runner 10.8.0