Registry login issues

environment:

  • self hosted gitlab
  • gitlab: 13.12.4ee
  • Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2

Im a little stuck here.
I have used the registry successfully one time and was able to push a container, now the login fails… sort of…

if i do “docker login gitlab2.recurve-it.com -u root -p xxxxx” i can login, but cant push or pull containers.
if i do “docker login gitlab2.recurve-it.com:5000 -u root -p xxxxx” the login fails.

troubleshooting:

  • My root access token is enabled for everything (api, read_user, read_api, read_repository, write_repository, read_registry, write_registry, sudo)
  • Created a repo access token same results as above
  • checked that port is still open:
    ben@gitlab2:/etc/gitlab$ sudo lsof -i -P -n | grep registry
    registry 2308089 registry 3u IPv4 9468664 0t0 TCP 127.0.0.1:5000 (LISTEN)
  • tried changing the registry port in the gitlab.rb to 5050 but it didnt seem to take. the listening port was still 5000, and got the same results… i could login using the base url but not push or pull
  • tried adding /v1/ to the login url both with and without ports… same result
  • multiple restarts and reconfigures

after logging in to the url without the port and trying to push a container i get:

  • 404 The page could not be found or you don’t have permission to view it.The resource that you are attempting to access does not exist or you don’t have the necessary permissions to view it.

when pulling the container that exists in the repo i get:

the following is my registry config in gitlab.rb:

everything else for the registry is default and commented.

the one time i was able to get a login and push to work, i used the port in the url, now it cant even log in with the port.

I am at a loss now what to try next. any help would be appreciated.
Thanks

not sure if it helps or not, but here is the timeline:
registry worked, and i was able to push a container using a gitlab runner.
came in the next day and it was failing the login.
I noticed that the container registry was turned off on the admin page.
restarted gitlab using sudo gitlab-ctl restart
container registry turned back on.
haven’t been able to push or pull ever since, and only able to login with the port not in the URL(even though on the container registry page within the project the cli commands say to use the port “docker login gitlab2.recurve-it.com:5000” )