Git clone/push fails, http://127.0.0.1:8080 redirects to https://127.0.0.1:8080

How to Use GitLab Self-managed
,
1

After upgrading gitlab-ce to 11.3.4, some LDAP (Active Directory) users get an “Invalid credentials” message when logging in, and the error message

GitLab: API is not accessible
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

when trying to clone or push an existing project. Git logs in this case indicate “Access denied”. It is certain that the LDAP/AD account credentials are valid.

The complicating fact is that when I say “upgrade”, we went from 8.15.3 to 11.3.4 (with updates to 8.17.8, 9.5.10, and 10.8.7 in the middle), and I’m not certain where in that process the problem first started.

Any ideas on where to start tracking this issue down? Thanks in advance!

2

An additional data point… we are currently changing our email configuration from “user@subdomain.company.com” to “user@company.com”. My account was originally tied to my @subdomain.company.com email, and a test account was tied to my @company.com email address. Once I straightened that out, I can log in to the web UI again. However, another user with the same issue has only one account, which lists user@company.com as his email address. Is it possible the old email address is cached/saved somewhere, and that’s causing this issue?

I still get the “GitLab: API is not accessible” message when I try to clone one of my projects.

3

A little more information… I forgot to mention that this is a locally-hosted gitlab-ce server.

“ssh git@git.company.com” is successful, so my ssh key seems to be valid.

4

From gitlabb-rake gitlab:check:

    Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
    Check GitLab API access: gitlab-shell self-check failed
      Try fixing it:
      Make sure GitLab is running;
      Check the gitlab-shell configuration file:
      sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
      Please fix the error above and rerun the checks.

    Checking GitLab Shell ... Finished

But it doesn’t give any hints as to what the error is :frowning:

From gitlab-shell.log:

time="2018-11-15T15:52:17-06:00" level=info msg="finished HTTP request" duration=0.003335289 method=GET pid=4949 url="http://127.0.0.1:8080/api/v4/internal/check"
time="2018-11-15T15:52:17-06:00" level=error msg="Call failed" body= code=301 method=GET pid=4949 url="http://127.0.0.1:8080/api/v4/internal/check
5

So I think I may have found the root issue, but I’m not sure what to do about it.

When I try to test ssh, I get this:

# ssh -T git@git.aps.anl.gov
Welcome to GitLab, Anonymous!

And in gitlab-shell.log, I see the following:

time="2018-11-19T16:08:12-06:00" level=error msg="Call failed" body= code=301 method=GET pid=47229 url="http://127.0.0.1:8080/api/v4/internal/discover?key_id=59"

And, testing the URL in the message:

 # curl -vi http://127.0.0.1:8080/api/v4/internal/discover?key_id=59
* About to connect() to 127.0.0.1 port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /api/v4/internal/discover?key_id=59 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8080
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Mon, 19 Nov 2018 22:31:47 GMT
Date: Mon, 19 Nov 2018 22:31:47 GMT
< Connection: close
Connection: close
< Content-Type: text/html
Content-Type: text/html
< Location: https://127.0.0.1:8080/api/v4/internal/discover?key_id=59
Location: https://127.0.0.1:8080/api/v4/internal/discover?key_id=59
< 

* Closing connection 0

Note that I have HSTS enabled. It appears that authentication requests are getting redirected to HTTPS, but unicorn is not listening for https traffic, as far as I can tell.

Any ideas?