Https GitHttpController Completed 500 Internal Server Error

Hi there,

I have spend 2 days already to fix that issue and no luck for now.

My gitlab server is on Gentoo and compiled from source.

So I had working gitlab-8.9.5 with https configuration running for some time and with my dev team decided to upgrade to 8.13.5.

Now using web browser basic functions are as follow:
create projects : ok
create new branch in existsing project : failed
delete project : ok

cloning with ssh : ok
cloning with http : failed

logs:
nginx
“POST /ci/api/v1/builds/register.json HTTP/1.1” 404 27 “-” “Go-http-client/1.1”
“GET /gbos-public/gblabs-overlay.git/info/refs?service=git-upload-pack HTTP/1.1” 500 2408 “-” “git/2.7.3”

production.log
Processing by Projects::GitHttpController#info_refs as /
Parameters: {“service”=>“git-upload-pack”, “namespace_id”=>“gbos-public”, “project_id”=>“gblabs-overlay.git”}
Completed 500 Internal Server Error in 102ms (ActiveRecord: 6.8ms)

JWT::DecodeError (Nil JSON web token):
lib/gitlab/workhorse.rb:120:in verify_api_request!' app/controllers/projects/git_http_client_controller.rb:154:in verify_workhorse_api!’
lib/gitlab/request_profiler/middleware.rb:15:in call' lib/gitlab/middleware/go.rb:16:in call’

Also i can see a lot of
“POST /ci/api/v1/builds/register.json HTTP/1.1” 404 27 “-” “Go-http-client/1.1”

in my gitlab.rb
external_url ‘https://FQDN
gitlab_rails[‘gitlab_email_enabled’] = true
gitlab_rails[‘gitlab_email_from’] = ‘git@FQDN’
gitlab_rails[‘gitlab_email_display_name’] = ‘Gitlab’
gitlab_rails[‘gitlab_default_can_create_group’] = true
gitlab_rails[‘gitlab_username_changing_enabled’] = true
gitlab_rails[‘gitlab_default_theme’] = 2
gitlab_rails[‘gitlab_default_projects_features_issues’] = true
gitlab_rails[‘gitlab_default_projects_features_merge_requests’] = true
gitlab_rails[‘gitlab_default_projects_features_wiki’] = true
gitlab_rails[‘gitlab_default_projects_features_snippets’] = false
gitlab_rails[‘gitlab_default_projects_features_builds’] = true
gitlab_rails[‘gitlab_default_projects_features_container_registry’] = true
gitlab_rails[‘gitlab_repository_downloads_path’] = ‘tmp/repositories’
gitlab_rails[‘artifacts_enabled’] = true
gitlab_rails[‘artifacts_path’] = “/home/git/gitlab/shared/artifacts/”
gitlab_rails[‘lfs_enabled’] = true
gitlab_rails[‘lfs_storage_path’] = “/home/git/gitlab/shared/lfs-objects”
gitlab_rails[‘elasticsearch_enabled’] = false
gitlab_rails[‘elasticsearch_host’] = “localhost”
gitlab_rails[‘elasticsearch_port’] = 9200
gitlab_rails[‘pages_path’] = “/home/git/gitlab/shared/pages”
gitlab_workhorse[‘enable’] = true
gitlab_workhorse[‘auth_backend’] = “http://FQDN:8080
gitlab_workhorse[‘auth_socket’] = “’’”
nginx[‘enable’] = false
web_server[‘external_users’] = [‘nginx’]
unicorn[‘enable’] = true

i’m runnig nginx server
here is my nginx config for gitlab
https://paste.pound-python.org/show/5mCLCUfj19nvMrnm5cW1

git user is in nginx group

all self diagnostic test are returning OK

env:info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.1.9p490
Gem Version: 2.5.2
Bundler Version:1.12.5
Rake Version: 10.5.0
Sidekiq Version:4.2.1

GitLab information
Version: 8.13.5
Revision: 09cedb5
Directory: /opt/gitlabhq-8.13
DB Adapter: postgresql
URL: https://FQDN
HTTP Clone URL: https://FQDN/some-group/some-project.git
SSH Clone URL: git@FQDN:some-group/some-project.git
Using LDAP: no
Using Omniauth: no

GitLab Shell
Version: 3.6.6
Repository storage paths:

  • default: /home/git/repositories/
    Hooks: /var/lib/gitlab-shell/hooks/
    Git: /usr/bin/git

Also in gitlab.yml
production: &base

1. GitLab app settings

==========================

GitLab settings

gitlab:
host: FQDN
port: 443
https: true

So please help me fix that problem.

How I can enable more debug ?

ok so I have manage to fix both mentioned problems.
To fix branch creation problem on existing project I needed
chmod +x hooks/update

it’s a hack for now to fix major problem when http clone did not work.
in lib/gitlab/workhorse.rb
def verify_api_request!(request_headers)
#JWT.decode(
# request_headers[INTERNAL_API_REQUEST_HEADER],
# secret,
# true,
# { iss: ‘gitlab-workhorse’, verify_iss: true, algorithm: ‘HS256’ },
#)
true
end

So for now I have disabled that validation and now everything works.
The question is how I should configure JWT tokens validation ?