Gitlab UI shows empty project even after a git push and Slack notifications doesn't trigger when a git push is made

Hi!

I’m facing to two errors:

  • When I create a new repository and then I push to it, the Gitlab UI shows empty project. I’ve to rename the repository to temporally fix the bug. I tried the button Housekeeping but after that the repository become inaccessible (404 error).
    Similar issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/24046
  • When I do a git push, the Slack notifications service doesn’t trigger but it works if I use the button test settings manually.

Does anyone have an idea how to resolve this two issues?

How did I install Gitlab?

I was originally on a Debian VPS using the omnibus package then I migrated to Archlinux using the official package and I followed the guide who was a bit incorrect but in the end I got a working installation. Then I created a backup on my Debian VPS and after that I restored it on my ArchLinux VPS.
Note: I didn’t migrate the secret files because I didn’t know how to move secrets file for Omnibus to a source installation.

Technical information about my Gitlab instance

Gitlab is running behind a nginx reverse proxy using workhorse (template example) with sidekiq and unicorn.

Logs

production.log

[1]

I created a new repository then I pushed an example and I pressed the button Housekeeping.

Completed 200 OK in 344ms (Views: 194.3ms | ActiveRecord: 111.2ms)
Started GET "/projects/new" for 213.49.60.167 at 2017-01-18 14:18:23 +0100
Processing by ProjectsController#new as HTML
Completed 200 OK in 127ms (Views: 83.2ms | ActiveRecord: 5.8ms)
Started GET "/Dofus/Tofumanchou.git/info/refs?service=git-upload-pack" for 128.78.97.54 at 2017-01-18 14:18:25 +0100
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"Dofus", "project_id"=>"Tofumanchou.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 30ms (Views: 0.5ms | ActiveRecord: 1.8ms)
Started GET "/Dofus/Tofumanchou.git/info/refs?service=git-upload-pack" for 128.78.97.54 at 2017-01-18 14:18:25 +0100
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"Dofus", "project_id"=>"Tofumanchou.git"}
Completed 500 Internal Server Error in 112ms (ActiveRecord: 2.5ms)

JWT::VerificationError (Signature verification raised):
  lib/gitlab/workhorse.rb:137:in `decode_jwt'
  lib/gitlab/workhorse.rb:133:in `verify_api_request!'
  app/controllers/concerns/workhorse_request.rb:11:in `verify_workhorse_api!'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'


Started POST "/projects" for 213.49.60.167 at 2017-01-18 14:18:28 +0100
Processing by ProjectsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"VDnT/EPD1VO3uqETva/dIWVEceLPPRHopiXMNR13sefdE0UyvOlhRy8A8NHaoyMXP3bdpmmbbDt4YNYF3vcRKA==", "project"=>{"namespace_id"=>"1", "path"=>"test", "description"=>"", "visibility_level"=>"0"}}
Redirected to https://gitlab.aresrpg.fr/unixfox/test
Completed 302 Found in 1506ms (ActiveRecord: 16.5ms)
Started GET "/unixfox/test" for 213.49.60.167 at 2017-01-18 14:18:29 +0100
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"test"}
Completed 200 OK in 278ms (Views: 209.6ms | ActiveRecord: 13.4ms)
Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2017-01-18 14:18:48 +0100
Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2017-01-18 14:19:11 +0100
Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2017-01-18 14:19:11 +0100
Started GET "/api/v3/internal/broadcast_message" for 127.0.0.1 at 2017-01-18 14:19:12 +0100
Started GET "/api/v3/internal/merge_request_urls?project=/var/lib/gitlab/repositories/unixfox/test.git&changes=0000000000000000000000000000000000000000%20165dfffbf472b255b4d61473f62412b7d690f8df%20refs/heads/master%0A" for 127.0.0.1 at 2017-01-18 14:19:12 +0100
Started GET "/unixfox/test" for 213.49.60.167 at 2017-01-18 14:19:14 +0100
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"test"}
Completed 200 OK in 137ms (Views: 84.2ms | ActiveRecord: 8.2ms)
Started GET "/unixfox/test" for 213.49.60.167 at 2017-01-18 14:19:18 +0100
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"test"}
Completed 200 OK in 105ms (Views: 56.4ms | ActiveRecord: 7.6ms)
Started GET "/unixfox/test/edit" for 213.49.60.167 at 2017-01-18 14:19:21 +0100
Processing by ProjectsController#edit as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"test"}
Completed 200 OK in 91ms (Views: 48.5ms | ActiveRecord: 7.0ms)
Started GET "/Dofus/Tofumanchou.git/info/refs?service=git-upload-pack" for 128.78.97.54 at 2017-01-18 14:19:25 +0100
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"Dofus", "project_id"=>"Tofumanchou.git"}
Started POST "/unixfox/test/housekeeping" for 213.49.60.167 at 2017-01-18 14:19:25 +0100
Processing by ProjectsController#housekeeping as HTML
  Parameters: {"authenticity_token"=>"8sQIozxyvbutpnj5JVLf/cZ3DjSdwRuEr13wUm13qo177p5tw1gJrzUcKTtCXiHLnEWicDtnZldxGOpirvcKQg==", "namespace_id"=>"unixfox", "id"=>"test"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 36ms (Views: 0.3ms | ActiveRecord: 2.2ms)
Redirected to https://gitlab.aresrpg.fr/unixfox/test
Completed 302 Found in 39ms (ActiveRecord: 3.0ms)
Started GET "/Dofus/Tofumanchou.git/info/refs?service=git-upload-pack" for 128.78.97.54 at 2017-01-18 14:19:25 +0100
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"Dofus", "project_id"=>"Tofumanchou.git"}
Started GET "/unixfox/test" for 213.49.60.167 at 2017-01-18 14:19:25 +0100
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"test"}
Filter chain halted as :assign_ref_vars rendered or redirected
Completed 404 Not Found in 43ms (Views: 3.3ms | ActiveRecord: 2.6ms)
Completed 500 Internal Server Error in 119ms (ActiveRecord: 2.7ms)

JWT::VerificationError (Signature verification raised):
  lib/gitlab/workhorse.rb:137:in `decode_jwt'
  lib/gitlab/workhorse.rb:133:in `verify_api_request!'
  app/controllers/concerns/workhorse_request.rb:11:in `verify_workhorse_api!'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

[2]

I imported a repository then I made a push to trigger the slack notification but I didn’t receive any notification.

import job started for unixfox/aur with job ID 6362becce4815fd7b38c02e6
Redirected to https://gitlab.aresrpg.fr/unixfox/aur
Completed 302 Found in 950ms (ActiveRecord: 27.4ms)
Started GET "/unixfox/aur" for 213.49.60.167 at 2017-01-18 14:12:42 +0100
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"aur"}
Redirected to https://gitlab.aresrpg.fr/unixfox/aur/import
Completed 302 Found in 50ms (ActiveRecord: 3.2ms)
Started GET "/unixfox/aur/import" for 213.49.60.167 at 2017-01-18 14:12:42 +0100
Processing by Projects::ImportsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "project_id"=>"aur"}
Completed 200 OK in 74ms (Views: 29.6ms | ActiveRecord: 5.2ms)
Updating repository size for project 91
Started GET "/unixfox/aur/import" for 213.49.60.167 at 2017-01-18 14:12:48 +0100
Processing by Projects::ImportsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "project_id"=>"aur"}
Redirected to https://gitlab.aresrpg.fr/unixfox/aur
Completed 302 Found in 56ms (ActiveRecord: 4.1ms)
Started GET "/unixfox/aur" for 213.49.60.167 at 2017-01-18 14:12:48 +0100
Processing by ProjectsController#show as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"aur"}
Read fragment views/unixfox/aur-ecd679ffcf1c1aabbfae206b583d149d45db6c5c-readme/716bb814ea5cebdd8fa6bb16d4b0cdce (0.3ms)
Completed 200 OK in 206ms (Views: 136.5ms | ActiveRecord: 21.4ms)
Started GET "/unixfox/aur/edit" for 213.49.60.167 at 2017-01-18 14:12:54 +0100
Processing by ProjectsController#edit as HTML
  Parameters: {"namespace_id"=>"unixfox", "id"=>"aur"}
Completed 200 OK in 125ms (Views: 76.8ms | ActiveRecord: 9.0ms)
Started GET "/unixfox/aur/services" for 213.49.60.167 at 2017-01-18 14:12:58 +0100
Processing by Projects::ServicesController#index as HTML
  Parameters: {"namespace_id"=>"unixfox", "project_id"=>"aur"}
Completed 200 OK in 475ms (Views: 46.3ms | ActiveRecord: 7.3ms)
Started GET "/unixfox/aur/services/slack/edit" for 213.49.60.167 at 2017-01-18 14:13:02 +0100
Processing by Projects::ServicesController#edit as HTML
  Parameters: {"namespace_id"=>"unixfox", "project_id"=>"aur", "id"=>"slack"}
Completed 200 OK in 131ms (Views: 46.7ms | ActiveRecord: 4.8ms)
Started PUT "/unixfox/aur/services/slack" for 213.49.60.167 at 2017-01-18 14:13:06 +0100
Processing by Projects::ServicesController#update as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"xx/8AShbH1CqlvJUevIdJPlyhhtn2ZWA==", "service"=>{"active"=>"1", "push_events"=>"1", "push_channel"=>"", "issues_events"=>"1", "issue_channel"=>"", "confidential_issues_events"=>"1", "confidential_issue_channel"=>"", "merge_requests_events"=>"1", "merge_request_channel"=>"", "note_events"=>"1", "note_channel"=>"", "tag_push_events"=>"1", "tag_push_channel"=>"", "build_events"=>"0", "build_channel"=>"", "pipeline_events"=>"0", "pipeline_channel"=>"", "wiki_page_events"=>"1", "wiki_page_channel"=>"", "webhook"=>"[FILTERED]", "username"=>"bot.lennon", "notify_only_broken_builds"=>"0", "notify_only_broken_pipelines"=>"0"}, "namespace_id"=>"unixfox", "project_id"=>"aur", "id"=>"slack"}
Redirected to https://gitlab.aresrpg.fr/unixfox/aur/services/slack/edit
Completed 302 Found in 94ms (ActiveRecord: 6.2ms)
Started GET "/unixfox/aur/services/slack/edit" for 213.49.60.167 at 2017-01-18 14:13:06 +0100
Processing by Projects::ServicesController#edit as HTML
  Parameters: {"namespace_id"=>"unixfox", "project_id"=>"aur", "id"=>"slack"}
Completed 200 OK in 147ms (Views: 67.0ms | ActiveRecord: 4.6ms)
Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2017-01-18 14:13:14 +0100
Started GET "/Dofus/Tofumanchou.git/info/refs?service=git-upload-pack" for 128.78.97.54 at 2017-01-18 14:13:25 +0100
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"Dofus", "project_id"=>"Tofumanchou.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 38ms (Views: 0.3ms | ActiveRecord: 2.5ms)
Started GET "/Dofus/Tofumanchou.git/info/refs?service=git-upload-pack" for 128.78.97.54 at 2017-01-18 14:13:25 +0100
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"Dofus", "project_id"=>"Tofumanchou.git"}
Completed 500 Internal Server Error in 127ms (ActiveRecord: 3.5ms)

JWT::VerificationError (Signature verification raised):
  lib/gitlab/workhorse.rb:137:in `decode_jwt'
  lib/gitlab/workhorse.rb:133:in `verify_api_request!'
  app/controllers/concerns/workhorse_request.rb:11:in `verify_workhorse_api!'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'


Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2017-01-18 14:13:34 +0100
Started POST "/api/v3/internal/allowed" for 127.0.0.1 at 2017-01-18 14:13:34 +0100
Started GET "/api/v3/internal/broadcast_message" for 127.0.0.1 at 2017-01-18 14:13:35 +0100
Started GET "/api/v3/internal/merge_request_urls?project=/var/lib/gitlab/repositories/unixfox/aur.git&changes=ecd679ffcf1c1aabbfae206b583d149d45db6c5c%20f7841da657374ebd5253f0423f1eac08b9918303%20refs/heads/master%0A" for 127.0.0.1 at 2017-01-18 14:13:35 +0100

rake gitlab:check

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by gitlab:gitlab?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
AresRpg-archive / AresRpg ... ok
AresRpg-archive / SceatNetwork ... ok
AresRpg-archive / SceatProxy ... ok
AresRpg-archive / SPhantom ... ok
AresRpg-archive / SceatHub ... ok
AresRpg-archive / Symbiote ... ok
Infra / ScorchedRoot ... ok
Commons / Documentation ... ok
Commons / AresCommons ... ok
AresRpg-archive / AresInternal ... ok
AresRpg-archive / UUIDServer ... ok
Systems / teamcity-gitlab-auth-plugin ... ok
Jefyx / Jefyx ... repository is empty
Eywa / Eywa ... ok
AresRpg-archive / AresNetwork ... ok
Gradle / Obfuscation ... ok
Gradle / Api ... ok
Sceat / Quetzal-old ... ok
Quetzal / Quetzal-Old ... ok
Infra / Backup ... ok
Systems / Compileck ... repository is empty
Infra / PhantomEngine ... ok
Systems / gocd-gitlab-auth-plugin ... ok
Systems / gocd-gitlab-unit-report ... ok
Web / wait-template ... ok
Quetzal / MinecraftProtocol ... ok
Infra / AMI-Builder ... ok
Web / website ... ok
AresRPG / AresRPG-STD ... ok
Infra / RESTful ... ok
Quetzal / Quetzal ... ok
Quetzal / QuetzalBridge ... ok
Web / DeveloperDashboard ... ok
Sceat / AresV1 ... repository is empty
AresRPG / AresRPG-INFRA ... ok
Web / portail ... ok
AresRPG / models ... ok
AresRPG / AresRPG-CLASSIC ... ok
Sceat / JawadBot ... ok
Dofus / BotFather ... ok
Dofus / dofus-protocol-1.29 ... ok
Dofus / Tofumanchou ... ok
Running /usr/share/webapps/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/lib/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... no
  Try fixing it:
  Install the init script
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
projects have namespace: ... 
AresRpg-archive / AresRpg ... yes
AresRpg-archive / SceatNetwork ... yes
AresRpg-archive / SceatProxy ... yes
AresRpg-archive / SPhantom ... yes
AresRpg-archive / SceatHub ... yes
AresRpg-archive / Symbiote ... yes
Infra / ScorchedRoot ... yes
Commons / Documentation ... yes
Commons / AresCommons ... yes
AresRpg-archive / AresInternal ... yes
AresRpg-archive / UUIDServer ... yes
Systems / teamcity-gitlab-auth-plugin ... yes
Jefyx / Jefyx ... yes
Eywa / Eywa ... yes
AresRpg-archive / AresNetwork ... yes
Gradle / Obfuscation ... yes
Gradle / Api ... yes
Sceat / Quetzal-old ... yes
Quetzal / Quetzal-Old ... yes
Infra / Backup ... yes
Systems / Compileck ... yes
Infra / PhantomEngine ... yes
Systems / gocd-gitlab-auth-plugin ... yes
Systems / gocd-gitlab-unit-report ... yes
Web / wait-template ... yes
Quetzal / MinecraftProtocol ... yes
Infra / AMI-Builder ... yes
Web / website ... yes
AresRPG / AresRPG-STD ... yes
Infra / RESTful ... yes
Quetzal / Quetzal ... yes
Quetzal / QuetzalBridge ... yes
Web / DeveloperDashboard ... yes
Sceat / AresV1 ... yes
AresRPG / AresRPG-INFRA ... yes
Web / portail ... yes
AresRPG / models ... yes
AresRPG / AresRPG-CLASSIC ... yes
Sceat / JawadBot ... yes
Dofus / BotFather ... yes
Dofus / dofus-protocol-1.29 ... yes
Dofus / Tofumanchou ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/usr/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.0)
Active users: 6

Checking GitLab ... Finished

rake gitlab:env:info

System information
System:		
Current User:	gitlab
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	
Bundler Version:unknown
Rake Version:	10.5.0
Sidekiq Version:4.2.7

GitLab information
Version:	8.15.4
Revision:	fatal: Not a git repository (or any of the parent directories): .git
Directory:	/usr/share/webapps/gitlab
DB Adapter:	postgresql
URL:		https://gitlab.aresrpg.fr
HTTP Clone URL:	https://gitlab.aresrpg.fr/some-group/some-project.git
SSH Clone URL:	gitlab@gitlab.aresrpg.fr:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	4.1.1
Repository storage paths:
- default: 	/var/lib/gitlab/repositories/
Hooks:		/usr/share/webapps/gitlab-shell/hooks/
Git:		/usr/bin/git

Note: Bundler version is unknown because I’m using the package ruby-2.3 who provides only this following binary: gem-2.3, bundler-2.3 and ruby-2.3. The version of ruby is shown because I did a symlink of the command ruby-2.3 to ruby because Gitlab was complaining about ruby command not found when I made git push.

Configuration files

PS: I’m sorry if I made some spelling mistakes because I’m still learning English.

Thank you in advance,

Emilien

I am having a similar problem. Did you ever find the fix?

I didn’t find any solution. I got fed up with Gitlab so I migrated to Gitea a few months ago.