Ghost projects with no namespace causing HTTP/1.1 500 errors?

Hi,

I deleted a group which contained some projects, but the projects are still in the system somewhere. This is causing intermittent " 500 Whoops, something went wrong on our end." errors.

We are using gitlab-ce version 8.17.5.

The following is the output from sudo gitlab-rake gitlab:check --trace (with large snips to remove irrelevant and potentially revealing information):

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 

[SNIP]

vrt-common-json-contract ... repository is empty
vrt-model-v2 ... repository is empty
vrt-registration ... repository is empty
vrt40-model-v2 ... repository is empty
vrt40-xml-v2 ... repository is empty

[SNIP]

Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)

[SNIP]

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... no
  Try fixing it:
  Install the init script
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
projects have namespace: ... 

[SNIP]

vrt40-model-v2 ... no
  Try fixing it:
  Migrate global projects
  For more information see:
  doc/update/5.4-to-6.0.md in section "#global-projects"
  Please fix the error above and rerun the checks.
vrt40-xml-v2 ... no
  Try fixing it:
  Migrate global projects
  For more information see:
  doc/update/5.4-to-6.0.md in section "#global-projects"
  Please fix the error above and rerun the checks.
vrt-common-json-contract ... no
  Try fixing it:
  Migrate global projects
  For more information see:
  doc/update/5.4-to-6.0.md in section "#global-projects"
  Please fix the error above and rerun the checks.
vrt-model-v2 ... no
  Try fixing it:
  Migrate global projects
  For more information see:
  doc/update/5.4-to-6.0.md in section "#global-projects"
  Please fix the error above and rerun the checks.
vrt-registration ... no
  Try fixing it:
  Migrate global projects
  For more information see:
  doc/update/5.4-to-6.0.md in section "#global-projects"
  Please fix the error above and rerun the checks.

[SNIP]

Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 254

Checking GitLab ... Finished

I know the ID of the group, and the listed projects were in that group. However, I can’t determine the IDs of these projects.

When I attempt to navigate /dashboard/groups I get the 500 error.

The following is (what I believe to be) the relevant output from sudo gitlab-ctl tail:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/dashboard/groups" for 10.60.88.107 at 2017-05-12 14:24:00 +0100
Processing by Dashboard::GroupsController#index as HTML
Completed 500 Internal Server Error in 46ms (ActiveRecord: 8.7ms)

ActionView::Template::Error (undefined method `description' for nil:NilClass):
    1: - group_member = local_assigns[:group_member]
    2: - css_class = '' unless local_assigns[:css_class]
    3: - css_class += " no-description" if group.description.blank?
    4: 
    5: %li.group-row{ class: css_class }
    6:   - if group_member
  app/views/shared/groups/_group.html.haml:3:in `_app_views_shared_groups__group_html_haml__2728067484902628609_70143802506860'
  app/views/dashboard/groups/index.html.haml:11:in `block in _app_views_dashboard_groups_index_html_haml__3412045984334342022_70143906071280'
  app/views/dashboard/groups/index.html.haml:9:in `_app_views_dashboard_groups_index_html_haml__3412045984334342022_70143906071280'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'



==> /var/log/gitlab/gitlab-workhorse/current <==
2017-05-12_13:24:00.79084 2017/05/12 14:24:00 ErrorPage: serving predefined error page: 500
2017-05-12_13:24:00.79102 revcode.revenuedomain.ie @ - - [2017-05-12 14:24:00.722416429 +0100 IST] "GET /dashboard/groups HTTP/1.1" 500 2648 "https://revcode.revenuedomain.ie/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0" 0.068522

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.60.88.107 - - [12/May/2017:14:24:00 +0100] "GET /dashboard/groups HTTP/2.0" 500 2857 "https://revcode.revenuedomain.ie/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0"

Any suggestions as to how I could clean this up?

Thanks,

Éibhear

Have a resolution, though I can’t speak for how safe it is:

  1. Connect to the gitlab database with $ sudo gitlab-rails db
  2. Find the projects that don’t have valid namespaces in the database with select id from projects where namespace_id in (select distinct namespace_id from projects where namespace_id not in (select id from namespaces))
  3. Delete the projects with the command delete from projects where id in ( id1, id2, id3... )

Worked for me.