Hi everybody,
little preface: damaged gitlab vm image, it got silently corrupted somehow and I cannot boot it anymore, backups are silently corrupted too… shitty situation.
I’ve already reinstalled and restored from local git repos, but I have some repos that where not used and that are “trapped” into the corrupted server image.
I can mount it and I can see that all the files are there in /var/opt/gitlab/git-data/repositories/
Is there something I can do to recover the missing repos? maybe on a clean install replace that folder and the postgres one and try to start services that way?
Thanks in advance for any help you can give.
update… I’ve managed to bring up the machine and login to the admin area, projects area gives error 500
gitlab-ctl tail shows this, everything else in the web ui seems ok
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin/projects" for 88.149.149.195 at 2016-11-24 15:51:22 +0000
Processing by Admin::ProjectsController#index as HTML
Completed 500 Internal Server Error in 105ms (ActiveRecord: 2.4ms)
ActionView::Template::Error (Object not found - no match for id (d2d67064cc01fd2aec5e4b6fda6686cf4b0eb2d6)):
77: = link_to [:admin, project.namespace.becomes(Namespace), project] do
78: .dash-project-avatar
79: .avatar-container.s40
80: = project_icon(project, alt: '', class: 'avatar project-avatar s40')
81: %span.project-full-name
82: %span.namespace-name
83: - if project.namespace
app/models/repository.rb:519:in `root_ref'
app/models/repository.rb:48:in `block (2 levels) in cache_method'
lib/repository_cache.rb:20:in `fetch'
app/models/repository.rb:1162:in `cache_method_output'
app/models/repository.rb:48:in `block in cache_method'
app/models/repository.rb:625:in `head_commit'
app/models/repository.rb:636:in `tree'
app/models/repository.rb:1176:in `file_on_head'
app/models/repository.rb:569:in `avatar'
app/models/repository.rb:48:in `block (2 levels) in cache_method'
lib/repository_cache.rb:20:in `fetch'
app/models/repository.rb:1162:in `cache_method_output'
app/models/repository.rb:48:in `block in cache_method'
app/models/project.rb:812:in `avatar_in_git'
app/models/project.rb:818:in `avatar_url'
app/helpers/application_helper.rb:43:in `project_icon'
app/views/admin/projects/index.html.haml:80:in `block (2 levels) in _app_views_admin_projects_index_html_haml__2025001689257193649_70004303821940'
app/views/admin/projects/index.html.haml:77:in `block in _app_views_admin_projects_index_html_haml__2025001689257193649_70004303821940'
app/views/admin/projects/index.html.haml:67:in `_app_views_admin_projects_index_html_haml__2025001689257193649_70004303821940'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-11-24_15:51:22.17331 2016/11/24 15:51:22 ErrorPage: serving predefined error page: 500
2016-11-24_15:51:22.17338 git.nordest.systems @ - - [2016-11-24 15:51:22.059175855 +0000 UTC] "GET /admin/projects HTTP/1.1" 500 2408 "https://git.nordest.systems/admin" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36" 0.114164
==> /var/log/gitlab/nginx/gitlab_access.log <==
88.149.149.195 - - [24/Nov/2016:15:51:22 +0000] "GET /admin/projects HTTP/2.0" 500 2617 "https://git.nordest.systems/admin" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin/projects" for 88.149.149.195 at 2016-11-24 15:51:22 +0000
Processing by Admin::ProjectsController#index as HTML
Completed 500 Internal Server Error in 61ms (ActiveRecord: 2.6ms)
ActionView::Template::Error (Object not found - no match for id (d2d67064cc01fd2aec5e4b6fda6686cf4b0eb2d6)):
77: = link_to [:admin, project.namespace.becomes(Namespace), project] do
78: .dash-project-avatar
79: .avatar-container.s40
80: = project_icon(project, alt: '', class: 'avatar project-avatar s40')
81: %span.project-full-name
82: %span.namespace-name
83: - if project.namespace
app/models/repository.rb:519:in `root_ref'
app/models/repository.rb:48:in `block (2 levels) in cache_method'
lib/repository_cache.rb:20:in `fetch'
app/models/repository.rb:1162:in `cache_method_output'
app/models/repository.rb:48:in `block in cache_method'
app/models/repository.rb:625:in `head_commit'
app/models/repository.rb:636:in `tree'
app/models/repository.rb:1176:in `file_on_head'
app/models/repository.rb:569:in `avatar'
app/models/repository.rb:48:in `block (2 levels) in cache_method'
lib/repository_cache.rb:20:in `fetch'
app/models/repository.rb:1162:in `cache_method_output'
app/models/repository.rb:48:in `block in cache_method'
app/models/project.rb:812:in `avatar_in_git'
app/models/project.rb:818:in `avatar_url'
app/helpers/application_helper.rb:43:in `project_icon'
app/views/admin/projects/index.html.haml:80:in `block (2 levels) in _app_views_admin_projects_index_html_haml__2025001689257193649_70004320818380'
app/views/admin/projects/index.html.haml:77:in `block in _app_views_admin_projects_index_html_haml__2025001689257193649_70004320818380'
app/views/admin/projects/index.html.haml:67:in `_app_views_admin_projects_index_html_haml__2025001689257193649_70004320818380'
lib/gitlab/request_profiler/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:16:in `call'
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-11-24_15:51:22.34818 2016/11/24 15:51:22 ErrorPage: serving predefined error page: 500
2016-11-24_15:51:22.34819 git.nordest.systems @ - - [2016-11-24 15:51:22.278662204 +0000 UTC] "GET /admin/projects HTTP/1.1" 500 2408 "https://git.nordest.systems/admin" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36" 0.069440
==> /var/log/gitlab/nginx/gitlab_access.log <==
88.149.149.195 - - [24/Nov/2016:15:51:22 +0000] "GET /admin/projects HTTP/2.0" 500 2617 "https://git.nordest.systems/admin" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"
health check says everything is ok but I found errors in this logs:
this is freaking long so i patebin:
sidekiq.log
https://pastebin.com/raw/s5hwdFn1
repocheck.log
# Logfile created on 2016-11-24 15:20:02 +0000 by logger.rb/54362
E, [2016-11-24T15:20:02.801591 #567] ERROR -- : command failed: nice git --git-dir=/var/opt/gitlab/git-data/repositories/nes/uboot-dotnet.git fsck
error: file /var/opt/gitlab/git-data/repositories/nes/uboot-dotnet.git/objects/pack/pack-ddc9e1672deedbf39a945f689cf321a04f574366.pack is not a GIT packfile
fatal: packfile /var/opt/gitlab/git-data/repositories/nes/uboot-dotnet.git/objects/pack/pack-ddc9e1672deedbf39a945f689cf321a04f574366.pack cannot be accessed
E, [2016-11-24T15:20:02.850554 #567] ERROR -- : command failed: nice git --git-dir=/var/opt/gitlab/git-data/repositories/nes/uboot-unity.git fsck
error: file /var/opt/gitlab/git-data/repositories/nes/uboot-unity.git/objects/pack/pack-1b9e5ba1e8c951e5aa085e9bed62be51a2717a2a.pack is not a GIT packfile
fatal: packfile /var/opt/gitlab/git-data/repositories/nes/uboot-unity.git/objects/pack/pack-1b9e5ba1e8c951e5aa085e9bed62be51a2717a2a.pack cannot be accessed
in sidekiq.log seems that redis is not running but actually is
[root@git ~]# gitlab-ctl status redis
run: redis: (pid 505) 947s; run: log: (pid 504) 947s
[root@git ~]# ls -lsah /var/opt/gitlab/redis/redis.socket
0 srwxrwxrwx. 1 gitlab-redis gitlab-redis 0 24 nov 15.33 /var/opt/gitlab/redis/redis.socket
[root@git ~]# ps ax | grep redis
485 ? Ss 0:00 runsv redis
504 ? S 0:00 svlogd -tt /var/log/gitlab/redis
505 ? Ssl 0:05 /opt/gitlab/embedded/bin/redis-server 127.0.0.1:0
do you have any suggestions?
Upping the post once. is anyone having a suggestion about this?