Gitlab throws 500 error when accessing page 4 on projects

Hello folks,
I’ve recently upgraded my gitlabs server from 8.0.3 to 10.5.4 as mentioned in given below post
(Upgrade from gitlab-ce 8.0.3-ce.1 to latest)

Recently imported one repo from other server

renamed and deleted old one.

after that I’ve started renaming all the repos on my new server adding prefix-old to all of them.
all went well while renaming projects when I try to hit 4th page of project list on my new server gitlab it throws 500 error

given below is the output of gitlab-ctl tail command

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET “/admin/projects?page=4&sort=latest_activity_desc” for at 2018-03-30 18:04:08 +0530
Processing by Admin::ProjectsController#index as HTML
Parameters: {“page”=>“4”, “sort”=>“latest_activity_desc”}
Completed 500 Internal Server Error in 89ms (ActiveRecord: 12.8ms)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{“method”:“GET”,“path”:"/admin/projects",“format”:“html”,“controller”:“Admin::ProjectsController”,“action”:“index”,“status”:500,“error”:“ActionView::Template::Error: object not found - no match for id (0012ca3818bcd196e5f44dd8eb7fe9b3eb710db3)”,“duration”:90.41,“view”:0.0,“db”:12.81,“time”:“2018-03-30T12:34:08.950Z”,“params”:[{“key”:“page”,“value”:“4”},{“key”:“sort”,“value”:“latest_activity_desc”}],“remote_ip”:"",“user_id”:2,“username”:“pix”,“gitaly_calls”:3}

==> /var/log/gitlab/gitlab-rails/production.log <==

ActionView::Template::Error (object not found - no match for id (0012ca3818bcd196e5f44dd8eb7fe9b3eb710db3)):
20: = link_to [:admin, project.namespace.becomes(Namespace), project] do
21: .dash-project-avatar
22: .avatar-container.s40
23: = project_icon(project, alt: ‘’, class: ‘avatar project-avatar s40’)
24: %span.project-full-name
25: %span.namespace-name
26: - if project.namespace
lib/gitlab/git/repository.rb:1627:in target' lib/gitlab/git/repository.rb:1627:in block (2 levels) in branches_filter’
lib/gitlab/git/repository.rb:1625:in each' lib/gitlab/git/repository.rb:1625:in each’
lib/gitlab/git/repository.rb:1625:in map' lib/gitlab/git/repository.rb:1625:in block in branches_filter’
lib/gitlab/gitaly_client.rb:297:in allow_n_plus_1_calls' lib/gitlab/git/repository.rb:1624:in branches_filter’
lib/gitlab/git/repository.rb:169:in block in branches' lib/gitlab/gitaly_client.rb:255:in block (2 levels) in migrate’
lib/gitlab/gitaly_client.rb:297:in allow_n_plus_1_calls' lib/gitlab/gitaly_client.rb:250:in block in migrate’
lib/gitlab/metrics/influx_db.rb:98:in measure' lib/gitlab/gitaly_client.rb:248:in migrate’
lib/gitlab/git/repository.rb:1338:in gitaly_migrate' lib/gitlab/git/repository.rb:165:in branches’
lib/gitlab/git/repository.rb:158:in block in branch_names' lib/gitlab/gitaly_client.rb:255:in block (2 levels) in migrate’
lib/gitlab/gitaly_client.rb:297:in allow_n_plus_1_calls' lib/gitlab/gitaly_client.rb:250:in block in migrate’
lib/gitlab/metrics/influx_db.rb:98:in measure' lib/gitlab/gitaly_client.rb:248:in migrate’
lib/gitlab/git/repository.rb:1338:in gitaly_migrate' lib/gitlab/git/repository.rb:154:in branch_names’
lib/gitlab/git/repository.rb:371:in discover_default_branch' lib/gitlab/git/repository.rb:118:in block in root_ref’
lib/gitlab/gitaly_client.rb:255:in block (2 levels) in migrate' lib/gitlab/gitaly_client.rb:297:in allow_n_plus_1_calls’
lib/gitlab/gitaly_client.rb:250:in block in migrate' lib/gitlab/metrics/influx_db.rb:98:in measure’
lib/gitlab/gitaly_client.rb:248:in migrate' lib/gitlab/git/repository.rb:1338:in gitaly_migrate’
lib/gitlab/git/repository.rb:114:in root_ref' app/models/repository.rb:496:in root_ref’
app/models/repository.rb:71:in block (2 levels) in cache_method' lib/repository_cache.rb:20:in fetch’
app/models/repository.rb:961:in cache_method_output' app/models/repository.rb:70:in block in cache_method’
app/models/repository.rb:631:in head_commit' app/models/repository.rb:642:in tree’
app/models/repository.rb:979:in file_on_head' app/models/repository.rb:550:in block in avatar’
lib/gitlab/gitaly_client.rb:297:in allow_n_plus_1_calls' app/models/repository.rb:549:in avatar’
app/models/repository.rb:71:in block (2 levels) in cache_method' lib/repository_cache.rb:20:in fetch’
app/models/repository.rb:961:in cache_method_output' app/models/repository.rb:70:in block in cache_method’
app/models/project.rb:917:in avatar_in_git' app/models/project.rb:921:in avatar_url’
app/models/concerns/avatarable.rb:18:in avatar_url' app/helpers/application_helper.rb:43:in project_icon’
app/views/admin/projects/_projects.html.haml:23:in block (2 levels) in _app_views_admin_projects__projects_html_haml__3676912586730327931_70133804095880' app/views/admin/projects/_projects.html.haml:20:in block in _app_views_admin_projects__projects_html_haml__3676912586730327931_70133804095880’
app/views/admin/projects/_projects.html.haml:4:in _app_views_admin_projects__projects_html_haml__3676912586730327931_70133804095880' app/views/admin/projects/index.html.haml:44:in _app_views_admin_projects_index_html_haml___2198546739798069715_70133791829380’
app/controllers/admin/projects_controller.rb:12:in index' lib/gitlab/i18n.rb:47:in with_locale’
lib/gitlab/i18n.rb:53:in with_user_locale' app/controllers/application_controller.rb:325:in set_locale’
lib/gitlab/middleware/multipart.rb:95:in call' lib/gitlab/request_profiler/middleware.rb:14:in call’
lib/gitlab/middleware/go.rb:17:in call' lib/gitlab/etag_caching/middleware.rb:11:in call’
lib/gitlab/middleware/read_only.rb:31:in call' lib/gitlab/request_context.rb:18:in call’
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call’

==> /var/log/gitlab/gitlab-workhorse/current <==
2018-03-30_12:34:09.04464 Server-ip:Port @ - - [2018-03-30 18:04:08.937162614 +0530 IST m=+169814.583343247] “GET /admin/projects?page=4&sort=latest_activity_desc HTTP/1.1” 500 2902 “http://Server-ip:Port/admin/projects” “Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36” 0.107331

==> /var/log/gitlab/nginx/gitlab_access.log <==
- - [30/Mar/2018:18:04:09 +0530] “GET /admin/projects?page=4&sort=latest_activity_desc HTTP/1.1” 500 2902 “http://Server-ip:Port/admin/projects” “Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36”

Hey folks,
Iam able to fix this, not a clean way but what I’ve learned is,

last time I took the dump some how that was broken data for one of the repo in the backup.

so what I did was took fresh dump from old source restored it fresh gitlab instance and upgraded gitlab on this instance to match the version of broken instance and restored recreated backup and restored the backup on broken instance
instance A (version 8.0.3)
Instance B (version 10.5.4) <-this broken

took backup on A
created new instance C (version 8.0.3)
restored backup from A to C
upgraded version of C to 10.5.4
took fresh backup on C and restored it on B

( Note:- given below method is yet to be tested if it actually works.)

This seems to be another way to fix that by simply over writing the directory files in /var/opt/gitlabs
/repositories/path/to/broken/repo
identify the bundel for respective broken repo and tye diffrent possible ways of restoring it (using git bundle options.)

regards