Gitlab restore from helm to EC2 omnibus, group not working

How to Use GitLab Self-managed
1

I migrated gitlab from EKS(helm installation) to EC2(Omnibus). I did a backup and restore using gitlab-rake utility. Everything in new gitlab works fine, except for groups.

when clicking on groups, I get 500 : Internal Server Error. Need urgent help on it.

Thank you very much in Advance !!

Below is detailed error-

Processing by ProjectsController#show as HTML
Parameters: {“namespace_id”=>“mim”, “id”=>“mim-oem”}
Completed 500 Internal Server Error in 361ms (ActiveRecord: 67.4ms | Elasticsearch: 0.0ms)

ActionView::Template::Error (Object Storage is not enabled):
1: - page_title @project.full_name
2: - page_description @project.description unless page_description
3: - header_title project_title(@project) unless header_title
4: - nav “project”
5: - @left_sidebar = t

2

Hi, thanks for reporting the issue.

  • What GitLab version are you using?
  • Do you see this error only for groups? Is it working fine if you try to create a new group?
  • Are you able to open all projects on your instance?
  • Is it a full error? I suppose there should be a full stack trace right after this piece, it would be useful if you can share it here
3

Thank you very much @atanayno for your response. Below are the answers to your questions -

Gitlab version - GitLab Enterprise Edition [12.2.4-ee]

  1. I cant list (explore groups/your groups) groups . I get - An Error occurred, Please try again…However, I can list projects and from there, I can go to some of the groups

  2. I cant create groups as I cant go to groups list page (and hence, dont see that button)

  3. I can open some of the projects and navigate to their group through breadcrumbs, not all

  4. Logs -

     ActionView::Template::Error (Object Storage is not enabled):
 1: - page_title       @project.full_name
 2: - page_description @project.description    unless page_description
 3: - header_title     project_title(@project) unless header_title
 4: - nav              "project"
 5: - @left_sidebar    = true
 6:


 app/uploaders/object_storage.rb:396:in `storage_for'
 app/uploaders/object_storage.rb:231:in `object_store='
 app/uploaders/object_storage.rb:64:in `upload='
 app/uploaders/object_storage.rb:49:in `retrieve_from_store!'
 app/models/concerns/avatarable.rb:57:in `uncached_avatar_path'
 app/models/concerns/avatarable.rb:53:in `avatar_path'
 app/models/concerns/avatarable.rb:24:in `avatar_url'
 app/helpers/groups_helper.rb:159:in `block in group_title_link'
 app/helpers/groups_helper.rb:158:in `group_title_link'
 app/helpers/groups_helper.rb:75:in `group_title'
 app/helpers/projects_helper.rb:86:in `project_title'
 app/views/layouts/project.html.haml:3:in `_app_views_layouts_project_html_haml__1071149216879965695_69815001191600'
 app/controllers/application_controller.rb:118:in `render'
 app/controllers/application_controller.rb:533:in `block in allow_gitaly_ref_name_caching'
 lib/gitlab/gitaly_client.rb:300:in `allow_ref_name_caching'
 app/controllers/application_controller.rb:532:in `allow_gitaly_ref_name_caching'
 ee/lib/gitlab/ip_address_state.rb:10:in `with'
 ee/app/controllers/ee/application_controller.rb:39:in `set_current_ip_address'
 lib/gitlab/session.rb:11:in `with_session'
4

@saurabh thanks for sharing the details. Seems that the objects were not properly migrated to the local storage. Do you still have the old GitLab EKS instance and object storage available. We don’t yet have a fully documented guide for migrating from a Helm chart instance to Omnibus but in short, the migration can be performed in the following way:

  • install GitLab Omnibus
  • create a clone of your original object storage and connect it to your GitLab Omnibus instance
  • create a backup of your original GitLab instance and move it to GitLab Omnibus instance together with secrets and GitLab configuration file
  • restore the backup
  • use the rake tasks to migrate from the object storage to local storage: Jobs artifacts administration | GitLab
5

Thank you @atanayno , I did perform most of these steps while migration

However, when I perform the last step (rake migration), I get below error, Please help -

 > gitlab-rake gitlab:artifacts:migrate_to_local --trace
> rake aborted!
> Don't know how to build task 'gitlab:artifacts:migrate_to_local' (See the list of available tasks with `rake --tasks`)
> Did you mean?  gitlab:artifacts:migrate
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/task_manager.rb:59:in `[]'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:159:in `invoke_task'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block (2 levels) in top_level'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `each'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:116:in `block in top_level'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:125:in `run_with_threads'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:110:in `top_level'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:83:in `block in run'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:186:in `standard_exception_handling'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/lib/rake/application.rb:80:in `run'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.2/exe/rake:27:in `<top (required)>'
> /opt/gitlab/embedded/bin/rake:23:in `load'
> /opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
> /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
> /opt/gitlab/embedded/bin/bundle:23:in `load'
> /opt/gitlab/embedded/bin/bundle:23:in `<main>'
6

Ah, the migration to local storage was introduced in GitLab 12.4 according to the MR [ObjectStorage] Allow migrating back to local storage for some objects (!16868) · Merge Requests · GitLab.org / GitLab · GitLab, so you’ll need to upgrade both source and destination GitLab instances to at least 12.4 to be able to migrate to the local storage.

7

@atanayno after upgrade, I get below error -

sudo gitlab-rake gitlab:artifacts:migrate_to_local
I, [2021-02-06T07:54:47.310910 #10980] INFO – : Starting transfer of artifacts to local storage
E, [2021-02-06T07:54:47.357946 #10980] ERROR – : Failed to transfer artifact of type trace and ID 1 with error: Object Storage is not enabled for JobArtifactUploader

8

Do you have object storage connected to this GitLab instance? What’s inside your /etc/gitlab/gitlab.rb file? Please make sure to sanitize it before sharing here.