Gitlab-ce stuck with restore w\o secrets

Infrastructure as Code & Cloud Native
1

I have installed fresh Gitlab-ce-14.10. My secrets is gone with part of backups =(

Now im not able to login via Web - i have 500 error on page and this error in log:

Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 97)
Processing by RootController#index as HTML
Redirected to http://gitlab.local/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 12ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 4274)
Started GET "/users/sign_in" for 10.0.200.251 at 2022-07-12 15:36:53 -0400
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Processing by SessionsController#new as HTML
  Rendered layout layouts/devise.html.haml (Duration: 17.6ms | Allocations: 7671)
Completed 500 Internal Server Error in 35ms (ActiveRecord: 1.3ms | Elasticsearch: 0.0ms | Allocations: 13210)

ActionView::Template::Error (Object Storage is not enabled for AttachmentUploader):
    1: %header.navbar.fixed-top.navbar-empty
    2:   = brand_header_logo

app/uploaders/object_storage.rb:442:in `storage_for'
app/uploaders/object_storage.rb:254:in `object_store='
app/uploaders/object_storage.rb:66:in `upload='
app/uploaders/object_storage.rb:51:in `retrieve_from_store!'
app/helpers/appearances_helper.rb:39:in `brand_header_logo'

All guides for secrets cleaning was studied and double checked. All storages disabled. In gitlab.rb i have only http://gitlab.local/
and rewrite for 503 for tasks as per official manual.

Now i decided to upgrade to 14.10.5 and i have an erorr =(

There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

gitlab-rake db:migrate --trace shows me new errors which im trying to fix via database access, but when im fixing one - i have another one =( Whats going on, people?

PG::DuplicateFunction: ERROR: function "set_has_external_issue_tracker" already exists with same argument types
==> REMOVED

Now  =>
 PG::DuplicateTable: ERROR:  relation "product_analytics_events_experimental" already exists
ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR:  relation "product_analytics_events_experimental" already exists

=>THEN:
PG::DuplicateTable: ERROR:  relation "product_analytics_events_experimental" already exists

=>THEN:
ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR:  relation "abuse_reports_id_seq" already exists

=>THEN:
PG::DuplicateTable: ERROR:  relation "alert_management_alert_assignees" already exists
2

Fresh update, reinstalled to version 14.10.0-ce and restored db and repos from backup:

Started GET "/api/v4/geo/proxy" for 127.0.0.1 at 2022-07-13 14:40:21 -0400
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Started GET "/" for 10.0.200.242 at 2022-07-13 14:40:23 -0400
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Processing by RootController#index as HTML
Redirected to http://gitlab.local/users/sign_in
Filter chain halted as :redirect_unlogged_user rendered or redirected
Completed 302 Found in 16ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 5818)
Started GET "/users/sign_in" for 10.0.200.242 at 2022-07-13 14:40:23 -0400
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Processing by SessionsController#new as HTML
  Rendered layout layouts/devise.html.haml (Duration: 226.0ms | Allocations: 177492)
Completed 500 Internal Server Error in 364ms (ActiveRecord: 59.1ms | Elasticsearch: 0.0ms | Allocations: 225602)

ActionView::Template::Error (Object Storage is not enabled for AttachmentUploader):
    1: %header.navbar.fixed-top.navbar-empty
    2:   = brand_header_logo

app/uploaders/object_storage.rb:442:in `storage_for'
app/uploaders/object_storage.rb:254:in `object_store='
app/uploaders/object_storage.rb:66:in `upload='
app/uploaders/object_storage.rb:51:in `retrieve_from_store!'
app/helpers/appearances_helper.rb:39:in `brand_header_logo'
app/views/layouts/header/_empty.html.haml:2
app/views/layouts/devise.html.haml:8
app/controllers/application_controller.rb:142:in `render'
app/controllers/sessions_controller.rb:63:in `new'
app/controllers/application_controller.rb:520:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:511:in `set_session_storage'
lib/gitlab/i18n.rb:105:in `with_locale'
lib/gitlab/i18n.rb:111:in `with_user_locale'
app/controllers/application_controller.rb:505:in `set_locale'
app/controllers/application_controller.rb:499:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:46:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Started GET "/-/metrics" for 127.0.0.1 at 2022-07-13 14:40:29 -0400
Processing by MetricsController#index as HTML
Completed 200 OK in 43ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 1209)
Started GET "/api/v4/geo/proxy" for 127.0.0.1 at 2022-07-13 14:40:31 -0400
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Completed 200 OK in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)

And i have an error 500

500

Whoops, something went wrong on our end.

Request ID: 01G7WDSE5ADZXRGQZDB93GTJH1

Try refreshing the page, or going back and attempting the action again.

Please contact your GitLab administrator if this problem persists.

Maybe someone can help with this? Error looks like depends on some logo?? and AttachmentUploader , but i dont have any attachments in backup.

Trying to list repos via shell:
sudo gitlab-rake gitlab:list_repos

rake aborted!
Gitlab::GitalyClient::StorageSettings::DirectPathAccessError: git disk access denied
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/gitaly_client/storage_settings.rb:65:in legacy_disk_path' /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:110:in path_to_repo’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/list_repos.rake:15:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/list_repos.rake:14:in block (2 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ’
Tasks: TOP => gitlab:list_repos
(See full trace by running task with --trace)

3

Dear Community, problem solved - main issue is that no group/users avatars in backup.
its not funny at all - huge, slow and stupid software Maybe it will help someone next time.

FIX:
find tables with avatar\images and clean:

update TABLE set avatar =‘’;

4

Why are you blaming the Gitlab software for it? You didn’t make a backup of your gitlab-secrets.json so this is the source of your problem. Had you made a backup of this file as well as your gitlab.rb then you wouldn’t have had any problem restoring your system.

I have restored mine multiple times, a procedure I check and test every 90 days or so to ensure that my backups are working successfully, and that I am able to recover in a disaster.

The Gitlab documentation does mention what to do when restoring without gitlab-secrets.json as 2FA will need to be disabled on any users that had it enabled.

Maybe next time you will ensure you have a full backup of your system. It’s not the software that is the problem here…

5

Missing avatar shouldn’t be an issue. 21century, space ships, tesla etc - and software which cannot run w\o image? realy? Its a shame. I dont want to talk about console which starting in 30sec on 6core 16gb ram. Btw, Gitlab documentation can help if you lost secrets file, but it looks like: “hey, fix an error and all will be ok”.
https://docs.gitlab.com/ee/raketasks/backup_restore.html#when-the-secrets-file-is-lost

Now i can get an access to repos via web, not via shell, because of another error - google didnt help.
And few groups throwing error 500 with


Completed 200 OK in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 93)
Processing by GroupsController#show as HTML
  Parameters: {"id"=>"Aws2"}
  Rendered layout layouts/group.html.haml (Duration: 122.7ms | Allocations: 45195)
Completed 500 Internal Server Error in 192ms (ActiveRecord: 11.2ms | Elasticsearch: 0.0ms | Allocations: 55485)

    10:       .d-flex.flex-column.flex-wrap.align-items-baseline
    11:         .d-inline-flex.align-items-baseline
    12:           %h1.home-panel-title.gl-mt-3.gl-mb-2{ itemprop: 'name' }

app/uploaders/object_storage.rb:442:in `storage_for'
app/uploaders/object_storage.rb:254:in `object_store='
app/uploaders/object_storage.rb:66:in `upload='
app/uploaders/object_storage.rb:51:in `retrieve_from_store!'
app/models/concerns/avatarable.rb:76:in `uncached_avatar_path'
app/models/concerns/avatarable.rb:72:in `avatar_path'
app/models/concerns/avatarable.rb:37:in `avatar_url'
app/helpers/avatars_helper.rb:135:in `source_icon'
app/helpers/avatars_helper.rb:11:in `group_icon'
app/views/groups/_home_panel.html.haml:9
app/views/groups/show.html.haml:31
app/controllers/application_controller.rb:142:in `render'
app/controllers/groups_controller.rb:230:in `render_show_html'
app/controllers/groups_controller.rb:98:in `block (2 levels) in show'

Looks like something with storage(another image lost?) and no info about that. Will dig deeper.

FIXED:
Find all tables with column avatar and clean them. For example:
update namespaces SET avatar=‘’