Internal API available: FAILED - Internal API error (500) -> GitLab 13.6

How to Use GitLab Self-managed
1

Hello,

I installed a new version (13.6) of GitLab from source. Everything went fine but running the check command give Error 500:

--> Starting GitLab
Starting GitLab web server (puma)
Starting GitLab Sidekiq
Starting GitLab Workhorse
Starting Gitaly
{"timestamp":"2020-12-31T08:08:25.726Z","pid":566,"message":"Puma starting in cluster mode..."}
{"timestamp":"2020-12-31T08:08:25.726Z","pid":566,"message":"* Version 4.3.5.gitlab.3 (ruby 2.7.2-p137), codename: Mysterious Traveller"}
{"timestamp":"2020-12-31T08:08:25.726Z","pid":566,"message":"* Min threads: 1, max threads: 16"}
{"timestamp":"2020-12-31T08:08:25.727Z","pid":566,"message":"* Environment: production"}
{"timestamp":"2020-12-31T08:08:25.727Z","pid":566,"message":"* Process workers: 3"}
{"timestamp":"2020-12-31T08:08:25.727Z","pid":566,"message":"* Preloading application"}
{"timestamp":"2020-12-31T08:15:36.477Z","pid":566,"message":"* Listening on unix:///home/git/gitlab/tmp/sockets/gitlab.socket"}
{"timestamp":"2020-12-31T08:15:36.477Z","pid":566,"message":"! WARNING: Detected 2 Thread(s) started in app boot:"}
{"timestamp":"2020-12-31T08:15:36.478Z","pid":566,"message":"! #\u003cThread:0x00000000089764d8 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:334 sleep\u003e - /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.3.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:337:in `sleep'"}
{"timestamp":"2020-12-31T08:15:36.479Z","pid":566,"message":"! #\u003cThread:0x000000001d5bdc00 /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/scheduler.rb:91:in `sleep'"}
{"timestamp":"2020-12-31T08:15:36.480Z","pid":566,"message":"* Daemonizing..."}
Sidekiq output will be written to /home/git/gitlab/log/sidekiq.log
.
The GitLab web server with pid 1465 is running.
The GitLab Sidekiq job dispatcher with pid 1477 is running.
The GitLab Workhorse with pid 1485 is running.
Gitaly with pid 1487 is running.
GitLab and all its components are up and running.

Running check command:

$ sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.13.0 ? ... OK (13.13.0)
Running /home/git/gitlab-shell/bin/check
Internal API available: FAILED - Internal API error (500)
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /home/git/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... no
  Try fixing it:
  Install the init script
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
GitLab Instance / Monitoring ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

So I checked production.log file and I got this:

Started GET "/gitlab/api/v4/internal/check" for 10.42.0.5 at 2020-12-31 12:18:30 +0400
Processing by ApplicationController#route_not_found as HTML
  Parameters: {"unmatched_route"=>"gitlab/api/v4/internal/check", "application"=>{}}
Redirected to https://xxx.xxx.xx/users/sign_in
Completed 302 Found in 246ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 5103)
Started GET "/users/sign_in" for 10.42.0.5 at 2020-12-31 12:18:31 +0400
Processing by SessionsController#new as HTML
  Parameters: {"session"=>{}}
Completed 500 Internal Server Error in 2110ms (ActiveRecord: 169.0ms | Elasticsearch: 0.0ms | Allocations: 118752)
  
ActionView::Template::Error (Could not load compiled manifest from /home/git/gitlab/public/assets/webpack/manifest.json.

Have you run `rake gitlab:assets:compile`?

(original error Errno::ENOENT: No such file or directory @ rb_sysopen - /home/git/gitlab/public/assets/webpack/manifest.json)):
    62: 
    63:   = yield :page_specific_javascripts
    64: 
    65:   = webpack_controller_bundle_tags
    66:   = webpack_bundle_tag "chrome_84_icon_fix" if browser.chrome?([">=84", "<84.0.4147.125"]) || browser.edge?([">=84", "<84.0.522.59"])
    67: 
    68:   = yield :project_javascripts
  
lib/gitlab/webpack/manifest.rb:112:in `rescue in load_static_manifest'
lib/gitlab/webpack/manifest.rb:109:in `load_static_manifest'
lib/gitlab/webpack/manifest.rb:84:in `load_manifest'
lib/gitlab/webpack/manifest.rb:76:in `block in manifest'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/gitlab/webpack/manifest.rb:76:in `manifest'
lib/gitlab/webpack/manifest.rb:67:in `manifest_bundled?'
lib/gitlab/webpack/manifest.rb:30:in `entrypoint_paths'
app/helpers/webpack_helper.rb:39:in `webpack_entrypoint_paths'
app/helpers/webpack_helper.rb:22:in `webpack_controller_bundle_tags'
app/views/layouts/_head.html.haml:65
app/views/layouts/devise.html.haml:3
app/controllers/application_controller.rb:134:in `render'
app/controllers/sessions_controller.rb:60:in `new'
ee/app/controllers/ee/sessions_controller.rb:21:in `new'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'
app/controllers/application_controller.rb:499:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:490:in `set_session_storage'
lib/gitlab/i18n.rb:73:in `with_locale'
lib/gitlab/i18n.rb:79:in `with_user_locale'
app/controllers/application_controller.rb:484:in `set_locale'
lib/gitlab/error_tracking.rb:52:in `with_context'
app/controllers/application_controller.rb:549:in `sentry_context'
app/controllers/application_controller.rb:477:in `block in set_current_context'
lib/gitlab/application_context.rb:54:in `block in use'
lib/gitlab/application_context.rb:54:in `use'
lib/gitlab/application_context.rb:21:in `with_context'
app/controllers/application_controller.rb:469:in `set_current_context'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/multipart.rb:234:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses.
An enum element in UserStatus uses the prefix 'not_'. This will cause a conflict with auto generated negative scopes.
Creating scope :open. Overwriting existing method AlertManagement::Alert.open.
Creating scope :with_associations. Overwriting existing method Board.with_associations.
Creating scope :closed. Overwriting existing method Iteration.closed.
Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe.
Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :with_user. Overwriting existing method GroupMember.with_user.
Creating scope :reporters. Overwriting existing method GroupMember.reporters.
Creating scope :with_logging_info. Overwriting existing method PagesDomain.with_logging_info.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :regular. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.regular.
Creating scope :code_owner. Overwriting existing method EE::Gitlab::BackgroundMigration::MigrateApproverToApprovalRules::ApprovalMergeRequestRule.code_owner.
Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :with_user. Overwriting existing method GroupMember.with_user.
Creating scope :reporters. Overwriting existing method GroupMember.reporters.

Any idea about what is going on ?

2

Please send a Link to the related documentation. I have only found Integrity check Rake task | GitLab, but there is no gitlab:check command.

Edit: I have found the Documentation: Maintenance Rake tasks | GitLab

3

Maybe try to follow the following instructions from
https://docs.gitlab.com/13.6/ee/administration/raketasks/maintenance.html:

  • Rebuild authorized_keys file
  • Clear Redis cache
  • Precompile the assets

The first error indicates that your assets are broken due to some upgrades. In this case, the Precompile assets part should be executed, as in my understanding. Otherwise, I have no Idea :roll_eyes: