Gitlab seems to not want to work at all

Hello all. If this is the wrong forum i apologize. I am new to all things gitlab.

I have recently been appointed the new gitlab admin for our local installation at our university. The previous admin unfortunately chose to move on to greener pastures.

Our installation is as follows:
gitlab-ee-13.10.3-ee.0.el7.x86_64
running on
CentOS Linux release 7.9.2009 (Core)

Unfortunately I have inherited a broken gitlab and i cannot seem to figure out how to resolve
our issues.

our gitlab webpages only returns a 502 Whoops, GitLab is taking too much time to respond. error
and various gitlab commands in bash:
gitlab-rails console
gitlab-ctl reconfigure

return the same error message:

System Info:
------------
chef_version=15.14.0
platform=centos
platform_version=7.9.2009
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client

Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 70) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20210430-22009-1e5cdt6” ----
STDOUT: rake aborted!
Feature::InvalidFeatureFlagError: Feature flag ‘usage_data_api’ is already defined in ‘/opt/gitlab/embedded/service/gitlab-rails/config/feature_flags/development/usage_data_api.yml’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:165:in block in load_all_from_path!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:161:in each’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:161:in load_all_from_path!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:146:in block in load_all!’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in each_with_object’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:145:in load_all!' /opt/gitlab/embedded/service/gitlab-rails/lib/feature/definition.rb:102:in reload!’
/opt/gitlab/embedded/service/gitlab-rails/lib/feature.rb:137:in register_definitions' /opt/gitlab/embedded/service/gitlab-rails/config/initializers/0_inject_feature_flags.rb:7:in <top (required)>’
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20210430-22009-1e5cdt6” ----
Ran “bash” “/tmp/chef-script20210430-22009-1e5cdt6” returned 1

Ive looked around on the net, but i am unable to find anyone else with this problem.
Ive also tried to remove the offending usage_data_api.yml file, but then the commands fails with a new error message(unable to load --json file)

There are articles on the net describing how to enable disable features using the browser,
but since our webpages are all returning 502 i cannot use that option.

That being said, i dont even know if the error message as describe above is the root cause for our webpages not working, or if that is something else.

Any and all help appriciated.

regards

noob-gitlab-admin-who-really-dont-know-what-he-is-doing

Was GitLab really broken when you inherited it? That’s not nice.

Do you know what broke the GitLab instance?

That error message sounds like the result of a partial database migration, but it’s not anything I’ve seen before.

As far as i know gitlab was working when my predecessor left, but then we had some network
problems and the maching gitlab was running on, had to be restarted (along with several other machines). Our problems started after this restart.

afaik it seems that the gitlab-rails database migration might not be the problem itself, but that silly: " ‘usage_data_api’ is already defined" error seems to stop both “gitlab-rails console” and “gitlab-ctl reconfigure” from working…I have tried various changes in gitlab.rb (and lots of other things), but my unfamiliarity with gitlab and ruby means that i am unable to get past that error

Everything offered in this post is just guess work, and might leave your GitLab installation worse off than before.

Problems showing after a reboot are often problems that is due to a failure to restart the service properly after some change has been done.

That seems consistent with my assumption that it’s a database migration (perhaps as part of an upgrade) that wasn’t completed.

Maybe you should try the database migration anew? The command to do so is something like gitlab-rake db:migrate, perhaps also upgrade it to the newest database version included in the omnibus package with gitlab-ctl pg-upgrade.

You say that you’ve tried to remove usage_data_api.yml without luck, have you tried emptying it (completely or down to the first line, so it’s still valid yaml)?

Hi nappy73,

I noticed that you are using ee rather than ce. Have you tried to find a Gitlab support team for further debugging? This is a free forum and mainly for those who use ce.

From the log I agree with grove that you might have a broken db. Try to use sudo gitlab-rake db:migrate:status to see if all migrations are UP, and if not, run gitlab-rake db:migrate to try to fix it.

Do this before back up everything.