500 Error and Internal Server Error

I am getting a 500 error on the main page now for some reason. I checked the production.log file and noticed this error.

Completed 500 Internal Server Error in 67ms (ActiveRecord: 18.5ms | Elasticsearch: 0.0ms | Allocations: 11910)

ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "project_topics" does not exist
LINE 8:  WHERE a.attrelid = '"project_topics"'::regclass
                            ^
):

I also ran a rake and the system comes back as fine so I am not sure what is going on.

** Invoke gitlab:check (first_time)
** Invoke gitlab_environment (first_time)
** Execute gitlab_environment
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:check
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.21.0 ? ... OK (13.21.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'helpsystems.com:636'
LDAP authentication... Success
Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
VMware Automation / vmware_automation ... yes
Domain Integrations / domain_integrations ... yes
Active Directory / active_directory ... yes
Testing-Development / testing_development ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.31.0 ? ... yes (2.33.0)
Git user has default SSH configuration? ... yes
Active users: ... 4
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Also here is my system info for this instance…

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.7.4p191
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.14
Git Version:    2.33.0.
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.3.0
Revision:       ceec8accb09
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            http://itgitlab.helpsystems.com
HTTP Clone URL: http://itgitlab.helpsystems.com/some-group/some-project.git
SSH Clone URL:  git@itgitlab.helpsystems.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.21.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Any chance someone may be able to point me in the right direction by chance?

We’re also experiencing this error after upgrading to 14.3.0-ce.0.el7

That is when it happened to us as well.

Can anyone help with this issue or should I go ahead and initiate a rebuild?

The same error after upgrade.

Ruby: ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
GitLab: 14.3.0 (ceec8accb09) FOSS
GitLab Shell: 13.21.0
PostgreSQL: 12.7

Maybe anybody has some workaround to return to the previous version or to fix it?

Hi! have you fixed it? And what do you mean under rebuild?

I have not been able to fix it and thus far I have received no feedback from this forum by way of a fix action. My statement on rebuild means a complete reinstall and starting from scratch as I need to get this working again.

thank you for reply! I restored from backup too

We had the same issue. i could fix it by running the db migration multiple times until there where no migration left. between the runs we also had to run some manuell commands:

gitlab-rake db:migrate
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake db:migrate
gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
gitlab-rake db:migrate

after the migration we had to reboot our server.

The manuell commands where written after the db:migrate command failed.

Does it tell you what the manual commands are that need to be run by chance? How were you coming up with those commands?

That seems to have resolved my issue. All of the down items now show as up and the site seems to be acting normally again. I am going to do some more testing, but thus far this is looking much better! Thank you so much for your response!!

I had the same issue.

db.migrate fixed things. Thanks!

And the same here… followed the instructions on upgrading from 13.12.12, so first upgraded to 14.0.11, waited for all migrations, upgraded to 14.1.7 to be sure, everything working. Upgraded tot 14.4.0… getting missing counters / statistics and seeing ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation “project_topics” does not exist in the log.

The loop of running gitlab-rake db:migrate and then running whatever command it tells you to, fixes it.
For me it took some more loops, here’s my history:

    gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_build_needs,id,'[["build_id"]\, ["build_id_convert_to_bigint"]]']
    gitlab-rake db:migrate
    gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
    gitlab-rake db:migrate
    gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_stages,id,'[["id"]\, ["id_convert_to_bigint"]]']
    gitlab-rake db:migrate
    gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
    gitlab-rake db:migrate
    gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']
    gitlab-rake db:migrate
    gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']
    gitlab-rake db:migrate
    gitlab-rake db:migrate
    gitlab-rake db:migrate
    gitlab-ctl restart

after the last one db:migrate finally didn’t give any output
why does this happen?
I’m lucky this is a clone of the production server, but here I am close to midnight finally seeing a working 14.4.0

The error is too generic and does not tell you exact problem. What you need is to find out what is the real issue. If your site is serving a Internal Server Error, this can be caused by a number of things, such as:

• Server permission
• Server timeout
• Script timeout
• Errors in .htaccess files
• Exceeded PHP memory limit.
• Or some other Web Server Issues

Normally, a permissions issue on the file (or files) would be one cause of the 500 internal server error. The simple solution is to run chmod 644 on the problem file(s) or chmod 755 on the directories.

$ sudo chmod -R 777 /"your files location"

The best way to debug the error depends upon the server and what’s actually running at the time. Consider some debugging tips to help diagnose and fix common causes of this problem.