Restore from backup - 500 Internal Error

How to Use GitLab Upgrade
1

I have am restoring a gitlab backup on a new server (same version which is 8.0.5). Goal is to upgrade to 13.x, but I needed to move to a new EC2 instance so hence the same version.

  • Went from source install to omnibus
  • Backup restored just fine
  • Check shows nothing wrong
  • DB is now postgresql (was mysql) and looks ok.

When I login, the tail output shows the following

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/users/sign_in" for 127.0.0.1 at 2021-05-19 23:59:46 +0000
Processing by SessionsController#new as HTML
Completed 500 Internal Server Error in 47ms (ActiveRecord: 7.0ms)

NoMethodError (undefined method `namespace' for nil:NilClass):
  app/models/project_services/gitlab_issue_tracker_service.rb:35:in `project_url'
  app/models/project.rb:374:in `default_issue_tracker'
  app/controllers/application_controller.rb:195:in `add_gon_variables'

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.124.52.33 - - [19/May/2021:23:59:47 +0000] "GET /users/sign_in HTTP/1.1" 500 415 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
10.124.52.33 - - [19/May/2021:23:59:47 +0000] "GET /favicon.ico HTTP/1.1" 200 5430 "http://10.232.134.84/users/sign_in" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36

Any ideas what could be causing this?

2

That looks like something wrong in your DB. When you migrated from mysql to postgresql, nothing happened?

How about cleaning up cache?

sudo gitlab-rake cache:clear

3

One thing I had to skip was the secrets file. In the source file installation it was called .secret and in my new install it is a JSON file.

I’m wondering if ignoring this file is the cause or not.

4

Migration was without any errors. I’ll try that cache clear

5

Did not help :frowning:

6

Hmm…Can you check if namespace table or projects table is in your DB?

Line 71 shows projects table has a relationship with namespace table, and your error shows an undefined method `namespace’ for nil:NilClass.

https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-0-stable/app/models/project.rb#L71

7

I see tables namespaces and projects

When I dump out the projects table I see a namespace_id column. The values in this column match the ID from rows in the namespaces table

8

The DB conversion showed no errors when I executed it

[/home/git/gitlab/tmp/backups/postgresql]: sudo -u git -H python27 mysql-postgresql-converter/db_converter.py gitlabhq_production.mysql db/database.sql
Line 2105 (of 2105: 100.00%) [59 tables] [135 inserts] [ETA: 0 min 0 sec]

Clearly something about the conversion has failed to translate in postgresql. Not not sure what.

9

I also used PostgreSQL 13.1 in RDS. Should I opt for an older version of PostgreSQL?

10

Hi @cwainwright
you cannot restore without the secrets file. You need to adjust to the JSON format.
The secrets file is essential to preserve your database encryption key.

You can also check doc for steps how to convert Source to Omnibus.

1 Like
11

So I only have db_key_base in the secrets.yml and a .secret file

I don’t have a secret_key_base or otp_key_base

If I read the doc correctly

db_key_base → db_key_base
.secret → secret_token

Hopefully not having secret_key_base will not be an issue.

12

So I applied the changes, reconfigured.

Good news is I get the login screen :slight_smile:
Bad news is my projects are not detected and my login get’s refused.

I guess there is something amiss with my secrets file.

13

Check output

$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...

GitLab Shell version >= 2.6.5 ? ... OK (2.6.5)
    Repo base directory exists? ... yes
    Repo base directory is a symlink? ... no
    Repo base owned by git:git? ... yes
    Repo base access is drwxrws---? ... yes
    hooks directories in repos are links: ... can't check, you have no projects
    Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
    Check GitLab API access: OK
    Check directories and files:
    	/var/opt/gitlab/git-data/repositories: OK
    	/var/opt/gitlab/.ssh/authorized_keys: OK
    Test redis-cli executable: redis-cli 2.8.21
    Send ping to redis server: PONG
    gitlab-shell self-check successful

    Checking GitLab Shell ... Finished

    Checking Sidekiq ...

    Running? ... yes
    Number of Sidekiq processes ... 1

    Checking Sidekiq ... Finished

    Checking Reply by email ...

    Reply by email is disabled in config/gitlab.yml

    Checking Reply by email ... Finished

    Checking LDAP ...

    LDAP is disabled in config/gitlab.yml

    Checking LDAP ... Finished

    Checking GitLab ...

    Git configured with autocrlf=input? ... yes
    Database config exists? ... yes
    Database is SQLite ... no
    All migrations up? ... yes
    Database contains orphaned GroupMembers? ... no
    GitLab config exists? ... yes
    GitLab config outdated? ... no
    Log directory writable? ... yes
    Tmp directory writable? ... yes
    Uploads directory setup correctly? ... yes
    Init script exists? ... skipped (omnibus-gitlab has no init script)
    Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
    projects have namespace: ... can't check, you have no projects
    Redis version >= 2.4.0? ... yes
    Ruby version >= 2.1.0 ? ... yes (2.1.6)
    Your git bin path is "/opt/gitlab/embedded/bin/git"
    Git version >= 1.7.10 ? ... yes (2.4.3)
    Active users: 0

    Checking GitLab ... Finished
14

Here is my JSON secrets file.

I paste the db_key_base into “gitlab_rails”, but after reconfigure, it gets removed. It is like it wants to remain in “gitlab_ci”. Same for the value of db_key_base under “gitlab_ci”, keeps reverting to some other value.

{
  "gitlab_shell": {
    "secret_token": "Value from .secret"
  },
  "gitlab_rails": {
    "secret_token": "Value from .secret"
  },
  "gitlab_ci": {
    "secret_token": null,
    "secret_key_base": "6cdb4905645da78d1dceefe9f40a911e22536cf50355243071178fe30dafff85ee87a7911176744af825217cdc7b3c456090dccf45376c2a27f2e78ac59740fc",
    "db_key_base": "Value from secrets.yml"
  },
  "mattermost": {
    "service_invite_salt": "268e0f84e32c49f32dda507e3a37f9df73087a2435fb7fa826506b099c3dbac30afe11d9a649eec95d9cff42429894bae878e2002aba7075c579613daf114447",
    "service_public_link_salt": "54ef7c67503730ca1ee9923bd5eab008099b24cbf46f4645aacb6d017dbd5d7464740e57269aa173b3108d1bb879da3dc32142d01ea84fc98246bafd6e53c8b2",
    "service_reset_salt": "2ec2cc7c88b087e9787cc278204441bd23fb108654c25b87f346074ec4ecdd92632ae467bcc24fd5d868d0f27899ab131621c8c86a14c0409adf774cfc21d99d",
    "sql_at_rest_encrypt_key": "f5110f2809d1ef43190b7fe76a64713cf1c3cb05a305dd0ed47a58b94b149580a89adaf1f772ae87d4ee8fe69f4f129b2245c4a955d0c379a2f865c42b87e603"
  }
}
15

If I cat the contents of /var/opt/gitlab/gitlab-rails/etc/secret I see the values for db_key_base and secret_key_base are tye same as the value for db_key_base under “gitlab_ci”.

This is all very confusing

16

So the value of db_key_base I put under “gitlab_ci” gets stored in /var/opt/gitlab/gitlab-rails/etc/secrets.yml (I pasted the wrong file earlier) after the reconfigure and I see this change reported during execution.

The value of /var/opt/gitlab/gitlab-rails/etc/secret has my secret_token which is correct.

Yet I still cannot login or perform a check.

17

So for some strange reason my DB was empty, it was previously populated before I applied the secret. Maybe I accidentally ran a db:reset in the wrong window.

So I reset the DB and restored from latest backup.
My secrets.json is correct, yet I get the original namespace error.

Checkout output reports all is OK.

18

I suggest to narrow down the root cause of the issue. Don’t do the mysql->postgres and source->omnibus at the same time. Do one first and if it’s working do the other one. I would personally start with source->omnibus.
Doing both at the same time makes troubleshooting unnecessary complicated.

19

I thought I had to switch to PostgreSQL with omnibus.

In that case I’ll stick with MySQL first. I don’t see any documentation for omnibus and MySQL? Do I just mirror the settings from database.yml into gitlab.rb?

I just need to find a guide to migrate MySQL to PostgreSQL in omnibus. Though according to the documents I can wait until I’m running 12.x

If that doesn’t work I can stay with source installations.

Lots to try.

20

You are right, Omnibus only works with Postgres. So MySQL->Postgres first :slight_smile: