Reinstall GitLab with old data

General
1

Good day.
I made an oopsie with my GitLab install last week Friday. I messed up the permissions for everything and now I can’t get my server back up and running. Can I reinstall GitLab and just copy the files to the new instance?

For those who would like to troubleshoot it, here is my output for gitlab-ctl tail | grep error:

2022-08-22_06:53:52.51090 error: stat of /var/log/gitlab/gitlab-workhorse/*.log failed: Permission denied
2022-08-22_07:32:30.98287 error: error opening /var/log/gitlab/puma/puma_stdout.log: Permission denied
2022-08-22_10:17:19.00038 error: error setting owner of /var/log/gitlab/puma/puma_stderr.log.1 to uid 984 and gid 0: Operation not permitted
{"level":"warning","msg":"[core] [Channel #4 SubChannel #5] grpc: addrConn.createTransport failed to connect to {\n  \"Addr\": \"/var/opt/gitlab/gitaly/run/gitaly-40077/sock.d/ruby.1\",\n  \"ServerName\": \"/var/opt/gitlab/gitaly/run/gitaly-40077/sock.d/ruby.1\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/run/gitaly-40077/sock.d/ruby.1: connect: no such file or directory\"","pid":40077,"system":"system","time":"2022-08-22T10:34:42.188Z"}
	from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.3.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-08-22T12:39:38+02:00","uri":""}
{"correlation_id":"01GB2J63KFRAZCWZ7KRY6D57C6","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-08-22T12:39:45+02:00","uri":"/"}
{"correlation_id":"01GB2J645Z16XC9KQNTH760XCK","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-08-22T12:39:45+02:00","uri":"/favicon.ico"}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-08-22T12:39:48+02:00","uri":""}
{"correlation_id":"01GB2J6AZTTHJCD6YX4G66MWBE","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"POST","msg":"","time":"2022-08-22T12:39:52+02:00","uri":"/api/v4/jobs/request"}
{"correlation_id":"01GB2J6BR2AP2X03Y512HJBKXG","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"POST","msg":"","time":"2022-08-22T12:39:53+02:00","uri":"/api/v4/jobs/request"}
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
2022-08-22_10:17:52.28973 level=error ts=2022-08-22T10:17:52.289Z caller=manager.go:316 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2022-08-22_10:17:52.28987 level=error ts=2022-08-22T10:17:52.289Z caller=manager.go:316 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2022-08-22_10:17:52.28987 level=error ts=2022-08-22T10:17:52.289Z caller=manager.go:316 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2022-08-22_10:17:57.29773 level=error ts=2022-08-22T10:17:57.297Z caller=manager.go:196 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-nodes
2022-08-22_10:17:57.29775 level=error ts=2022-08-22T10:17:57.297Z caller=manager.go:196 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-pods
2022-08-22_10:17:57.29776 level=error ts=2022-08-22T10:17:57.297Z caller=manager.go:196 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-cadvisor
2022-08-22_10:17:49.14924 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2022-08-22T12:17:49.149239+02:00","queryType":"random_walk_with_error"}
2022-08-22_10:17:49.14926 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2022-08-22T12:17:49.149248+02:00","queryType":"server_error_500"}
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2022-08-22T12:39:58+02:00","uri":""}
2

You need a backup from /var/opt/gitlab/backups directory, and need to restore it to the same version that you had installed. You also need gitlab.rb and gitlab-secrets.json from /etc/gitlab directory. See the backup/restore docs: Back up and restore GitLab | GitLab the docs explain the steps to restore the data.

3

Does the /etc/gitlab/gitlab.rb have to be the exact same? Can’t I just reconfigure the new one? I don’t mind having a new instance, I just need to get my repos back

4

It doesn’t have to be the same one, but you will need to make sure you configure it identical to the previous one, otherwise you will have some functions enabled, and some disabled. The gitlab-secrets.json will need to be the same, else you will have problems with passwords, access tokens, 2FA, etc.

5

That is fine. Thank you… The only issue I have is getting the repositories back

6

I am trying to restore my backup, but I get this error:
ERROR: column application_settings.max_number_of_repository_downloads does not exist

Is there anything I can do?

7

Is there a way to add users through the shell because I restored my backup, but now I can’t log in. It keeps saying that the password is incorrect and it’s not. I can’t log in at all

8
9

I got it right to create an admin user. Thank you. I am busy importing my projects from the backup, because I can see that the projects exist, they show up, but when I try to navigate to them I get a 404

10

So I got my repos back onto the server. Is there any way to get the issues as well? I don’t think they’re stored with the repository

11

Issues would be stored in the postgresql database, which you would have had to have restored using the backup under /var/opt/gitlab/backups. If you didn’t do this, or don’t have a backup of the database then no you cannot restore it.

12

I thought so… At least I used Discord to send notifications when issues are logged, so that helps. Thank you for not just leaving me in the dark. I appreciate it… I got all my repositories up and running again, I had to create the users and groups again, but that is the easy part

13

One last thing. I see that I can access the .ssh folder of the old instance in my backups folder, can I just copy it to the new one to reinstate my users’ ssh keys or do we all have to copy new keys to the instance?

14

I think they are linked to gitlab-secrets.json so if this wasn’t copied, then you need to set up the ssh keys again from zero, just like with access tokens, 2FA, etc.

If you just copy them, they won’t be connected to any users that you had to create manually.

15

OK, not an issue… The only thing left to do now is to set up automatic snapshots haha