Backup skips the Registry part and says [DISABLED]

How to Use GitLab
,
#1

Backup skips the Registry, though they are on the same filesystem. Restore afterwards fails:

“…cannot open /var/opt/gitlab/backups/registry.tar.gz”

and exits

We are backing the GitLab data without LFS like this:

gitlab-backup create BACKUP=$(date +%s)-gitlab-ee SKIP=lfs
2020-03-27 08:31:25 -0400 – Dumping database …
Dumping PostgreSQL database gitlabhq_production … [DONE]
2020-03-27 08:31:27 -0400 – done
2020-03-27 08:31:27 -0400 – Dumping repositories …

  • root/rarchiver (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) … [DONE]
    [SKIPPED] Wiki
  • root/premium(@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) … [DONE]
    [SKIPPED] Wiki
  • root/test-web (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) … [DONE]
    [SKIPPED] Wiki
  • root/test-web-lfs (@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d) … [DONE]
    [SKIPPED] Wiki
  • test-maintainer/ghfghf (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) … [DONE]
    [SKIPPED] Wiki
    2020-03-27 08:31:56 -0400 – done
    2020-03-27 08:31:56 -0400 – Dumping uploads …
    2020-03-27 08:31:56 -0400 – done
    2020-03-27 08:31:56 -0400 – Dumping builds …
    2020-03-27 08:31:56 -0400 – done
    2020-03-27 08:31:56 -0400 – Dumping artifacts …
    2020-03-27 08:31:56 -0400 – done
    2020-03-27 08:31:56 -0400 – Dumping pages …
    2020-03-27 08:31:56 -0400 – done
    2020-03-27 08:31:56 -0400 – Dumping lfs objects …
    2020-03-27 08:31:56 -0400 – [SKIPPED]
    2020-03-27 08:31:56 -0400 – Dumping container registry images …
    2020-03-27 08:31:56 -0400 – [DISABLED]
    Creating backup archive: 1585312274-gitlab-ee_gitlab_backup.tar … done
    Uploading backup archive to remote storage … skipped
    Deleting tmp directories … done
    done
    done
    done
    done
    done
    done
    Deleting old backups … skipping
    Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
    and are not included in this backup. You will need these files to restore a backup.
    Please back them up manually.
    Backup task is done.

tar tvf /mnt/sda2/backups/1585312274-gitlab-ee_gitlab_backup.tar
drwx------ git/git 0 2020-03-27 08:31 repositories/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/6b/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/6b/86/
-rw-r–r-- git/git 24751 2020-03-27 08:31 repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.bundle
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/
-rw-r–r-- git/git 10240 2020-03-27 08:31 repositories/@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/custom_hooks.tar
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/d4/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/d4/73/
-rw-r–r-- git/git 2286603549 2020-03-27 08:31 repositories/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.bundle
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/4b/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/4b/22/
-rw-r–r-- git/git 37104040 2020-03-27 08:31 repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.bundle
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/ef/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/ef/2d/
-rw-r–r-- git/git 35207806 2020-03-27 08:31 repositories/@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.bundle
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/e7/
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/e7/f6/
-rw-r–r-- git/git 372284 2020-03-27 08:31 repositories/@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683.bundle
drwxr-xr-x git/git 0 2020-03-27 08:31 repositories/@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683/
drwxr-xr-x git/git 0 2020-03-27 08:31 db/
-rw------- git/git 22499261 2020-03-27 08:31 db/database.sql.gz
-rw------- git/git 151 2020-03-27 08:31 uploads.tar.gz
-rw------- git/git 278 2020-03-27 08:31 builds.tar.gz
-rw------- git/git 12265 2020-03-27 08:31 artifacts.tar.gz
-rw------- git/git 156 2020-03-27 08:31 pages.tar.gz
-rw-r–r-- git/git 196 2020-03-27 08:31 backup_information.yml

I tested the following:

  1. Updating gitlab from 12.7.6-ee to 12.9.1-ee.0 on debian 10 without success
  2. Copying the original registry and gzipping it to /var/opt/gitlab/backups/registry.tar.gz and it worked, but requires extra maintenance and is not by design.

Your help is highly appreciated

#2

Can you confirm that you have GitLab registry enabled on this instance? If you’re using letsencrypt integration to generate certificates using an external_url using https://, registry should be enabled and running on port 5050.

To help troubleshoot and find a solution, can you please provide some additional details:

If you run gitlab-ctl status, does it show Registry as ‘up’?

Have you pushed a docker image to your GitLab registry?

Are you able to docker push to and docker pull from your GitLab registry on your local machine?

When the restore fails, can you confirm that GitLab version being restored to is the same as version where backup was created?

Thanks!

#3

Thank you for the quick feedback!

I guess this was the right question. We don’t use let’s encrypt. We are using wildcard certificates and letsencrypt is currently disabled.

The registry service is not running and there is nothing on port 5050:

gitlab-ctl status
run: alertmanager: (pid 29484) 266027s; run: log: (pid 613) 268721s
run: gitaly: (pid 29496) 266026s; run: log: (pid 624) 268721s
run: gitlab-exporter: (pid 29507) 266026s; run: log: (pid 627) 268721s
run: gitlab-workhorse: (pid 29453) 266028s; run: log: (pid 628) 268721s
run: grafana: (pid 29590) 266026s; run: log: (pid 622) 268721s
run: logrotate: (pid 6057) 3225s; run: log: (pid 630) 268721s
run: nginx: (pid 29611) 266025s; run: log: (pid 626) 268721s
run: node-exporter: (pid 29619) 266025s; run: log: (pid 623) 268721s
run: postgres-exporter: (pid 29465) 266027s; run: log: (pid 621) 268721s
run: postgresql: (pid 2691) 268363s; run: log: (pid 632) 268721s
run: prometheus: (pid 29627) 266024s; run: log: (pid 605) 268721s
run: redis: (pid 2714) 268363s; run: log: (pid 629) 268721s
run: redis-exporter: (pid 29642) 266024s; run: log: (pid 620) 268721s
run: sidekiq: (pid 23639) 10330s; run: log: (pid 631) 268721s
run: unicorn: (pid 29656) 266023s; run: log: (pid 640) 268721s

We have a dedicated runner installed on a separate host for CI/CD tests with standard ruby docker image on the runner itself.

We did not try this yet. Our main goal was to test base install, backup and recovery.

Yes, both versions are the same. However your first question points us in the right direction. Our registry service is not running, because we use a self generated certificate and didn’t know additional configuration was needed for the registry service. Currently in /var/opt/gitlab/registry/config.yml, we have only this:

version: 0.1
log:
  level: info
  formatter: text
  fields:
    service: registry
    environment: production
storage: {"filesystem":{"rootdirectory":"/var/opt/gitlab/gitlab-rails/shared/registry"},"cache":{"blobdescriptor":"inmemory"},"delete":{"enabled":true}}
http:
  addr: localhost:5000
  secret: "93328bfc85c0c72cd5ced2333e81deed86599db49be23e9144ee5405ea025e4f6228042172ab87b657f8a6268ad079e6ab7a8441c11bbdd03f30368caa3c594f"
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
auth:
  token:
    realm: https://gitlab.home/jwt/auth
    service: container_registry
    issuer: omnibus-gitlab-issuer
    rootcertbundle: /var/opt/gitlab/registry/gitlab-registry.crt
    autoredirect: false
validation:
  disabled: true

File “/var/opt/gitlab/registry/gitlab-registry.crt” exists and is a certificate actually.

Can we do something to start the registry service in this situation and retry the backup?

Kind regards,
Todor

#4

Hi, Greg,

We brought up the registry service with Omnibus install by editing /etc/gitlab/gitlab.rb config:

registry_external_url ‘https://gitlab.example.com:4567’
registry[‘enable’] = true
registry[‘registry_http_addr’] = “localhost:5000”

and custom certs:

nginx[‘ssl_certificate’] = “/etc/gitlab/ssl/gitlab.example.com.crt”
nginx[‘ssl_certificate_key’] = “/etc/gitlab/ssl/gitlab.example.com.key”

The backup now includes the registry.gz file and restore is just fine.

Have a great week and thank you!

Regards,
Todor

1 Like