Use MinIO as Application ObjectStore, Azure SA as Backup Destination

Infrastructure as Code & Cloud Native
, ,
1

Hi everyone,

we are using Helm to deploy GitLab to Kubernetes. Everything is working so far; the MinIO from the minio sub-chart is used to backup the instance.
This also works - when triggered, a backup is created in bucket gitlab-backups.

Problem to solve

We want to use the K8s-Cluster internal MinIO service from the minio sub-chart as an object store for all the usual GitLab locations like gitlab-packages and gitlab-uploads.
But when backups are created, these should be saved in an Azure Storage Account instead of that MinIO service.

When we configure everything as documented at Backup and restore a GitLab instance | GitLab, a Backup containing some data is created and uploaded to the Azure Storage Account successfully, but you can see from the Job Logs that those elements from MinIO service are skipped. It looks like as soon as the backup backend is switched to azure instead of s3, backup-utility also looks for those buckets in azure instead of the MinIO object store:

Output when gitlab.toolbox.backups.objectStorage.backend: s3:

...
2024-07-03 16:38:25 UTC -- Dumping repositories ... done
2024-07-03 16:38:25 +0000 -- Deleting backup and restore lock file
Dumping registry ...
done
Dumping uploads ...
done
Dumping lfs ...
empty
Dumping packages ...
done
Dumping external_diffs ...
empty
Dumping terraform_state ...
empty
Dumping pages ...
empty
Dumping ci_secure_files ...
empty
Packing up backup tar
...

Output when gitlab.toolbox.backups.objectStorage.backend: azure:

...
2024-07-03 16:45:06 UTC -- Dumping repositories ... done
2024-07-03 16:45:06 +0000 -- Deleting backup and restore lock file
Bucket not found: registry. Skipping backup of registry ...
Bucket not found: gitlab-uploads. Skipping backup of uploads ...
Bucket not found: git-lfs. Skipping backup of lfs ...
Bucket not found: gitlab-packages. Skipping backup of packages ...
Bucket not found: gitlab-mr-diffs. Skipping backup of external_diffs ...
Bucket not found: gitlab-terraform-state. Skipping backup of terraform_state ...
Bucket not found: gitlab-pages. Skipping backup of pages ...
Bucket not found: gitlab-ci-secure-files. Skipping backup of ci_secure_files ...
Packing up backup tar
...

(How) can just the backup be configured to be uploaded to the Azure StorageAccount, but have backup-utility copy the data from the object store locations as GitLab is configured to use?

Steps to reproduce

  1. Configure MinIO sub-chart to be used as an object Store
  2. Upload some data to object Store Buckets, like user avatars by setting a profile picture
  3. Configure Azure as Backup Destination as described in Backup and restore a GitLab instance | GitLab

Configuration

global:
  edition: ce

  ingress:
    enabled: false

  appConfig:
    backups:
      bucket: gitlab-backup-storage
      tmpBucket: gitlab-tmp-storage

certmanager:
  install: false
  installCRDs: false

nginx-ingress:
  enabled: true

prometheus:
  install: false

registry:
  enabled: true

gitlab-runner:
  install: false

gitlab:
  sidekiq:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce

  toolbox:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-toolbox-ce
    backups:
      objectStorage:
        backend: s3
        config:
          ## Minio
          secret: my-minio-secret
          key: config
      cron:
        enabled: true
        schedule: "0 4 * * *"
        extraArgs: "--skip artifacts"

minio:
  persistence:
    size: 512Gi