Hi everyone.
We run GitLab in K8s, provided by using Helm (chart version 7.7.4).
So far, we have activated the toolbox-backup to also use the Minio-S3 objectStorage backend (s3) as the destination for the backups.
We now want to change this to use an Azure StorageAccount as a Backup destination, while all the GitLab data (registry, uploads, artifacts, …) remain on the Minio.
We changed our Helm-Chart to this:
global:
edition: ce
appConfig:
backups:
bucket: gitlab-backup-storage
tmpBucket: gitlab-tmp-storage
gitlab:
toolbox:
backups:
objectStorage:
backend: azure
config:
secret: backup-azure-creds
key: config
cron:
enabled: true
schedule: '0 0 ? * 3,5'
minio:
persistence:
size: 500Gi
While this seems to work at first (the backup .tar file is written to the Azure SA Bucket), looking at the resulting file-size and the Pod logs shows, that the backup script can’t find most of the Minio data.
A full backup is about 6,6 GB.
The Backup resulting from this new Azure SA approach is just about 646 MB in size.
The Pod-Log of a (working) Minio based Backup is this:
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Copying smtp_settings.rb into /srv/gitlab/config
Attempting to run '/bin/bash -c cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility' as a main process
2024-04-10 02:00:37 +0200 -- Dumping database ...
pg_dump: warning: could not find where to insert IF EXISTS in statement "-- *not* dropping schema, since initdb creates it
"
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2024-04-10 02:00:47 +0200 -- Dumping database ... done
2024-04-10 02:00:47 +0200 -- Deleting backup and restore PID file ... done
2024-04-10 02:01:14 +0200 -- Dumping repositories ...
...
2024-04-10 02:01:24 +0200 -- Dumping repositories ... done
2024-04-10 02:01:24 +0200 -- Deleting backup and restore PID file ... done
Dumping registry ...
empty
Dumping uploads ...
done
Dumping artifacts ...
done
Dumping lfs ...
empty
Dumping packages ...
empty
Dumping external_diffs ...
empty
Dumping terraform_state ...
empty
Dumping pages ...
empty
Dumping ci_secure_files ...
empty
Packing up backup tar
[DONE] Backup can be found at s3://gitlab-backups/1712707206_2024_04_10_16.7.4_gitlab_backup.tar
This is the output of an Azure SA based backup:
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Copying smtp_settings.rb into /srv/gitlab/config
Attempting to run '/bin/bash -c backup-utility' as a main process
2024-04-10 16:56:45 +0200 -- Dumping database ...
pg_dump: warning: could not find where to insert IF EXISTS in statement "-- *not* dropping schema, since initdb creates it
"
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2024-04-10 16:56:57 +0200 -- Dumping database ... done
2024-04-10 16:56:57 +0200 -- Deleting backup and restore PID file ... done
2024-04-10 16:57:23 +0200 -- Dumping repositories ...
...
2024-04-10 16:57:34 +0200 -- Dumping repositories ... done
2024-04-10 16:57:34 +0200 -- Deleting backup and restore PID file ... done
Bucket not found: registry. Skipping backup of registry ...
Bucket not found: gitlab-uploads. Skipping backup of uploads ...
Bucket not found: gitlab-artifacts. Skipping backup of artifacts ...
Bucket not found: git-lfs. Skipping backup of lfs ...
Bucket not found: gitlab-packages. Skipping backup of packages ...
Bucket not found: gitlab-mr-diffs. Skipping backup of external_diffs ...
Bucket not found: gitlab-terraform-state. Skipping backup of terraform_state ...
Bucket not found: gitlab-pages. Skipping backup of pages ...
Bucket not found: gitlab-ci-secure-files. Skipping backup of ci_secure_files ...
Packing up backup tar
Job 686756b7-ffaa-4a47-7f6b-2e72a0bc6bb4 has started
Log file is located at: /home/git/.azcopy/686756b7-ffaa-4a47-7f6b-2e72a0bc6bb4.log
Job 686756b7-ffaa-4a47-7f6b-2e72a0bc6bb4 summary
Elapsed Time (Minutes): 0.0673
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Number of Symlink Transfers: 0
Total Number of Transfers: 1
Number of File Transfers Completed: 1
Number of Folder Transfers Completed: 0
Number of File Transfers Failed: 0
Number of Folder Transfers Failed: 0
Number of File Transfers Skipped: 0
Number of Folder Transfers Skipped: 0
TotalBytesTransferred: 677928960
Final Job Status: Completed
[DONE] Backup can be found at /gitlab-backup-storage/1712760976_2024_04_10_16.7.4_gitlab_backup.tar
As you might have noticed, pretty much all of the Buckets can’t be found.
We assume that the backup is looking for these Buckets in the Azure Storage Account as well, while they are living in the Minio S3 storage.
Is this scenario not supported?
What are we doing wrong?