Hello,
I hope everyone is fine.
Here’s my problem.
Problem to solve
GitLab’s backup-utility fails to connect to MinIO and find the default buckets for data backup. When executing the backup-utility command, I receive errors indicating that buckets are not found and the connection is refused.
Steps to reproduce
- Deploy GitLab using Helm with the provided configuration
- Execute the command:
kubectl exec gitlab-toolbox-d3e3gccdg-czxbg -it -n prod -- backup-utility
Configuration
resource "kubernetes_secret" "gitlab-keycloak" {
metadata {
name = "gitlab-keycloak"
namespace = data.kubernetes_namespace.prod.metadata[0].name
}
data = {
"gitlab-keycloak" = file("prod_secrets/gitlab-keycloak")
}
type = "Opaque"
}
resource "helm_release" "gitlab" {
name = "gitlab"
namespace = data.kubernetes_namespace.prod.metadata[0].name
repository = "https://charts.gitlab.io/"
chart = "gitlab"
version = var.gitlab_version
depends_on = [data.kubernetes_namespace.prod, scaleway_rdb_database.gitlab_db]
# set {
# name = "global.appConfig.omniauth.enabled"
# value = "true"
# }
# set {
# name = "global.appConfig.omniauth.autoSignInWithProvider"
# value = "openid_connect"
# }
# set {
# name = "global.appConfig.omniauth.allowSingleSignOn"
# value = "\\[oauth2_generic\\]"
# }
# set {
# name = "global.appConfig.omniauth.blockAutoCreatedUsers"
# value = "false"
# }
# set {
# name = "global.appConfig.omniauth.providers[0].secret"
# value = "gitlab-keycloak"
# }
# set {
# name = "gitlab.kas.ingress.tls.secretName"
# value = "gitlab-kas-tls"
# }
# set {
# name = "minio.ingress.tls.secretName"
# value = "gitlab-minio-tls"
# }
# set {
# name = "registry.ingress.tls.secretName"
# value = "gitlab-registry-tls"
# }
# set {
# name = "gitlab.webservice.ingress.tls.secretName"
# value = "gitlab-gitlab-tls"
# }
set {
name = "global.initialRootPassword.secret"
value = "gitlab-secrets"
}
set {
name = "global.initialRootPassword.key"
value = "gitlab-root-password"
}
set {
name = "postgresql.install"
value = "false"
}
set {
name = "nginx-ingress.enabled"
value = "false"
}
set {
name = "global.ingress.configureCertmanager"
value = "false"
}
set {
name = "certmanager.install"
value = "false"
}
set {
name = "global.edition"
value = "ce"
}
set {
name = "global.hosts.domain"
value = var.domain_name
}
set {
name = "global.hosts.gitlab.name"
value = var.gitlab_url
}
set {
name = "global.minio.enabled"
value = "true"
}
set {
name = "gloabl.minio.persistence.size"
value = "10Gi"
}
# set_list {
# name = "global.minio.defaultBuckets"
# value = ["gitlab-lfs", "gitlab-artifacts", "gitlab-uploads", "gitlab-packages", "gitlab-registry", "gitlab-backup"]
# }
set {
name = "global.hosts.minio.name"
value = var.gitlab_minio_url
}
set {
name = "global.hosts.minio.https"
value = "false"
}
set {
name = "global.hosts.https"
value = "false"
}
set {
name = "global.hosts.gitlab.https"
value = "false"
}
set {
name = "global.psql.host"
value = var.db_gitlab_endpoint
}
set {
name = "global.psql.host"
value = var.db_gitlab_endpoint
}
set {
name = "global.psql.port"
value = "5432"
}
set {
name = "global.psql.database"
value = var.db_name_gitlab
}
set {
name = "global.psql.username"
value = var.db_gitlab_user
}
set {
name = "global.psql.password.useSecret"
value = "true"
}
set {
name = "global.psql.password.secret"
value = "gitlab-secrets"
}
set {
name = "global.psql.password.key"
value = "db-gitlab-pass"
}
set {
name = "gitlab-runner.install"
value = "true"
}
}
resource "kubernetes_ingress_v1" "gitlab" {
metadata {
name = "gitlab"
namespace = data.kubernetes_namespace.prod.metadata[0].name
annotations = {
"cert-manager.io/cluster-issuer" = "scaleway"
"kubernetes.io/tls-acme" = "true"
}
}
spec {
ingress_class_name = "nginx"
tls {
hosts = var.gitlab_hosts
secret_name = var.gitlab_cert
}
rule {
host = var.gitlab_url
http {
path {
path = "/"
path_type = "Prefix"
backend {
service {
name = "gitlab-webservice-default"
port {
number = 8181
}
}
}
}
}
}
tls {
hosts = var.gitlab_minio_hosts
secret_name = var.gitlab_minio_cert
}
rule {
host = var.gitlab_minio_url
http {
path {
path = "/"
path_type = "Prefix"
backend {
service {
name = "gitlab-minio"
port {
number = 9000
}
}
}
}
}
}
}
}
Logs
kubectl exec gitlab-toolbox-d3e3gccdg-czxbg -it -n prod -- backup-utility
Defaulted container "toolbox" out of: toolbox, certificates (init), configure (init)
2024-11-23 14:54:47 UTC -- Dumping database ...
2024-11-23 14:54:47 UTC -- Dumping PostgreSQL database gitlab-db ...
2024-11-23 14:54:54 UTC -- [DONE]
2024-11-23 14:54:54 UTC -- Dumping database ... done
2024-11-23 14:54:54 UTC -- Deleting backup and restore PID file at [/srv/gitlab/tmp/backup_restore.pid] ... done
2024-11-23 14:55:17 UTC -- Dumping repositories ...
2024-11-23 14:55:17 UTC -- Dumping repositories ... done
2024-11-23 14:55:17 UTC -- Deleting backup and restore PID file at [/srv/gitlab/tmp/backup_restore.pid] ... done
Bucket not found: registry. Skipping backup of registry ...
Bucket not found: gitlab-uploads. Skipping backup of uploads ...
Bucket not found: gitlab-artifacts. Skipping backup of artifacts ...
Bucket not found: git-lfs. Skipping backup of lfs ...
Bucket not found: gitlab-packages. Skipping backup of packages ...
Bucket not found: gitlab-mr-diffs. Skipping backup of external_diffs ...
Bucket not found: gitlab-terraform-state. Skipping backup of terraform_state ...
Bucket not found: gitlab-pages. Skipping backup of pages ...
Bucket not found: gitlab-ci-secure-files. Skipping backup of ci_secure_files ...
Packing up backup tar
WARNING: Retrying failed request: /1732373660_2024_11_23_17.3.1_gitlab_backup.tar ([Errno -2] Name or service not known)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /1732373660_2024_11_23_17.3.1_gitlab_backup.tar ([Errno -2] Name or service not known)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /1732373660_2024_11_23_17.3.1_gitlab_backup.tar ([Errno -2] Name or service not known)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /1732373660_2024_11_23_17.3.1_gitlab_backup.tar ([Errno -2] Name or service not known)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /1732373660_2024_11_23_17.3.1_gitlab_backup.tar ([Errno -2] Name or service not known)
WARNING: Waiting 15 sec...
Versions
- Self-managed
Versions:
- GitLab: 17.3.1
- GitLab Helm Chart: 7
Helpful resources
- I’ve checked the documentation on configuring MinIO with GitLab
- I’ve searched for similar issues in the GitLab forum
- I’ve examined the logs for MinIO-specific error
- Terraform Registry
- GitLab Helm chart | GitLab
- Kubernetes Documentation
- backup-utility fails to upload the backup when using fake domain on minikube (#1906) · Issues · GitLab.org / charts / GitLab Chart · GitLab
The issue appears to be related to MinIO configuration or connectivity. The default buckets are not created or accessible, preventing the backup from functioning correctly.
Thank you for your help and have a nice day.
Regards,
R