I cannot push images to GitLab Registry. The build job log simply says received unexpected HTTP status: 500 Internal Server Error. Registry logs show a strange error about a missing S3 bucket, but does not state which bucket it’s trying to access. I have created all buckets listed in the GitLab config docs.
I’m running the vanilla helm chart using microk8s with minor configuration alterations (setup.sh file shown below).
CI FILE:
build-docker-image:
image: docker:20-dind
tags: [docker]
stage: build
services:
- docker:20-dind
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $CI_REGISTRY/qrd/datalink/datalink:latest .
- docker push $CI_REGISTRY/qrd/datalink/datalink:latest
BUILD LOG:
Running with gitlab-runner 15.11.0 (436955cb)
on qrd-00 bmjbw9xG, system ID: r_NJIv1OqNbmi0
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-system
Using Kubernetes executor with image docker:20-dind ...
Using attach strategy to execute scripts...
Preparing environment
<unrelated job stuff>
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
Login Succeeded
$ docker build -t $CI_REGISTRY/qrd/datalink/datalink:latest .
Successfully built 19106081c3a7
Successfully tagged registry.website.io/qrd/datalink/datalink:latest
$ docker push $CI_REGISTRY/qrd/datalink/datalink:latest
The push refers to repository [registry.website.io/qrd/datalink/datalink]
272ae959d2bd: Preparing
09688275b69b: Preparing
11dbb9b06a6f: Preparing
3de73518c5ea: Preparing
68062a646169: Preparing
41c16281e12d: Preparing
ed7b0ef3bf5b: Preparing
41c16281e12d: Waiting
ed7b0ef3bf5b: Waiting
11dbb9b06a6f: Retrying in 5 seconds
<Lots of retrying>
09688275b69b: Retrying in 1 second
3de73518c5ea: Retrying in 1 second
68062a646169: Retrying in 1 second
41c16281e12d: Retrying in 5 seconds
received unexpected HTTP status: 500 Internal Server Error
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1
REGISTRY LOG:
{"CF-RAY":"7c6c5da1dc168c0c-EWR","auth_user_name":"fidcuevas","auth_user_type":"build","correlation_id":"0ae7ee3cc9be9bc12c3645e573b8646d","go_version":"go1.18.7","level":"info","msg":"authorized request","root_repo":"qrd","time":"2023-05-13T16:44:22.709Z","vars_name":"qrd/datalink/datalink","version":"v3.71.0-gitlab"}
{"CF-RAY":"7c6c5da1dc168c0c-EWR","auth_user_name":"fidcuevas","auth_user_type":"build","code":"UNKNOWN","correlation_id":"0ae7ee3cc9be9bc12c3645e573b8646d","detail":"s3aws: NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: 175EC1ADC1ED2741, host id: ","error":"unknown: unknown error","go_version":"go1.18.7","level":"error","msg":"unknown error","root_repo":"qrd","time":"2023-05-13T16:44:22.710Z","vars_name":"qrd/datalink/datalink","version":"v3.71.0-gitlab"}
{"CF-RAY":"7c6c5da1dc168c0c-EWR","content_type":"application/json","correlation_id":"0ae7ee3cc9be9bc12c3645e573b8646d","duration_ms":1,"host":"registry.website.io","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.1.153.146:52004","remote_ip":"10.1.153.146","status":500,"system":"http","time":"2023-05-13T16:44:22.710Z","ttfb_ms":0,"uri":"/v2/qrd/datalink/datalink/blobs/uploads/","user_agent":"docker/20.10.24 go/go1.19.7 git-commit/5d6db84 kernel/5.19.0-41-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.24 \\(linux\\))","written_bytes":123}
DEPLOYMENT CONFIG (K8s helm chart):
#!/bin/bash
microk8s.helm3 -n gitlab-system upgrade --install gitlab gitlab/gitlab \
--timeout 600s \
--set global.hosts.domain=website.io \
--set global.hosts.externalIP=192.168.1.113 \
--set global.hosts.ssh=gitlab-ssh.website.io \
--set certmanager-issuer.email=fidcuevas@website.io \
--set gitlab.webservice.ingress.tls.secretName=gitlab-tls \
--set global.kas.enabled=true
- Not sure how to troubleshoot.
- Have logged into Minio and ensured all default buckets exist and have the appropriate roles.