CI push to Gitlab Registry hangs, job times out

I’ve got a brand new installation of Gitlab 1.16.1 in Kubernetes, with an external redis cluster, patroni postgresql cluster, and a single gitaly instance. As it stands, now, all is working properly.

Except…for some reason when I attempt to push images to the Gitlab Registry using CI, the push seems to happen, and then the job eventually times out. I’ve reproduced this behavior with a Docker in Docker setup, with Kaniko, and with Buildah, so I don’t believe it is the job itself but something in the communications to the registry. I don’t see any errors in the minio or the registry logs that would help me to understand what is happening.

I am able to push and pull from the repository from a container inside of my cluster as well as from my local workstation. When I look at the minio bucket, I can see the images in the bucket.

When a CI job attempts to push an image, I can see the directory structure created in the bucket, but I do not see any images pushed.

I don’t believe that this is related to another issue I read about (the Gitlab ruby code returns before the repository has been created) because the same hang happens in a CI push regardless of whether the repository exists or not.

I have some debug output that I created with buildah when it was trying to push a newly-built image. I’ll paste it below as it’s too large for this post.

How can I trace the path that the requests are taking from the runner pods to the registry, and identify what might be causing this? Any help would be very much appreciated!

EDIT: I just created a pastebin for the output here

And here’s the gitlab-ci.yml file:

build:
  stage: build
  image: quay.io/buildah/stable
  variables:
    STORAGE_DRIVER: vfs
    BUILDAH_FORMAT: docker
    FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/test"
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
  script:
    - buildah images
    - buildah build -t "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}"
    - buildah images
    - buildah push --log-level debug --authfile /run/containers/0/auth.json "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}"

deploy:
  rules:
    - if: '$CI_COMMIT_REF_NAME == "main"'
  stage: deploy
  image: 
    name: k8s.gcr.io/kustomize/kustomize:v5.0.1
    entrypoint: ["sh"]
  script:
    - cd k8s/
    - kustomize edit set image app="${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA}"
    - git config user.name "CI Pipeline"
    - git config user.email "gitlab@cureau.dev"
    - git add .
    - git commit -m "set deploy tag to ${CI_COMMIT_SHA} [skip ci]"
    - git push HEAD:$CI_COMMIT_REF_NAME 

Well, I found my issue. The upload buffer wasn’t large enough to accept the built image. Needed to twiddle the envoy filter to get enough buffer to do the job.