Yesterday I installed GitLab 14.9.0-ee to a brand new Kubernetes Cluster in order to migrate my old Docker Containerized Gitlab 14.9.0-ce to it. The Migration itself worked as I would have expected.
But now that everything seems to be working I still have one issue. User Uploads (Profile Avatars and Group Avatars) do not work.
What I can see is that when I upload a valid JPG file as Profile Picture, GitLab transforms it into a valid PNG. But then before uploading it to S3 it adds a hex number and “chunk-signature” with hash to the beginning of the file.
Example copied from a HEX viewer:
This means that when the Browser gets to download the image (with download_proxy enabled) it is no longer a valid PNG file.
I have tested and when I manually upload a valid PNG into the place the invalid PNG was uploaded, the manually saved PNG works correctly.
From looking into the S3 bucket all images are there, just invalid.
The S3 Configuration must be correct because Gitlab can upload the images to it, and also download it. Its just that the images are invalid.
GitLab 14.9.0-ee migrated from 14.9.0-ce
Kubernetes 1.23.4 in Scaleway Kubernetes Kapsule
S3 Storage Buckets in Scaleway S3
I have just seen that there are security updates available for 14.9.x and will be installing those now.
So. I have just tried and unlinked all avatars by just iterating over projects, users and groups in Rails Console, and then changed the target S3 bucket and upload an image.
Sadly the same issue occurres. After I upload it to Gitlab, it is loaded into the HTML DOM correctly and then uploaded into S3 with the chunk hash.
I have no idea why this is broken.
Here is my Helm-Values Bucket configuration for reference
And here the S3 Secrets in censored form
Example of the issue:
Image uploaded to Gitlab:
Image downloaded from S3:
Okay, so upgrading to the latest version (14.10.2) didnt fix the issue. Also moving back to Community Edition did nothing.
Today I thankfully found a configuration setting in the GitLab object-storage documentation that streaming uploads to S3 can be disabled, so I tried it.
AND IT WORKS NOW!
I have put this to my
gitlab-rails-storage connection block, updated it and recreated the webservice pods.
I am not sure if this now means that the max size of file GitLab can transfare to Scaleway S3 is 5GB, or if it now uses “normal” multi-part uploads (which allows for up to 5TB per object and is limited to max. 1000 parts with >= 5MB and <= 5GB).
Sadly I couldnt find anything about that in that documentation.
If anyone knows how GitLab will behave with this disabled and > 5GB files, please tell me