Issues with Git LFS and workspaces: ErrCode=ENHANCE_YOUR_CALM

I use workspaces for virtual dev enviroments for our developers. I also implemented LFS into the projects as there is a lot of bigger files.

On desktop client, I can download all the files including LFS files. but on Coder workspaces, I end up on this error:
Error downloading object: web/wp-admin/images/about-header-about.svg (9ef97ae): Smudge error: Error downloading web/wp-admin/images/about-header-about.svg (9ef97ae66cd7713fe9056e146448dafaf4ba3779e3bc24a5611697d6e31cacb5): batch response: Post "https://gitlab.domain.tld/artevio/internal/project-base-codered.git/info/lfs/objects/batch": http2: server sent GOAWAY and closed the connection; LastStreamID=1, ErrCode=ENHANCE_YOUR_CALM, debug=""

Its great to note all the workspaces are running as docker containers, so they share one public IP. But in this case, my workspace was the only one running. I tried to search for this issue and was hinted to edit the rate limits in Gitlab network settings, so I pushed all the rate limits to nearly unlimited or the highest number I was able to put in, but that did nto solve the issue.

The command used for pulling the repo is basic git clone

I also contacted the Coder team and tried to debug the issue with them, unfortunately, they were unable to help me and send me here to create a topic on this forum. You can read the original thread on their Discord.

Do you have any ideas what can cause this issue and help me resolve it? Thank you!