CI/CD Pipeline Job artifact not downloadable

Hi, I have a strange problem with our self-hosted gitlab instance.

In our pipeline we have job where we build an angular app and provide the build as artifacts.
The job is named build-app-job. On our server we have a shell script that downloads the artifacts
and replace the files. We want to download the latest build and therefore make the following call:

curl -L -k --header "PRIVATE-TOKEN: GITLAB-TOKEN" "https://gitlab.instance.domain/api/v4/projects/999/jobs/artifacts/dev/download?job=build-app-job" --output artifact.zip

This worked fine for a time. Now we have the issue that we always get a 404-error while downloading the artifacts from the server where the app should run. What is even more strange, if we make the same curl call locally it works perfectly and after calling it locally once, it also works again on the server. If we try to download it directly with the job ID we get the correct zip file both, locally and on the server of the app.

We have already upgraded gitlab to the version 16.7.3-ee.

Has anybody faced a similar problem or has a clue what a possible reason for this could be? I am not sure if it is a bug or issue of gitlab?

Thank you in advance!

Hi, have you looked in Gitlab logs if there is something that might help to determine the cause?

@balonik Thanks for your answer.
There was nothing into this direction. We could not solve this problem but found a walkaround.

We are now pushing the artifacts to the package registry and download them to the production
server from there.
Was in the end the cleanest and easiest way