I got a pipeline up and working.
My supervisor gave me access to a VM, on which I installed gitlab-runner and all needed dependencies (I’m using the shell mode).
The build task runs without problems, but I also want to create some tests.
The Problem is, that the program input are files, which contents are secret and therefore are not included in the repository.
I stored the files on the VM (e.g. in /path/to/file.txt). But when I pass the (absolute) path to the file in the test job, in the script section:
then I get the (it’s a c++ program) (test-job)pipeline error output:
terminate called after throwing an instance of 'std::runtime_error'
what(): ERROR: File /path/to/file.txt not found
bash: line 145: 52372 Aborted ./build/my_program $FILE
How can I pass local files, which are stored on the machine and not in the repository to a job?
Or is there even a best practice on passing local/ secret files to a test?
some questions that maybe help you with the file not found error:
Who is the owner of /path/to/file.txt?
What are the permissions of /path/to/file.txt?
Which user is owner of the gitlab-runner process?
Is the owner of the gitlab-runner process able to read the file?
A shell runner runs the command in the context of the user that started the gitlab-runner process. If the user that is running the jobs from the pipeline (the gitlab-runner process owner) is not able to read the file from the local server that would totally explain the output from your pipeline.
Best practice for secrets depends on if you are hosting GitLab and/or the runner yourself, what you are able to use or setup yourself and of course how paranoid you are. There is no easy answer for that
Some examples from our GitLab environment here. All have their own pros and cons:
Secrets stored in AWS Secrets Manager loaded and decrypted during job by aws-cli commands
Secrets stored in Encrypted AWS SSM Parameters and decrypted during job by aws-cli commands
Secrets stored in Hashicorp Vault and loaded during job by cli commands
Secrets stored in Hashicorp Vault and loaded by runner into pipeline variables according to the pipeline definition (see External Secrets)
Secrets stored in SOPS encrypted file within the repository and unencrypted by runner which has the private key avaliable
I first tried to modify the permissions and after that didn’t work, then I tried to set the owner to gitlab-runner. But both solutions didn’t work.
Who is the owner of /path/to/file.txt?
What are the permissions of /path/to/file.txt?
[…]
Is the owner of the gitlab-runner process able to read the file?
I set the permissions to freely accessible, because I wanted a working solution and then strip down permissions to the bare minimum. After that I (still resulted in a /path/to/file.txt not found error) I changed the owner and the group to be gitlab-runner. But this still resulted in a /path/to/file.txt not found error.
Running ls -l /path/to/file.txt returns: -rwxrwxrwx 1 gitlab-runner gitlab-runner 1246 Oct 25 08:34 /path/to/file.txt.
