Cannot dynamically map a device on gitlab-runner

Hi,

First of all, sorry for the long topic and sorry if the category is not the right one. None seemed to fit.

I’m working in a project involving a HW of our own design connected through USB for test purposes. I use GitLab pipelines for the build, test and deploy of the Firmware running on this HW. The runners are implemented as docker containers running in gitlab-runners.

I use the USB interface as a serial terminal (mapped to /dev/ttyACM0) to send test commands and as a usb mass storage to load the test files on the HW’s memory.
I have a gitlab job use the USB terminal, then switch to the USB mass storage, load some test files, then switch back to USB terminal and perform some tests.

There’s either the USB terminal or the USB mass storage available at one point in time. The terminal is the default one, so I configure the runner to map /dev/ttyACM0 (no issues there), but as there’s no USB mass storage, I cannot map it in the runner’s config (obviously).

So I need to map it and dynamically mount the USB volume. To do that I did the following:

1. I set a udev rule in the runner’s host to a fixed path reading the device’s vendor ID and product ID (symlinked to a name in the like of “/dev/disk/by-dname/my-favourite-usb-drive”)
2. I set an fstab rule to be able to mount the symlinked volume without super user permissions (using the “users” option) to a fixed mount point.
3. Configure the runner to run the docker container as privileged and map the whole /dev/ directory using the runner config “devices” field (I know, this is super unsafe, but this is only for internal use and the idea is to get it running, then restrict the permissions to a minimum)

I tested this approach by executing a docker container in the same host where the runner is and then calling the test script in the same way I did in the pipeline’s job. The result was a success, the test script was working, the mass storage was being mapped, I could mount it, copy my test files, everything was great.

However, once I executed the runner and triggered the job, I saw the script failing. It could not mount the volume because the symlink I created on the udev rule didn’t exist.
I tried listing the contents of /dev/, and it wasn’t only my symlink that was missing, but the whole /dev/disk directory was missing too. I tried moving the symlink to “/dev/my-test-dir”, but nothing changed.

I tried checking the documentation, stack overflow, docker forums… nothing helped (I tried adding -net=host, seemed to help someone somewhere, but id didn’t help me)

I purposely left out several details because the might not be relevant and there’s enough stuff in this topic already, but I’ll answer questions the best I can.

All this was to ask what is going on here?
Why does this work on a manually executed docker container but it doesn’t in a gitlab-runner executed container? Something must be happening on the runner’s side to explain this right?
Maybe something to do with udev?

Thanks in advance.

Maybe this might help? [Docker Runner] Allow access to device (--device docker option) (#240) · Issues · GitLab.org / gitlab-runner · GitLab

That actually seems to be solved in: Add support for cap_add, cap_drop, and devices in Docker executor (!91) · Merge requests · GitLab.org / gitlab-runner · GitLab

Which would mean adding to the gitlab-runner /etc/gitlab/config.toml something like this:

concurrent = 1 

[[runners]]
  name = "devel-runner"
  url = "http://gitlab.example.com/ci/"
  token = "xxxxxxxx"
  limit = 1 
  executor = "docker"
  [runners.docker]
    privileged = false
    devices = ["/dev/kvm"]

mainly the devices = part. You could put the particular device path you need there, I guess?

If you need multiple devices, it seems from the first link an example was given like this:

    devices = ["/dev/nvidia1:/dev/nvida1" "/dev/nvidia0:/dev/nvida0", "/dev/nvidiactl:/dev/nvidiactl", "/dev/nvidiactl:/dev/nvidiactl" ]

Thanks for your answer, I am already mapping the USB terminal (/dev/ttyACM0) using the devices field and it works, the problem is with the mass storage device. I tried adding the path of the symlink of the udev rule. I even tried mapping the whole /dev/ directory, like this:

devices = ["/dev/usb-msc", "/dev/", "/dev/ttyACM0"]

But it’s not working…

Again, I thik this has something to do with the gitlab-runner (the config? the way I run it?), because this same approach works when manually running the docker container.

One more idea, Advanced configuration | GitLab

device_cgroup_rules

does the standard docker have any device cgroup rules that perhaps the gitlab runner is not using? Apparently this appeared in Docker 1.28 and later. Perhaps a cgroup rule is blocking access perhaps?

Is there anything else shown in the gitlab runner logs that might hint at why there is a problem accessing the device?

It’s all I can think of right now, sorry if I haven’t been much help.

No worries man, thanks for your help =)

I haven’t checked the cgroup rules, but I didn’t specifically set them… Maybe that’s the issue, I’ll check it out.

Thanks

Hi Guys ,
I am facing same issue here , but my setup is not docker . It is just windows based system and trying to run the python script which need t open the USB serial and try to flash the files . And this python file I am triggering from GitLab .
Any help for this . I used same which you mentioned above but not suscces