Build on a runner but publish on an other?

Hello Gitlab community.

I’m struggling with a context : I have a gitlab instance on premises which is seen by 2 runners : one on the build machine and one on the “publish” machine (a web server hosting a yum repository).
On the 1st runner i’m building the rpm package and at the end of the day the rpm package is supposed to land on the 2nd machine in the yum repository… but the 2 machines are not on the same network and they dont see each other.

So i cannot “push” the rpm with scp and trigger the createrepo sith ssh from the build machine.

I’ve seen that gitlab stores artifacts produced by a runner during a build sequence. So I searched if i could spread my build sequence on 2 runners.

In other words : trigger the build of the rpm on the 1st machine then trigger the rest of the process on the 2nd builder which would get the rpm from the gitlab server (= download the artifact just produced by the 1st runner).

So I installed a runner on the 2nd machine, linked it to the project… but i cant find anywhere something that would put me on the right track about what to put in the .gitlab-ci.yml file to make the 2nd runner download the artifacts produced by the 1st …

Did I miss something obvious ?

(Thanks for the time if you’ve gone so far in reading my post… and thank you even more i you can help unblock me there… ^^)

Hi,

you can use runner tags to assign specific CI/CD jobs to tags. Runners registered and configured with these tags will be able to pick up the jobs, ensuring that scenarios with different networks and DMZs work using GitLab. I don’t think you missed something obvious, this is an advanced topic and needs some explanation and examples.

Let’s try to model the rpm build and upload pipeline

There will be 2 jobs, build and upload, assigned to different stages, and using different runner tags.

For debugging purposes, I have added the echo statements that will print the runner ID and its configured tags from predefined CI/CD variables. Will make it easier to verify that the runners are configured properly i.e. when they do not pick up the job.

The build job needs to generate a dummy artifact (using Linux dd command to generate 1MB file chunk here) which is inherited using !reference (extends would override the script section). Both jobs need to know where the artifacts are stored and consumed from - to avoid duplication, the job template .rpm-tmpl provides the artifact attributes. This also answers your question in

what to put in the .gitlab-ci.yml file to make the 2nd runner download the artifacts produced by the 1st

but also needs the runner tags to avoid both runners picking up the jobs at random, resulting in weird behavior.

Last but not least, the ls command lists the rpm files in the directory to verify that the artifacts have been exchanged - and can be processed properly.

stages:
  - build
  - upload 

variables:
  MB_COUNT: 1 

.gen-tmpl:
  script:
    - dd if=/dev/urandom of=$MB_COUNT.rpm bs=1048576 count=$MB_COUNT

.rpm-tmpl:
  artifacts:
    paths: 
      - '*.rpm'

build-rpm:
  extends: .rpm-tmpl
  stage: build
  tags: 
    - build-rpm 
  script:
    - !reference [.gen-tmpl,script]
    - echo "Build rpm job executed on runner ID $CI_RUNNER_ID with tags $CI_RUNNER_TAGS"
    - ls -la *.rpm 

upload-rpm:
  extends: .rpm-tmpl
  stage: upload 
  tags:
    - upload-rpm
  script:
    - echo "Upload rpm job executed on runner ID $CI_RUNNER_ID with tags $CI_RUNNER_TAGS"
    - ls -la *.rpm 

Tip: The Pipeline Editor in the CI/CD menu helps with live linting.

image1808×1441 254 KB

Does it work?

To test my theory, I’ve spun up 2 VMs in Hetzner Cloud (no ads, I just love the fast VM spin-up), installed GitLab runner, and registered them into my GitLab.com SaaS project (did that on the CLI manually, for larger setups I’d recommend Terraform).

Build-rpm runner 1

hcloud server create --image ubuntu-22.04 --type cx11 --name gl-u22-runner-1

ssh -4 root@... 

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash 
sudo apt-get install gitlab-runner

sudo gitlab-runner register --url https://gitlab.com/ --registration-token XXX --tag-list build-rpm --executor shell --non-interactive

systemctl restart gitlab-runner

upload-rpm runner 2

hcloud server create --image ubuntu-22.04 --type cx11 --name gl-u22-runner-2

ssh -4 root@... 

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash 
sudo apt-get install gitlab-runner

sudo gitlab-runner register --url https://gitlab.com/ --registration-token XXX --tag-list upload-rpm --executor shell --non-interactive

systemctl restart gitlab-runner

Verify them in the CI/CD Runner settings.

That also shows the assigned runner tags.

Trigger a CI/CD pipeline

build-rpm - build-rpm (#3544242121) · Jobs · Michael Friedrich / ci-cd-playground · GitLab

image1808×1441 405 KB

upload-rpm - upload-rpm (#3544242122) · Jobs · Michael Friedrich / ci-cd-playground · GitLab

image1808×1441 422 KB

Cleanup

hcloud server delete gl-u22-runner-1

hcloud server delete gl-u22-runner-2

GitLab Project Settings > CI/CD > Runners - remove the registered runners.

Conclusion

Verifies my theory and shows that both runners exchange artifacts, while the jobs are only picked up by assigned runner tags. You should be able to clone/copy the basic structure from my comment. I’ve never done this specific use case before, learned something new myself.

Pipeline and MR are in Create runner tag pipeline (!21) · Merge requests · Michael Friedrich / ci-cd-playground · GitLab (that’s my playground project for wider community questions). Please share whether this example worked for you, and maybe write a blog post too

Cheers,
Michael

Wow !

I did not expect a such long and rich answer…

This reminds me of why I loved the open source world so much (when I accidentally entered it a number of years ago): helping each other, learning from each other, contributing, improving the world a little bit or a lot

Thank you for this unexpected Christmas gift

Of course i will share… and ok for blogging about it

Thank you so much Michael for enlighting my day !

You are welcome I joined open source in 2009 and helping users to learn myself was one of the first things I really enjoyed doing. And I love challenges - your question allowed me to verify my knowledge of GitLab runner tags with a practical example. The scenario with different DMZs reminded me of a customer setup many years ago, when I was doing GitLab consultancy before joining GitLab

Amazing answer. But I would have additional question. Is it possible to run upload-rpm in Shared runner while build-rpm in self-hosted runner?

In my case I need to run Webpack which is superslow on RaspberryPi. But then upload is limited only to specific IP addresses.

I think I know the answer, which is NO. Because sharing artifacts needs some S3 bucket etc. Which I will not have access from custom runner. But maybe there is another solution or I’m wrong.

It should be, provided that only artifacts are exchanged by the configured jobs, running on the same GitLab server.

Artifacts are uploaded to the GitLab server by default, which then other jobs requiring these artifacts are able to download again (you will see a job log in the beginning that says that it will download the artifacts).

Caching is done on the GitLab Runners, which will need more work for access/sync, e.g. with a secured S3 bucket. See the explanation in the docs.

My suggestion is to try the above example, just without the specific runner for upload-rpm tags, and removing the tag upload-rpm on the CI/CD job. That will default to shared runners on GitLab.com SaaS.