I’ve been tasked with setting up a proof-of-concept on our development self-hosted GitLab service to show how CI/CD can be delivered as flexibly as possible. We are using AWS to host our infrastructure.
So, the things I’m looking to achieve include:
- Ability to see how much each job has cost so that it can be charged back to the owner of the project that initiated the job.
- Ability to support directives that provide guidance/specification of processor type (Intel, Arm), processor size, memory size.
- Each job must run in its own environment. Privacy/security is paramount here.
- Support caching.
I started off looking at docker+machine with EC2 autoscaling, per Autoscaling GitLab Runner on AWS EC2 | GitLab. However, it doesn’t seem to support the second bullet point easily - the only workaround/solution I could think of would be to have multiple runner, each configured for different CPUs/memory and then use tags to allow a job to specify which runner to use.
Given that GitLab are looking at how to replace docker+machine (since Docker deprecated it) and that the Kubernetes support does support the second bullet point, I’ve now started looking at using AWS EKS. How to create a Kubernetes cluster on Amazon EKS in GitLab | GitLab describes how this can be done and The Kubernetes executor for GitLab Runner | GitLab documents the VERY many parameters available, including processor type, cpu size, memory size.
However, with the k8s approach, I’m concerned about:
-
how to figure out the billing side of things. With autoscaling EC2, GitLab names each EC2 instance with a process ID and my intention was to read the GitLab logs to connect that ID back to the job that initiated it. I can’t (readily) find any mention of how the various bits in a k8s cluster might be tagged or references so I’m concerned that I won’t be able to figure out the costs. It may be that I’ll still be able to figure something out from the logs; I won’t know for certain until I have AWS EKS up and running for GitLab, which is something I’m currently working on. Another option might be
pod_labelswhere the description says “A set of labels to be added to each build pod created by the runner. The value of these can include environment variables for expansion.” … although I need to dig further into what I might be able to do there. -
privacy/security. The Helm chart suggests that a new pod is created for each job to be run. I’m presuming that this means that the sequence is create pod, run job in that pod, destroy pod? In other words, a job runs in its own environment with no risk of data being available to another job and that all resources running that job are destroyed afterards?
-
caching. For autoscaling EC2, there is support built into GitLab to use a S3 bucket to copy files to and from that buck as a cache. For k8s, the documentation says:
When the cache is used with the Kubernetes executor, a specific volume called
/cacheis mounted on the pod. The cache volume can be configured in theconfig.tomlfile by using thecache_dirsetting.
Kubernetes cache support (#1906) · Issues · GitLab.org / gitlab-runner · GitLab suggests that the S3 cache mechanism is supported but that isn’t clear from the documentation.
It is essential that any cache implemented only exposes the cached files used by that project and nobody else’s project.
Thank you for any insight or help provided 