Where should you install gitlab-runner?

For example, you have a few servers with different environments like test, and prod. And you want to apply GitLab CI/CD, should you install gitlab-runner on each server or install it on a separate machine. Which one is better or are there any so-called best practices. I’m new to this concept and a bit confused since GitLab official docs didn’t mention this point. Any guidance would be greatly appreciated.

For testing, I installed it on each server (vm1-test, vm2-prod) and run the pipeline there accordingly when the pipeline is triggered. It works really well but not sure if there is any security concern or any downsides of doing so.

I think this depends very much on your own (network) architecture and security policies. The only advice (from GitLab, because of performance) is to put Runner on a dedicated VM (separately from GitLab, and I’d say separately from any application environment). Don’t forget that you can use same runner to run jobs for different environments :slight_smile: (you don’t need to have 1-per-environment, unless perhaps if you are using just a shell runner - which I wouldn’t necessarily recommend for most cases)

Now, where exactly you put it, depends on your environments, what exactly your “deploy” job needs to do, if Runner can reach those environments and execute task, etc etc.

In any case - GitLab and Runners are communicating over GitLab’s HTTP API (Runner → GitLab). So, if you are using proper SSL certificates, I believe that should be fine, no matter where you put your Runners. And then again, from Runners to anything else - just make sure you’re following general security best practices (e.g. when using SSH - use ssh key pair and store private key securely, when communicating with Docker Registries - don’t turn off certificate check, etc etc).

If you have some specific questions / concerns, feel free to share your architecture and I’m sure community will be happy to help.

Best regards!

2 Likes

I use shell runner due to some reasons that I can’t use docker. So you mean it’s impossible to just use one runner if I’m using shell runner. What other executors would you recommend? Currently my architecture is I have 2 server dev and prod. and my pipeline is really simple when there are any changes pushed to dev then the runner on dev server will do its job when there are any changes pushed to prod then the runner on prod will do its job respectively. I’m not sure if it’s a good approach.

Best regards,

Well, shell runner executes task on the machine where it’s installed - that’s why it’s very limiting.

What I normally use is docker executor, even for executing tasks on different machines. For example, I install and configure ssh, copy anything I need to target machine, ssh into target machine and execute a one-liner to deploy - all with docker executor. And this way I can use same runner(s) for all internal environments. (docker executor makes runners quite flexible, at least I made mine in that way to optimize usage for different project needs)

Example (I also took this long time ago from somewhere, so use with a pinch of salt :wink: ):

deploy to dev: 
  image: alpine:latest
  cache: []
  dependencies: []
  before_script:
    - apk add --no-cache openssh-client ca-certificates
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    # if you want more security, rather configure your HostKeys then turning this check off
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - eval $(ssh-agent -s)
    - ssh-add <(echo "$GITLAB_PRIVATE_KEY")
  script:
    - scp -r conf/* $USER_DEVELOP@$HOST_DEVELOP:~/my-project/
    - ssh $USER_DEVELOP@$HOST_DEVELOP "cd my-project && docker login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD && docker compose pull && docker compose up -d"
  only:
    - develop

Again, this is just for inspiration. This way you should be able to copy any smaller scripts or binaries to target and boot things up. As I don’t know your exact use case, cannot give you anything more specific and it could be that you cannot do it this way.

I also have cases where it’s a bit more difficult, where I need to use Windows Server to execute builds (because I cannot install software in docker), etc, so using shell runner is just easier and more convenient in that case.

P.S. I’m not saying your approach is wrong. If you follow security best practices, and it works - why not? :wink:
I avoid shell runners for different reasons - you need to do manual cleanup and install everything you need manually there (or ok maybe with some other tools like Terraform), but I personally just prefer docker images when possible, as it gives me way more flexibility and separates “infrastructure” from “environment needs”.

3 Likes

Thank you so much. That really helps a lot.

2 Likes