How many parallel runners are recommended?

We have set up a GitLab runner on our Ubuntu Azure VM and I am wondering how many parallel jobs I should configure (in the /etc/gitlab-runner/config.toml file, runners/limit). Our machine has 4 cores and 16 GB RAM. Usually, fast CI builds are our most urgent requirement. So how many parallel jobs would you recommend us before they start blocking each other? Or does this totally depend on the kind of build steps that we have got in our CI?

Thanks in advance!