Gitlab runner fails when running too many job same time

I’ve set some scheduled jobs on gitlab. It is running some parallel jobs every minute and reaches about 500 simultaneous process . After some point runner throws below error and fails.

Running with gitlab-runner 15.5.0 (0d4137b8)
  on XXXX y4s23Bpz
Preparing the "shell" executor
Using Shell executor...
Preparing environment
ERROR: Job failed (system failure): prepare environment: failed to start process: fork/exec /usr/bin/su: too many open files. Check for more information

I’ve increased open files limit but didn’t resolve problem. I couldn’t figure out which limit it hits.

~$ ulimit -a 
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515311
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 999999
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 515311
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited