A small update: the reduced number of workers actually works contrary to my previous post. It seems that restarting gitlab using gitlab-ctl isn't the right solution to clean up all memory leaks. Instead the memory allocation is smaller after complete reboot.
Unfortunately after some time or usage the memory allocation is going up quickly up to 973MB again.
Maybe it will be useful to other Rpi fans.