Rootless podman-compose ulimit

Hello guys,
I’m trying to deploy gitlab-ee using podman-compose (rootless). That seems to work, at least I have the ui, I can login with the admin and over LDAP. Nevertheless, On a fresh run of podman-compose up -d I have log entries like:

cat: /var/opt/gitlab/gitlab-rails/VERSION: No such file or directory
Starting services...
Configuring GitLab...
/opt/gitlab/embedded/bin/runsvdir-start: line 34: ulimit: max user processes: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system

At the beginning the installation was running forever, basically it was stuck, so I’ve saw this post: "cannot modify limit: Operation not permitted" while using the Docker image (#1350) · Issues · GitLab.org / omnibus-gitlab · GitLab and therefore set the limits. Now it was not freezing anymore, but I still got the file-max, etc. log entries. I even increased them and tried again:

GITLAB_FILEMAX=2000000
[[ $(cat /proc/sys/fs/file-max) -lt ${GITLAB_FILEMAX} ]] && echo $GITLAB_FILEMAX > /proc/sys/fs/file-max

Added the following Hard limits to: /etc/security/limits.conf

MYUSER        hard    nproc           231072
MYUSER        hard    sigpending      65535
MYUSER        hard    nofile          120000

and also to my docker-compose.yaml file

    ...
    ulimits:
      nproc: 231072
      sigpending: 65535
      nofile: 120000
  1. How do I know, what values to use here?
    As I said, it seems that everything is running, all the services are running when I execute: podman exec gitlab_web_1 gitlab-ctl status. I have access to the frontend, I can login via root and via LDAP.
    Any suggestions, or should I simply ignore that?

  2. An other thing which is bothering me, when executing podman container ls -a it says (unhealthy) under the status.
    puma_stderr.log is empty except the first “puma startup …” entry, gitlab-rails/application.log seems also ok.

According (2.) I encountered the culprit here was podman 4.0.2. There is a regression bug where symlinks are not correctly re-labled for SELinux when using :Z on the volume mount. Downgraded to podman 3.0.1.
At the beginning the container simply stopped after some time (5 - 10 hours). I increased RAM for the VM to 10GB and right now, after two days, it is still running (podman rootless). Any recommendations for some optimisations here, or should I simply leave it on 10GB ram? IIRC the docs mentioned way less memory usage, but with 10GB it seems Gitlab is running fine.

Any ideas for (1.)?

I wonder if someone knows what linux capabilities are needed? For example, some containers need capabilities like: --cap-add=NET_ADMIN,MKNOD,NET_RAW. Are there any for Gitlab?

How much ram did you have before this? Normally a minimal spec would be 4cpu and 8gb ram and would probably be enough for your needs. But this can vary, so you just increase the ram where needed, maybe 10gb is your sweet spot for now if you tried 8gb already.

Not sure if you tried this link for podman and gitlab? Deploying Gitlab with Podman | Christofer J. Ekervhén it has some info relating to selinux.

For rootless, eg, podman as a normal user, you can create your systemd script to start it under /home/username/.config/systemd/user, for example /home/iwalker/.config/systemd/user/container-gitlab.service

Before I had 8gb. You say minimal spec 8gb? Where do you got that from? The docs for example say: GitLab installation minimum requirements | GitLab

4Gb for up to 500 Users. That’s why I ask, is there a need for optimisation or anything on wrong on my side, as I’m using 10Gb atm.

Thanks for the link, I saw that one already. I got my systemd file in place (using another approach with podman-compose instead podman run like in the link), but I’m afraid I don’t know what you want to tell me, how your link is related to my questions above (regarding ulimit: max user processes: cannot modify limit, memory usage or any further needed linux capabilities)?

Yeah the docs should really be updated, because it’s practically impossible to run it on 4gb without disabling a load of stuff. It is possible to run it on a Raspberry PI4 with 4gb, have done it at home, but you need to follow that guide then in terms of what to disable etc, or reduce configuration wise.

I’ve recommended in a lot of posts here to run with 8gb purely by the experience I’ve had with it. But this can vary from one installation to the next so 10gb seems to work for you.

I don’t use podman, I’ve only briefly experimented with it, was hoping that link I provided may help with some of your issues. From searching I haven’t been able to find anything else related to it. But as the Gitlab docs only mention running it with docker, it could be that podman isn’t quite like docker is, and hence why a few issues exist.

I know you’re running it as rootless, maybe try on a test server with podman doing everything as root. Does it work any better? The link I provided was root-run podman, so perhaps whilst podman can be ran rootless, maybe something specific with Gitlab requires it to be run as root. That would at least be a good debug step to see if it’s related to lack of permissions as a standard user, compared to root. Or whether it’s a generic issue attempting to run Gitlab under podman. Although, if people have already done it as root and it runs fine, then that would say it isn’t a generic Gitlab/podman issue.

Oh ok I see, so no specific reason, anyway thanks for the effort.
Actually I don’t have any issues right now, I just wondered if the ulimit and /proc/sys/fs/file-max messages are errors, warnings, anything I can do about it (except runnning as root). I’m sure, it will go away if executed as root, because as root of course it can set this values. But I would like to know, someone must have implemented that feature which decides to what value the max user processes will be set, I can simply set it before and Gitlab should be satisfied. Now I simply picked a pretty high value, but don’t know if it is still too low, maybe even too high, but the implemented check in gitlab is somehow stupid, that it maybe just checks: “can I change the value, yes/ no” but totally ignores the fact that the value maybe is already configured correctly.

As said, right now, rootless podman using systemd and docker-compose.yaml seems to run fine. At least, the container doesn’t stop, I can use the UI, ldap is working. I will import a few projects and test it, but I don’t expect any issues here anymore. The logs also look okay except the above mentioned entries.

This might explain some stuff: "cannot modify limit: Operation not permitted" while using the Docker image (#1350) · Issues · GitLab.org / omnibus-gitlab · GitLab related to those lines, so looks like nothing to worry about based on that issue. But some people in that issue have used a --privileged parameter with docker run, so whether you can add that with podman run - worth a try maybe?

Link to that particular message: "cannot modify limit: Operation not permitted" while using the Docker image (#1350) · Issues · GitLab.org / omnibus-gitlab · GitLab

And a post to someone having the exact messages as you: "cannot modify limit: Operation not permitted" while using the Docker image (#1350) · Issues · GitLab.org / omnibus-gitlab · GitLab

Whilst I realise that is docker, it seems the errors are known of, now whether you can get around that or not to quieten it down with podman and --privileged I don’t know, but try anyway and let us know.

That’s the link I’ve posted in my initial post above :smile: so yeah, saw that of course. Maybe if I find some minutes, I try to run on a separate VM with sudo podman.

Ah :laughing: did the --privileged help with podman? As normal user or rather not?

Either way it seems the issue can be ignored with the errors warnings due to lack of permissions.

haven’t run it with the privileged flag yet. I don’t think I will do that, that’s why I asked above if there are any linux capabilities which should be used, so I can explicitly activate only the needed ones. Privileged activates all of them and I think it opens even more.
There is a good post about that, if someone’s interested:

I think the ulimit check is not well implemented (as it seems to run fine without any, looks like there are no extra linux capabilities needed) and the docs could be improved to specify what limits should be set beforehand.

1 Like