Docker installation: numerous failures

How to Use GitLab Self-managed
1

Problem to solve

TL;DR: I just really would like to run the official gitlab/gitlab-ce Docker container. It appears to be completely unserviceable.

I’m trying to create a fresh installation using the documentation. I’ve been fighting numerous errors and failures to start and, after 4 hours, I have still not been able to bring a container up successfully. It seems like either the Docker Installation instructions are outdated or the Docker image is severely broken. I started out trying the gitlab-ce:latest image, then I realized that perhaps that might be why this experience has been so terrible so I downgraded to the latest official release 18.11.3 and still had issues. I imagine I’m doing something wrong, but … surely this build can’t be this broken? Please help!

/assets/update-permissions is effectively useless for fixing the initial failed build environment

The script only attempts to fix permissions for directories that already exist. Given that the default container image created doesn’t seem to have (m)any of the required directories, this script skips over just about everything I was hoping it would fix.

/assets/init-container produces errors even under expected circumstances

This caused me to waste time tracking down errors that weren’t actually errors

cat: /var/opt/gitlab/gitlab-rails/VERSION: No such file or directory

This error is actually not a problem, but caused by the following line, which is produces an error even under expected circumstances:

if old_version=$(cat /var/opt/gitlab/gitlab-rails/VERSION)

/opt/gitlab/embedded/bin/runsvdir-start Doesn’t do sanity checks and attempts to perform privileged operations that fail

/opt/gitlab/embedded/bin/runsvdir-start: line 34: ulimit: max user processes: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system

I didn’t see anything in the install documentation suggesting that this container requires privileged:true to run properly. Furthermore, at least on my system, some of the default container settings exceed what runsvdir-start attempts to increase. It seems like, if it is intended that the container attempt to execute privileged operations, the documentation should indicate as much. Furthermore, I don’t think privileged:true is a blanket solution: the documentation should explain what this container actually requires to run, so that we can properly configure the container to have only what it requires rather than simply setting privileged:true.

Permissions configuration seems to be completely broken

This container seems to break its own permissions requirements, requiring manual intervention just to get it to run for the first time.

I spent hours tracking down this error:

Recipe: gitaly::git_data_dirs

  • storage_directory[/var/opt/gitlab/git-data/repositories] action create

    • ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run

      ================================================================================
      Error executing action run on resource ‘ruby_block[directory resource: /var/opt/gitlab/git-data/repositories]’
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received ‘1’
      ---- Begin output of stat --printf=‘%U’ $(readlink -f /var/opt/gitlab/git-data/repositories) ----
      STDOUT:
      STDERR: stat: missing operand
      Try ‘stat --help’ for more information.
      ---- End output of stat --printf=‘%U’ $(readlink -f /var/opt/gitlab/git-data/repositories) ----
      Ran stat --printf=‘%U’ $(readlink -f /var/opt/gitlab/git-data/repositories) returned 1

In my case, the /var/opt/gitlab/git-data/repositories directory existed and had proper permissions. Furthermore, attempting to manually run stat --printf=‘%U’ $(readlink -f /var/opt/gitlab/git-data/repositories) returned the expected output of git. I eventually figured out that this command was failing when run as git: it was returning nothing. Further investigation yielded that the parent directory, /var/opt/gitlab was owned by root and not accessible to the git user. Not sure why this would be broken on a brand new container build with no modification settings, but fine, I issued a chown git:git /var/opt/gitlab and re-ran /assets/init-container.

That’s when it got even more frustrating: the init process resets the directory back to root ownership! Every time the init process runs, it resets the ownership and blocks the git user from being able to access the required paths.

I overcame this issue by issuing a setfacl from the host to create an ACL to override whatever the heck is going on here and force the git user to have permission to /var/opt/gitlab.

After I got passed the first failure with /var/opt/gitlab/git-data/repositories, I ran the init script again and got the same error with a different directory. And then another. And another. It seems this script is going to create each directory with bad permissions that I apparently have to manually fix, one crash at a time, until everything is created. This is a disaster.

I also attempted to run /assets/init-container as git instead of root, hoping perhaps that was an oversight and it was meant to be run as the app user. Nope. That fails miserably too, with numerous Permission denied errors (although, thankfully, it fails much faster, earlier in the init process).

This doesn’t seem like “the way…”

I did some more research and I reviewed /assets/setup … this file seems like it should have created all of the directories I’m struggling with. Based on that, I re-created my container hoping perhaps something simply failed during the initial build. Same issue: the directories don’t exist on first run, and aren’t created until the init process attempts to build them (which then fails, because it assigns them all root ownership).

So, I attempted to run /assets/setup manually in the container. I can’t. It depends on environment variables, and I can’t find suitable values to provide to it. So, it just fails miserably while attempting to use non-existent variables that it doesn’t sanity-check before passing missing parameters to other commands and generating syntax errors.

root@67af1113f604:/var/opt/gitlab# /assets/setup
Unknown TARGETARCH: DOWNLOAD_URL not set
Downloading package as artifact -
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
There was an error downloading . Please check the output for more information
mkfifo: cannot create fifo '/opt/gitlab/sv/sshd/supervise/ok': File exists
mkfifo: cannot create fifo '/opt/gitlab/sv/sshd/log/supervise/ok': File exists

Notwithstanding, there is evidence that this script ran at least once, at least partially successfully: all of the users from the series of useradd commands in the file are listed in /etc/passwd. For some reason, despite the useradd -m, none of the home directories for these users are present after the container is built. Because I can’t successfully run the command, I’m not sure how to troubleshoot it further.
I’m not sure if there’s something else broken during the app startup process that removes these directories after they’ve been created, or if the useradd -m is failing to initially create the directories. The final result, however, is a very broken image that cannot run successfully.

How the heck does one run this container???

Steps to reproduce

Configuration

1 networks:
2   default:
3     external: true
4     name: [network]
5
6 services:
7   gitlab:
8     image: gitlab/gitlab-ce:18.11.3-ce.0
9     environment:
10       GITLAB_OMNIBUS_CONFIG: |
11         external_url 'http://[domain.tld]'
12         letsencrypt['enable'] = false
13         nginx['listen_port'] = 80
14         nginx['listen_https'] = false
15     labels:
16       - [reverse proxy config]
17       - [reverse proxy config]
18       - [reverse proxy config]
19     restart: unless-stopped
20     shm_size: '256m'
21     volumes:
22       - ./data:/var/opt/gitlab
23       - ./logs:/var/log/gitlab
24       - ./config:/etc/gitlab

Versions

Please add an x whether options apply, and add the version information.

  • Self-managed: Docker Current version: gitlab-ce=18.11.3-ce.0
  • GitLab.com SaaS
  • Dedicated

Versions

  • GitLab (Web: /help or self-managed system information sudo gitlab-rake gitlab:env:info):
  • Error:

  • root@67af1113f604:/# gitlab-rake gitlab:env:info
    /opt/gitlab/bin/gitlab-rake error: could not load /opt/gitlab/etc/gitlab-rails-rc
    Either you are not allowed to read the file, or it does not exist yet.
    You can generate it with: sudo gitlab-ctl reconfigure

2

Created Bug report in the GitLabs repo