Where should I look to diagnose a startup failure using GitLab's Docker image?

How to Use GitLab
1

GitLab Docker image startup never finished

I’ve followed these instructions to start a Docker container. After 12 hours of 250% CPU there are 20 MB of logs and when I try to connect through a browser the server responds with GitLab took too long to respond (HTTP) or Can’t establish a secure connection to localhost (HTTPS). I’m on macOS 10.15 and have Docker Desktop 3.0.4 and Docker Engine 20.10.2.

The top of the log output is:

Thank you for using GitLab Docker Image!
Current version: gitlab-ee=13.7.4-ee.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab vim /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Cleaning stale PIDs & sockets
Installing gitlab.rb config...
Generating ssh_host_rsa_key...
Generating public/private rsa key pair.
Your identification has been saved in /etc/gitlab/ssh_host_rsa_key.
Your public key has been saved in /etc/gitlab/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:aa6A7mSHrjOyvA8F3lUhak9qVMbUAQGXE8HrtqTwMNo root@gitlab.example.com

The bottom of the log is:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/help" for 127.0.0.1 at 2021-01-16 21:14:25 +0000

==> /var/log/gitlab/gitaly/current <==
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:25Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:25Z","wrapper":296}
{"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"gitaly-ruby worker health check failed","time":"2021-01-16T21:14:24.988Z","worker.name":"gitaly-ruby.0"}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"01EW6GRPMQVCTE8HX7PE6HQT2K","duration_ms":9056,"error":"badgateway: failed to receive response: context canceled","level":"error","method":"GET","msg":"error","time":"2021-01-16T21:14:26Z","uri":"/help"}

==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2021-01-16T21:14:27.115Z","pid":718,"message":"PumaWorkerKiller: Consuming 1945.64453125 mb with master and 2 workers."}

==> /var/log/gitlab/gitaly/current <==
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:28Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:28Z","wrapper":296}

==> /var/log/gitlab/gitlab-workhorse/current <==
{"content_type":"text/html; charset=utf-8","correlation_id":"01EW6GRPMQVCTE8HX7PE6HQT2K","duration_ms":19425,"host":"localhost","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-01-16T21:14:29Z","ttfb_ms":19226,"uri":"/help","user_agent":"curl/7.74.0-DEV","written_bytes":2940}

==> /var/log/gitlab/gitaly/current <==
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:30Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:30Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:31Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:32Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:32Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:34Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:34Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:35Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:35Z","wrapper":296}
{"level":"info","msg":"I, [2021-01-16T21:02:15.280488 #408]  INFO -- sentry: ** [Raven] Raven 3.0.4 configured not to capture errors: DSN not set","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","346","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-01-16T21:14:35.091Z"}
{"level":"info","msg":"GRPC-RUBY: WARN: Using gitaly-proto 13.7.4","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","346","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-01-16T21:14:35.565Z"}
{"level":"info","msg":"GRPC-RUBY: WARN: failed call: #\u003cGRPC::ActiveCall:0x000000000411d420\u003e","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","346","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-01-16T21:14:35.721Z"}
{"level":"info","msg":"call#run_batch failed somehow","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","346","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-01-16T21:14:35.736Z"}
{"error":null,"level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","346","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-01-16T21:14:36.133Z"}
{"level":"warning","msg":"spawned","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","346","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","supervisor.pid":1931,"time":"2021-01-16T21:14:36.138Z"}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:36Z","wrapper":296}
{"gitaly":346,"level":"warning","msg":"forwarding signal","signal":23,"time":"2021-01-16T21:14:36Z","wrapper":296}

And finally, here is the error output:

cat: /var/opt/gitlab/gitlab-rails/VERSION: No such file or directory
/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.14.0/lib/chef/provider/remote_file/local_file.rb:43: warning: URI.unescape is obsolete
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.14.0/lib/chef/provider/remote_file/local_file.rb:43: warning: URI.unescape is obsolete
/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.14.0/lib/chef/provider/remote_file/local_file.rb:43: warning: URI.unescape is obsolete
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.14.0/lib/chef/provider/remote_file/local_file.rb:43: warning: URI.unescape is obsolete
There was an error running gitlab-ctl reconfigure:

runit_service[grafana] (monitoring::grafana line 155) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana ----
STDOUT: fail: /opt/gitlab/service/grafana: runsv not running
STDERR: 
---- End output of /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana ----
Ran /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana returned 1

/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.14.0/lib/chef/provider/remote_file/local_file.rb:43: warning: URI.unescape is obsolete
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/chef-15.14.0/lib/chef/provider/remote_file/local_file.rb:43: warning: URI.unescape is obsolete
2

Do you find solution for it? if it is so can you share

3

Sorry, no, I had to abandon that path because I wasn’t able to solve it.