Gitaly does not start up:
Git cannot connect to Gitaly as it does not start up.
Gitaly logs:
{"level":"info","msg":"Wrapper started","time":"2020-02-26T16:42:31Z","wrapper":6649}
{"level":"info","msg":"finding gitaly","pid_file":"/var/opt/gitlab/gitaly/gitaly.pid","time":"2020-02-26T16:42:31Z","wrapper":6649}
{"level":"info","msg":"spawning a process","time":"2020-02-26T16:42:31Z","wrapper":6649}
{"gitaly":6659,"level":"info","msg":"monitoring gitaly","time":"2020-02-26T16:42:31Z","wrapper":6649}
time="2020-02-26T16:42:31Z" level=info msg="Starting Gitaly" version="Gitaly, version 12.8.1"
time="2020-02-26T16:42:31Z" level=warning msg="git path not configured. Using default path resolution" resolvedPath=/opt/gitlab/embedded/bin/git
time="2020-02-26T16:42:31Z" level=info msg="clearing disk cache object folder" path=/export/data/gitlab/git-data/repositories
time="2020-02-26T16:42:31Z" level=info msg="moving disk cache object folder to /export/data/gitlab/git-data/repositories/+gitaly/tmp/diskcache299540575" path=/export/data/gitlab/git-data/repositories
time="2020-02-26T16:42:31Z" level=info msg="disk cache object folder doesn't exist, no need to remove" path=/export/data/gitlab/git-data/repositories
time="2020-02-26T16:42:31Z" level=info msg="Starting file walker for /export/data/gitlab/git-data/repositories/+gitaly/cache" path=/export/data/gitlab/git-data/repositories/+gitaly/cache
time="2020-02-26T16:42:31Z" level=info msg="Starting file walker for /export/data/gitlab/git-data/repositories/+gitaly/state" path=/export/data/gitlab/git-data/repositories/+gitaly/state
{"level":"fatal","msg":"open : no such file or directory","time":"2020-02-26T16:42:32Z"}
{"gitaly":6659,"level":"warning","msg":"forwarding signal","signal":17,"time":"2020-02-26T16:42:32Z","wrapper":6649}
{"error":"os: process already finished","gitaly":6659,"level":"error","msg":"can't forward the signal","signal":17,"time":"2020-02-26T16:42:32Z","wrapper":6649}
{"gitaly":6659,"level":"error","msg":"wrapper for gitaly shutting down","time":"2020-02-26T16:42:33Z","wrapper":6649}
I don’t really see which file or directory it misses, as it is missing from the logs:
{“level”:“fatal”,“msg”:“open : no such file or directory”,“time”:“2020-02-26T16:42:32Z”}
I tried checked the directories the logs mention above the error, they do not exist by default, but if I create them and give access to Git, then GitLab moves them away.
Gitaly config:
# The gitaly['enable'] option exists for the purpose of cluster
# deployments, see https://docs.gitlab.com/ee/administration/gitaly/index.html .
gitaly['enable'] = true
gitaly['dir'] = "/var/opt/gitlab/gitaly"
gitaly['log_directory'] = "/export/logs/gitlab/gitaly"
gitaly['bin_path'] = "/opt/gitlab/embedded/bin/gitaly"
gitaly['env_directory'] = "/opt/gitlab/etc/gitaly/env"
gitaly['env'] = {
'PATH' => "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin",
'HOME' => "/var/opt/gitlab",
}
##! internal_socket_dir is the directory that will contain internal gitaly sockets,
##! separate from socket_path which is the socket that external clients listen on
gitaly['internal_socket_dir'] = "/var/opt/gitlab/gitaly"
gitaly['socket_path'] = "/var/opt/gitlab/gitaly/gitaly.socket"
gitaly['listen_addr'] = "localhost:8075"
gitaly['tls_listen_addr'] = "localhost:9075"
gitaly['prometheus_listen_addr'] = "localhost:9236"
gitaly['logging_level'] = "warn"
gitaly['logging_format'] = "json"
gitaly['logging_sentry_environment'] = "production"
gitaly['prometheus_grpc_latency_buckets'] = "[0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]"
gitaly['auth_token'] = '[OMITTED]'
gitaly['auth_transitioning'] = false # When true, auth is logged to Prometheus but NOT enforced
gitaly['graceful_restart_timeout'] = '1m' # Grace time for a gitaly process to finish ongoing requests
gitaly['git_catfile_cache_size'] = 100 # Number of 'git cat-file' processes kept around for re-use
gitaly['open_files_ulimit'] = 15000 # Maximum number of open files allowed for the gitaly process
gitaly['ruby_max_rss'] = 300000000 # RSS threshold in bytes for triggering a gitaly-ruby restart
gitaly['ruby_graceful_restart_timeout'] = '10m' # Grace time for a gitaly-ruby process to finish ongoing requests
gitaly['ruby_restart_delay'] = '5m' # Period of sustained high RSS that needs to be observed before restarting gitaly-ruby
gitaly['ruby_rugged_git_config_search_path'] = "/opt/gitlab/embedded/etc" # Location of system-wide
gitconfig file
gitaly['ruby_num_workers'] = 3 # Number of gitaly-ruby worker processes. Minimum 2, default 2.
gitaly['concurrency'] = [
{
'rpc' => "/gitaly.SmartHTTPService/PostReceivePack",
'max_per_repo' => 20
},
{
'rpc' => "/gitaly.SSHService/SSHUploadPack",
'max_per_repo' => 5
},
]