Gitaly fails to start, unable to create file in /sys

I upgraded my gitlab host from debian 9 (stretch) to 11 (bullseye). I’m installing gitlab-ce from the gitlab package repo, using buster as the release. I had already upgraded gitlab to 14.4.0 last week, so the version did not change with my os update. After finishing the update and rebooting, I get the following message trying to run gitlab-ctl reconfigure:

[begin]
[snip]
    * Parent directory /sys/fs/cgroup/cpu does not exist, cannot create /sys/fs/cgroup/cpu/gitaly
    ================================================================================
    Error executing action `create` on resource 'directory[/sys/fs/cgroup/cpu/gitaly]'
    ================================================================================
[snip]
Running handlers:
There was an error running gitlab-ctl reconfigure:

directory[/sys/fs/cgroup/cpu/gitaly] (gitaly::enable line 53) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /sys/fs/cgroup/cpu does not exist, cannot create /sys/fs/cgroup/cpu/gitaly
[end]

The path /sys/fs/cgroup/cpu/ indeed does not exist. What is the workaround to get gitaly and my server back up and running again?

Thanks.
Seth

Hi,

I don’t have it on mine, but seems to work fine with gitlaly. Do you have some sort of specific gitaly config in /etc/gitlab.rb?

root@gitlab:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

root@gitlab:~# uname -r
5.10.0-9-amd64

root@gitlab:~# ls /sys/fs/cgroup/ | grep -i cpu
cpu.pressure
cpu.stat
cpuset.cpus.effective
cpuset.mems.effective

root@gitlab:~# gitlab-ctl status | grep -i gitaly
run: gitaly: (pid 592) 111s; run: log: (pid 584) 111s

Perhaps you need to remove any lines in gitlab.rb relating to this /sys/fs/cgroup/cpu by commenting it out, and letting the default gitlab settings work. All lines in config relating to gitaly:

root@gitlab:/etc/gitlab# cat gitlab.rb | grep -i gitaly
# gitlab_rails['backup_gitaly_backup_path'] = "/opt/gitlab/embedded/bin/gitaly-backup"
### Gitaly settings
# gitlab_rails['gitaly_token'] = 'secret token'
## Gitaly
# The gitaly['enable'] option exists for the purpose of cluster
# deployments, see https://docs.gitlab.com/ee/administration/gitaly/index.html .
# gitaly['enable'] = true
# gitaly['dir'] = "/var/opt/gitlab/gitaly"
# gitaly['log_directory'] = "/var/log/gitlab/gitaly"
# gitaly['bin_path'] = "/opt/gitlab/embedded/bin/gitaly"
# gitaly['env_directory'] = "/opt/gitlab/etc/gitaly/env"
# gitaly['env'] = {
##! internal_socket_dir is the directory that will contain internal gitaly sockets,
# gitaly['internal_socket_dir'] = "/var/opt/gitlab/gitaly"
# gitaly['socket_path'] = "/var/opt/gitlab/gitaly/gitaly.socket"
# gitaly['listen_addr'] = "localhost:8075"
# gitaly['tls_listen_addr'] = "localhost:9075"
# gitaly['certificate_path'] = "/var/opt/gitlab/gitaly/certificate.pem"
# gitaly['key_path'] = "/var/opt/gitlab/gitaly/key.pem"
# gitaly['prometheus_listen_addr'] = "localhost:9236"
# gitaly['logging_level'] = "warn"
# gitaly['logging_format'] = "json"
# gitaly['logging_sentry_dsn'] = "https://<key>:<secret>@sentry.io/<project>"
# gitaly['logging_ruby_sentry_dsn'] = "https://<key>:<secret>@sentry.io/<project>"
# gitaly['logging_sentry_environment'] = "production"
# gitaly['prometheus_grpc_latency_buckets'] = "[0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]"
# gitaly['auth_token'] = '<secret>'
# gitaly['auth_transitioning'] = false # When true, auth is logged to Prometheus but NOT enforced
# gitaly['graceful_restart_timeout'] = '1m' # Grace time for a gitaly process to finish ongoing requests
# gitaly['git_catfile_cache_size'] = 100 # Number of 'git cat-file' processes kept around for re-use
# gitaly['git_bin_path'] = "/opt/gitlab/embedded/bin/git" # A custom path for the 'git' executable
# gitaly['open_files_ulimit'] = 15000 # Maximum number of open files allowed for the gitaly process
# gitaly['ruby_max_rss'] = 300000000 # RSS threshold in bytes for triggering a gitaly-ruby restart
# gitaly['ruby_graceful_restart_timeout'] = '10m' # Grace time for a gitaly-ruby process to finish ongoing requests
# gitaly['ruby_restart_delay'] = '5m' # Period of sustained high RSS that needs to be observed before restarting gitaly-ruby
# gitaly['ruby_rugged_git_config_search_path'] = "/opt/gitlab/embedded/etc" # Location of system-wide gitconfig file
# gitaly['ruby_num_workers'] = 3 # Number of gitaly-ruby worker processes. Minimum 2, default 2.
# gitaly['concurrency'] = [
#     'rpc' => "/gitaly.SmartHTTPService/PostReceivePack",
#     'rpc' => "/gitaly.SSHService/SSHUploadPack",
# gitaly['daily_maintenance_start_hour'] = 22
# gitaly['daily_maintenance_start_minute'] = 30
# gitaly['daily_maintenance_duration'] = '30m'
# gitaly['daily_maintenance_storages'] = ["default"]
# gitaly['daily_maintenance_disabled'] = false
# gitaly['cgroups_count'] = 10
# gitaly['cgroups_mountpoint'] = '/sys/fs/cgroup'
# gitaly['cgroups_hierarchy_root'] = 'gitaly'
# gitaly['cgroups_memory_enabled'] = true
# gitaly['cgroups_memory_limit'] = 1048576
# gitaly['cgroups_cpu_enabled'] = true
# gitaly['cgroups_cpu_shares'] = 512
# gitaly['pack_objects_cache_enabled'] = true
# gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache'
# gitaly['pack_objects_cache_max_age'] = '5m'
##! Docs: https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_ha.md
#  'GITALY_PID_FILE' => "/var/opt/gitlab/praefect/praefect.pid",
# praefect['wrapper_path'] = "/opt/gitlab/embedded/bin/gitaly-wrapper"

all lines relating to gitaly and cgroup:

root@gitlab:/etc/gitlab# cat gitlab.rb | grep -i gitaly | grep cgroup
# gitaly['cgroups_count'] = 10
# gitaly['cgroups_mountpoint'] = '/sys/fs/cgroup'
# gitaly['cgroups_hierarchy_root'] = 'gitaly'
# gitaly['cgroups_memory_enabled'] = true
# gitaly['cgroups_memory_limit'] = 1048576
# gitaly['cgroups_cpu_enabled'] = true
# gitaly['cgroups_cpu_shares'] = 512

The problem turned out to be the move from cgroups v1 to v2 with the OS upgrade. I resolved it by forcing v1 /sys node structure by adding systemd.unified_cgroup_hierarchy=false to the kernel boot line in grub. I do like your idea of simply removing gitaly-related customization in gitlab.rb. I might also give that a try as it would cause fewer potential problems in the future. Thanks.