I upgraded my gitlab host from debian 9 (stretch) to 11 (bullseye). I’m installing gitlab-ce from the gitlab package repo, using buster as the release. I had already upgraded gitlab to 14.4.0 last week, so the version did not change with my os update. After finishing the update and rebooting, I get the following message trying to run gitlab-ctl reconfigure
:
[begin]
[snip]
* Parent directory /sys/fs/cgroup/cpu does not exist, cannot create /sys/fs/cgroup/cpu/gitaly
================================================================================
Error executing action `create` on resource 'directory[/sys/fs/cgroup/cpu/gitaly]'
================================================================================
[snip]
Running handlers:
There was an error running gitlab-ctl reconfigure:
directory[/sys/fs/cgroup/cpu/gitaly] (gitaly::enable line 53) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /sys/fs/cgroup/cpu does not exist, cannot create /sys/fs/cgroup/cpu/gitaly
[end]
The path /sys/fs/cgroup/cpu/
indeed does not exist. What is the workaround to get gitaly and my server back up and running again?
Thanks.
Seth
Hi,
I don’t have it on mine, but seems to work fine with gitlaly. Do you have some sort of specific gitaly config in /etc/gitlab.rb?
root@gitlab:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
root@gitlab:~# uname -r
5.10.0-9-amd64
root@gitlab:~# ls /sys/fs/cgroup/ | grep -i cpu
cpu.pressure
cpu.stat
cpuset.cpus.effective
cpuset.mems.effective
root@gitlab:~# gitlab-ctl status | grep -i gitaly
run: gitaly: (pid 592) 111s; run: log: (pid 584) 111s
Perhaps you need to remove any lines in gitlab.rb relating to this /sys/fs/cgroup/cpu by commenting it out, and letting the default gitlab settings work. All lines in config relating to gitaly:
root@gitlab:/etc/gitlab# cat gitlab.rb | grep -i gitaly
# gitlab_rails['backup_gitaly_backup_path'] = "/opt/gitlab/embedded/bin/gitaly-backup"
### Gitaly settings
# gitlab_rails['gitaly_token'] = 'secret token'
## Gitaly
# The gitaly['enable'] option exists for the purpose of cluster
# deployments, see https://docs.gitlab.com/ee/administration/gitaly/index.html .
# gitaly['enable'] = true
# gitaly['dir'] = "/var/opt/gitlab/gitaly"
# gitaly['log_directory'] = "/var/log/gitlab/gitaly"
# gitaly['bin_path'] = "/opt/gitlab/embedded/bin/gitaly"
# gitaly['env_directory'] = "/opt/gitlab/etc/gitaly/env"
# gitaly['env'] = {
##! internal_socket_dir is the directory that will contain internal gitaly sockets,
# gitaly['internal_socket_dir'] = "/var/opt/gitlab/gitaly"
# gitaly['socket_path'] = "/var/opt/gitlab/gitaly/gitaly.socket"
# gitaly['listen_addr'] = "localhost:8075"
# gitaly['tls_listen_addr'] = "localhost:9075"
# gitaly['certificate_path'] = "/var/opt/gitlab/gitaly/certificate.pem"
# gitaly['key_path'] = "/var/opt/gitlab/gitaly/key.pem"
# gitaly['prometheus_listen_addr'] = "localhost:9236"
# gitaly['logging_level'] = "warn"
# gitaly['logging_format'] = "json"
# gitaly['logging_sentry_dsn'] = "https://<key>:<secret>@sentry.io/<project>"
# gitaly['logging_ruby_sentry_dsn'] = "https://<key>:<secret>@sentry.io/<project>"
# gitaly['logging_sentry_environment'] = "production"
# gitaly['prometheus_grpc_latency_buckets'] = "[0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]"
# gitaly['auth_token'] = '<secret>'
# gitaly['auth_transitioning'] = false # When true, auth is logged to Prometheus but NOT enforced
# gitaly['graceful_restart_timeout'] = '1m' # Grace time for a gitaly process to finish ongoing requests
# gitaly['git_catfile_cache_size'] = 100 # Number of 'git cat-file' processes kept around for re-use
# gitaly['git_bin_path'] = "/opt/gitlab/embedded/bin/git" # A custom path for the 'git' executable
# gitaly['open_files_ulimit'] = 15000 # Maximum number of open files allowed for the gitaly process
# gitaly['ruby_max_rss'] = 300000000 # RSS threshold in bytes for triggering a gitaly-ruby restart
# gitaly['ruby_graceful_restart_timeout'] = '10m' # Grace time for a gitaly-ruby process to finish ongoing requests
# gitaly['ruby_restart_delay'] = '5m' # Period of sustained high RSS that needs to be observed before restarting gitaly-ruby
# gitaly['ruby_rugged_git_config_search_path'] = "/opt/gitlab/embedded/etc" # Location of system-wide gitconfig file
# gitaly['ruby_num_workers'] = 3 # Number of gitaly-ruby worker processes. Minimum 2, default 2.
# gitaly['concurrency'] = [
# 'rpc' => "/gitaly.SmartHTTPService/PostReceivePack",
# 'rpc' => "/gitaly.SSHService/SSHUploadPack",
# gitaly['daily_maintenance_start_hour'] = 22
# gitaly['daily_maintenance_start_minute'] = 30
# gitaly['daily_maintenance_duration'] = '30m'
# gitaly['daily_maintenance_storages'] = ["default"]
# gitaly['daily_maintenance_disabled'] = false
# gitaly['cgroups_count'] = 10
# gitaly['cgroups_mountpoint'] = '/sys/fs/cgroup'
# gitaly['cgroups_hierarchy_root'] = 'gitaly'
# gitaly['cgroups_memory_enabled'] = true
# gitaly['cgroups_memory_limit'] = 1048576
# gitaly['cgroups_cpu_enabled'] = true
# gitaly['cgroups_cpu_shares'] = 512
# gitaly['pack_objects_cache_enabled'] = true
# gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache'
# gitaly['pack_objects_cache_max_age'] = '5m'
##! Docs: https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_ha.md
# 'GITALY_PID_FILE' => "/var/opt/gitlab/praefect/praefect.pid",
# praefect['wrapper_path'] = "/opt/gitlab/embedded/bin/gitaly-wrapper"
all lines relating to gitaly and cgroup:
root@gitlab:/etc/gitlab# cat gitlab.rb | grep -i gitaly | grep cgroup
# gitaly['cgroups_count'] = 10
# gitaly['cgroups_mountpoint'] = '/sys/fs/cgroup'
# gitaly['cgroups_hierarchy_root'] = 'gitaly'
# gitaly['cgroups_memory_enabled'] = true
# gitaly['cgroups_memory_limit'] = 1048576
# gitaly['cgroups_cpu_enabled'] = true
# gitaly['cgroups_cpu_shares'] = 512
The problem turned out to be the move from cgroups v1 to v2 with the OS upgrade. I resolved it by forcing v1 /sys node structure by adding systemd.unified_cgroup_hierarchy=false
to the kernel boot line in grub. I do like your idea of simply removing gitaly-related customization in gitlab.rb. I might also give that a try as it would cause fewer potential problems in the future. Thanks.
1 Like
I have also started running into this error after Gitaly suddenly terminated and started again. I am using Gitlab-ce docker and the file system is read only /sys/fs/cgroup/cpu
All the cgroups are v1. the gitlab.rb dont have any gitaly configurations. Is this a bug it Gitlab version?
time=“2024-03-14T06:48:14Z” level=info msg=“Starting Gitaly, version 15.8.1”
{“latencies”:[0.001,0.005,0.025,0.1,0.5,1,10,30,60,300,1500],“level”:“info”,“msg”:“grpc prometheus histograms enabled”,“time”:“2024-03-14T06:48:14.252Z”}
{“level”:“info”,“msg”:“License database preloaded”,“time”:“2024-03-14T06:48:22.752Z”}
{“error”:“list gitaly process directory: open /sys/fs/cgroup/memory/gitaly: no such file or directory”,“level”:“error”,“msg”:“failed to clean up memory cgroups”,“time”:“2024-03-14T06:48:22.753Z”}
{“error”:“list gitaly process directory: open /sys/fs/cgroup/cpu/gitaly: no such file or directory”,“level”:“error”,“msg”:“failed to clean up cpu cgroups”,“time”:“2024-03-14T06:48:22.753Z”}
any recommendation on the solution to this problem