Please find some further info showing my working so far below; this is really as far as I can take this issue and I will need help from someone who knows the internals of Gitlab upgrades in order to address this issue -
Analysis
Files that were mentioned in the error exist and their metadata (only) was modified during the falied attempted upgrade
[root@myhost ~]# stat /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp
File: ‘/opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp’
Size: 1468 Blocks: 8 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 2349007031 Links: 1
Access: (0644/-rw-r–r–) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:usr_t:s0
Access: 2021-04-14 15:36:24.748332982 +0100
Modify: 2021-03-08 11:14:03.000000000 +0000
Change: 2021-04-14 15:34:22.811319087 +0100
Birth: -
[root@myhost ~]#
[root@myhost ~]# stat /usr/libexec/selinux/hll/pp
File: ‘/usr/libexec/selinux/hll/pp’
Size: 11336 Blocks: 24 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 135567832 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:bin_t:s0
Access: 2021-04-14 15:36:34.622334107 +0100
Modify: 2017-08-04 10:00:34.000000000 +0100
Change: 2018-02-20 15:06:52.866867518 +0000
Birth: -
[root@myhost ~]#
SELinux is installed and enabled in Permissive mode
[root@myhost ~]# getenforce
Permissive
It seems that Chef Recipe: gitlab::selinux. (i.e., /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/selinux.rb:32)
Runs semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp
Which calls /usr/libexec/selinux/hll/pp
In order to read policy package libsemanage.semanage_direct_commit: to compile hll files into cil files
When doing this Bash is asked to [Set proper security context on ssh files for selinux]
Which errors ValueError: Type gitlab_shell_t is invalid, must be a file or device type
Research says _ _t is nomenclature for SELinux process type
gitlab_shell_t is typical reference for SELinux type
semanage -t operates on the type (in this case gitlab_shell_t so it seems ok though I don’t fully understand SELinux
semanage SELinux Policy Management tool
fcontext Manage file context mapping definitions
restorecon is primarily used to set the security context (extended attributes) on one or more files.
can be run at any time to correct errors,
to add support for new policy,
or with the -n option it can just check whether the file contexts are all as you expect
It appears that the following is the recipe line that nose-dives
code "semanage fcontext -a -t gitlab_shell_t '/var/opt/gitlab/.ssh(/.*)?'\nrestorecon -R -v '/var/opt/gitlab/.ssh'\nsemanage fcontext -a -t gitlab_shell_t '/var/opt/gitlab/.ssh/authorized_keys'\nrestorecon -v '/var/opt/gitlab/.ssh/authorized_keys'\nsemanage fcontext -a -t gitlab_shell_t '/var/opt/gitlab/gitlab-shell/config.yml'\nrestorecon -v '/var/opt/gitlab/gitlab-shell/config.yml'\nsemanage fcontext -a -t gitlab_shell_t '/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret'\nrestorecon -v '/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret'\nsemanage fcontext -a -t gitlab_shell_t '/var/opt/gitlab/gitlab-workhorse/sockets'"
Of the files referenced in that command, I am missing only /var/opt/gitlab/gitlab-workhorse/sockets as follows
[root@myhost ~]# ls /var/opt/gitlab/.ssh/.* /var/opt/gitlab/.ssh /var/opt/gitlab/.ssh/authorized_keys /var/opt/gitlab/gitlab-shell/config.yml /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret /var/opt/gitlab/gitlab-workhorse/sockets
[cut]
ls: cannot access /var/opt/gitlab/gitlab-workhorse/sockets: No such file or directory
[cut]
[root@myhost ~]#
So I think Chef is trying to manage the SELinux file context on /var/opt/gitlab/gitlab-workhorse/sockets but that file doesn’t exist.
Next, I checked my test environment and found the following diff between Prod (above) and Test
Test: (Note plural dir and singular file)
[root@my-test-host sockets]# locate socket | grep workhorse
/var/opt/gitlab/gitlab-workhorse/sockets
/var/opt/gitlab/gitlab-workhorse/sockets/socket
Prod: (Note singular dir)
Singular socket was created today and does not match 1. Chef recipe plural sockets dir / singular socket file or 2. Those items on my test host
[root@ myhost ~]# stat /var/opt/gitlab/gitlab-workhorse/socket
File: ‘/var/opt/gitlab/gitlab-workhorse/socket’
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: fd00h/64768d Inode: 805337251 Links: 1
Access: (0777/srwxrwxrwx) Uid: ( 989/ git) Gid: ( 986/ git)
Context: system_u:object_r:var_t:s0
Access: 2021-04-14 15:27:37.473272896 +0100
Modify: 2021-04-14 15:27:34.431272549 +0100
Change: 2021-04-14 15:27:34.431272549 +0100
Birth: -
[root@ myhost ~]#
That’s my theory of what is wrong with the Gitlab upgrade - i.e., The socket named socket and missing the sockets parent dir (see below) was created by the upgrade (see timestamps above and below) and this socket does not match the sockets dir referenced in the selinux.rb Chef recipe.
[root@myhost ~]# ls -ld /var/opt/gitlab/gitlab-workhorse/socket
srwxrwxrwx. 1 git git 0 Apr 14 15:27 /var/opt/gitlab/gitlab-workhorse/socket
I realise that the above might be a 2+2=5 and the dir/socket might be wrong because the upgrade failed, but the symptoms seem to add up.
Please could someone ratify this theory and help me to obtain a fix for this problem?
Also, despite this failure – 1) My gitlab-ce package looks upgraded, 2) gitlab-ctl status says my services are all running but 3) My Gitlab GUI has a 500 error.
[root@myhost ~]# rpm -qa | grep gitlab-ce
gitlab-ce-13.9.3-ce.0.el7.x86_64
Transcript
Recipe: gitlab::selinux
-
execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-7.2.0-ssh-keygen.pp] action run (skipped due to not_if)
-
execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-10.5.0-ssh-authorized-keys.pp] action run (skipped due to not_if)
-
execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp] action run
[execute] libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
gitlab-13.5.0-gitlab-shell: libsepol.policydb_read: policydb module version 19 does not match my version range 4-17
gitlab-13.5.0-gitlab-shell: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
gitlab-13.5.0-gitlab-shell: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
(No such file or directory).
semodule: Failed!
================================================================================
Error executing action run on resource ‘execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp]’
Mixlib::ShellOut::ShellCommandFailed
Expected process to exit with [0], but received ‘1’
---- Begin output of semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp ----
STDOUT:
STDERR: libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
gitlab-13.5.0-gitlab-shell: libsepol.policydb_read: policydb module version 19 does not match my version range 4-17
gitlab-13.5.0-gitlab-shell: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
gitlab-13.5.0-gitlab-shell: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
(No such file or directory).
semodule: Failed!
---- End output of semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp ----
Ran semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp returned 1
Resource Declaration:
In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/selinux.rb
32: execute “semodule -i /opt/gitlab/embedded/selinux/rhel/7/#{gitlab_shell_module}.pp” do
33: not_if “getenforce | grep Disabled”
34: not_if “semodule -l | grep ‘^#{gitlab_shell_module}\s’”
35: end
36: end
Compiled Resource:
Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/selinux.rb:32:in `from_file’
execute(“semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp”) do
action [:run]
default_guard_interpreter :execute
command “semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp”
backup 5
declared_type :execute
cookbook_name “gitlab”
recipe_name “selinux”
domain nil
user nil
not_if “getenforce | grep Disabled”
not_if “semodule -l | grep ‘^gitlab-13.5.0-gitlab-shell\s’”
end
System Info:
chef_version=15.14.0
platform=centos
platform_version=7.4.1708
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
-
bash[Set proper security context on ssh files for selinux] action run
[execute] ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
================================================================================
Error executing action run on resource ‘bash[Set proper security context on ssh files for selinux]’
Mixlib::ShellOut::ShellCommandFailed
Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20210414-1784-k0mvb5” ----
STDOUT:
STDERR: ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
---- End output of “bash” “/tmp/chef-script20210414-1784-k0mvb5” ----
Ran “bash” “/tmp/chef-script20210414-1784-k0mvb5” returned 1
Resource Declaration:
In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/selinux.rb
40: bash “Set proper security context on ssh files for selinux” do
41: code lazy { SELinuxHelper.commands(node) }
42: only_if “id -Z”
43: not_if { !node[‘gitlab’][‘gitlab-rails’][‘enable’] }
44: action :nothing
45: end
Compiled Resource:
Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/selinux.rb:40:in `from_file’
bash(“Set proper security context on ssh files for selinux”) do
action [:nothing]
default_guard_interpreter :default
command nil
backup 5
interpreter “bash”
declared_type :bash
cookbook_name “gitlab”
recipe_name “selinux”
code “semanage fcontext -a -t gitlab_shell_t ‘/var/opt/gitlab/.ssh(/.*)?’\nrestorecon -R -v ‘/var/opt/gitlab/.ssh’\nsemanage fcontext -a -t gitlab_shell_t ‘/var/opt/gitlab/.ssh/authorized_keys’\nrestorecon -v ‘/var/opt/gitlab/.ssh/authorized_keys’\nsemanage fcontext -a -t gitlab_shell_t ‘/var/opt/gitlab/gitlab-shell/config.yml’\nrestorecon -v ‘/var/opt/gitlab/gitlab-shell/config.yml’\nsemanage fcontext -a -t gitlab_shell_t ‘/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret’\nrestorecon -v ‘/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret’\nsemanage fcontext -a -t gitlab_shell_t ‘/var/opt/gitlab/gitlab-workhorse/sockets’”
domain nil
user nil
not_if { #code block }
only_if “id -Z”
end
System Info:
chef_version=15.14.0
platform=centos
platform_version=7.4.1708
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: gitlab::puma
- runit_service[puma] action restart (up to date)
- runit_service[puma] action restart (up to date)
Recipe: gitlab::gitlab-rails
- execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Running handlers:
There was an error running gitlab-ctl reconfigure:
Multiple failures occurred:
- Mixlib::ShellOut::ShellCommandFailed occurred in Chef Infra Client run: execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp] (gitlab::selinux line 32) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp ----
STDOUT:
STDERR: libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/pp failed with code: 255. (No such file or directory).
gitlab-13.5.0-gitlab-shell: libsepol.policydb_read: policydb module version 19 does not match my version range 4-17
gitlab-13.5.0-gitlab-shell: libsepol.sepol_module_package_read: invalid module in module package (at section 0)
gitlab-13.5.0-gitlab-shell: Failed to read policy package
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
(No such file or directory).
semodule: Failed!
---- End output of semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp ----
Ran semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-13.5.0-gitlab-shell.pp returned 1
- Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: bash[Set proper security context on ssh files for selinux] (gitlab::selinux line 40) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20210414-1784-k0mvb5” ----
STDOUT:
STDERR: ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
ValueError: Type gitlab_shell_t is invalid, must be a file or device type
---- End output of “bash” “/tmp/chef-script20210414-1784-k0mvb5” ----
Ran “bash” “/tmp/chef-script20210414-1784-k0mvb5” returned 1
Running handlers complete
Chef Infra Client failed. 22 resources updated in 52 seconds
Running reconfigure: NOT OK
== Fatal error ==
Something went wrong during final reconfiguration, please check the output
== Reverting ==
ok: down: postgresql: 1s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 2245) 1s
== Reverted ==
== Reverted to 11.10. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: alertmanager: (pid 2306) 0s
ok: run: gitaly: (pid 2315) 1s
ok: run: gitlab-exporter: (pid 2333) 0s
ok: run: grafana: (pid 2343) 1s
ok: run: logrotate: (pid 2356) 0s
ok: run: node-exporter: (pid 2362) 1s
ok: run: postgres-exporter: (pid 2370) 0s
ok: run: prometheus: (pid 2377) 0s
ok: run: redis-exporter: (pid 2464) 1s
ok: run: sidekiq: (pid 2471) 0s
Toggling services: OK
Checking if a newer PostgreSQL version is available and attempting automatic upgrade to it: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs
warning: %posttrans(gitlab-ce-13.9.3-ce.0.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-13.9.3-ce.0.el7.x86_64
Verifying : gitlab-ce-13.9.3-ce.0.el7.x86_64 1/2
Verifying : gitlab-ce-13.1.11-ce.0.el7.x86_64 2/2
Updated:
gitlab-ce.x86_64 0:13.9.3-ce.0.el7
Complete!
[root@myhost ~]#