I am using GitLab within Docker. My current version is 12.8.5 and I was looking to upgrade to 12.10.1. When I started a new container with the new version I noticed it didn’t come up as other upgrades have. Further research has led me into a rathole that I need some assistance with.
When I tried to upgrade using the existing volumes, I noticed an error in the docker logs output. I have it at the bottom of this post to not clutter this text. I decided to try 12.9 and saw the same problem. Same with 12.8.9. These containers do come up if I move my /srv/gitlab directory out of the way so it’s more an upgrade problem. Please offer any guidance on how I can troubleshoot and fix this. Thank you.
* postgresql_query[enable pg_trgm extension] action run (skipped due to only_if)
(up to date)
* ruby_block[warn pending postgresql restart] action run (skipped due to only_if)
* execute[reload postgresql] action nothing (skipped due to action :nothing)
* execute[start postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/log/grpc.log
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/grpc.rb:3:in `new'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/grpc.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200424-22-1ptwcgl" ----
STDOUT: Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/log/grpc.log
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/grpc.rb:3:in `new'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/grpc.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20200424-22-1ptwcgl" ----
Ran "bash" "/tmp/chef-script20200424-22-1ptwcgl" returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
54: bash "migrate gitlab-rails database" do
55: code <<-EOH
56: set -e
57: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
58: umask 077
59: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
60: STATUS=${PIPESTATUS[0]}
61: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file}
62: echo $STATUS > #{db_migrate_status_file}
63: exit $STATUS
64: EOH
65: environment env_variables unless env_variables.empty?
66: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
67: dependent_services.each do |svc|
68: notifies :restart, svc, :immediately
69: end
70: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
71: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
72: end
73:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:54:in `from_file'
bash("migrate gitlab-rails database") do
action [:run]
default_guard_interpreter :default
command nil
backup 5
interpreter "bash"
declared_type :bash
cookbook_name "gitlab"
recipe_name "database_migrations"
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-f4e077c4310\n exit $STATUS\n"
domain nil
user nil
not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-f4e077c4310) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-f4e077c4310 | grep -Fx 0)"
only_if { #code block }
end
System Info:
------------
chef_version=14.14.29
platform=ubuntu
platform_version=16.04
ruby=ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
Running handlers complete
Chef Client failed. 8 resources updated in 28 seconds
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200424-22-1ptwcgl" ----
STDOUT: Rails Error: Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/log/grpc.log
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/grpc.rb:3:in `new'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/grpc.rb:3:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure => environment
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20200424-22-1ptwcgl" ----
Replace this template with your information
Describe your question in as much detail as possible:
What are you seeing, and how does it differ from what you expect to see?
Consider including screenshots, error messages, and/or other helpful visuals
What version are you on (Hint: /help) ? and are you using self-managed or gitlab.com?
What troubleshooting steps have you already taken? Can you link to any docs or other resources so we know where you have been?
Thanks for taking the time to be thorough in your request, it really helps!
Can you please shed some light on the commands / procedure you are using for doing so?
Maybe you are missing some steps with latest 12.8 → latest 12.9 → latest 12.10.
@dnsmichi I have a simple bash script I run which upgrades the Docker container. I set the version to what I want. I’m currently on 12.8.5 and want to move to 12.8.9 to start.
thanks. I’m wondering about sudo here. The error message above says that the log directories cannot be written.
Unable to access log file. Please ensure that /opt/gitlab/embedded/service/gitlab-rails/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /opt/gitlab/embedded/service/gitlab-rails/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
rake aborted!
I’d say that the mounted volumes from Docker are owned by root and are not writable inside the container. Can you share the output of the following commands please:
ls -lah /srv/gitlab/logs
ls -lah /srv/gitlab/data
Also, is the host running on RHEL/CentOS/Fedora where SELinux could come into play? Please share the output of that command too.
puh, this leaves me out of ideas for the moment unfortunately. I haven’t run CentOS 8 on my servers yet.
Do you have a staging environment around, as in, a new host with a fresh CentOS 8 install where you could move the Docker container temporarily and see if the problems persist? Or, likewise, do the whole dance on an Ubuntu VM too so see if that has an impact.
Just to confirm, you’re suggesting I move the container to another host and have it create a new environment since it won’t have the volumes? Or should I move the volumes with it?
But yes, this is in a lab so I have a lot of leeway on what I can deploy. Deploying a new environment will be easy.
I meant the entire environment - container + state data in directories. It would be worth a short to rule out said permission problems. Please do keep a backup to stay safe though
@dnsmichi I created a new system, this time running CentOS 7. I figured Docker isn’t very happy on CentOS 8 so a 7 box would be an okay place to start, if another variable. Unfortunately it doesn’t appear to have solved my problems. Logs are below.
All I did was create a new VM, install Docker, scp all the files from /srv to the new system and run the Docker run command.
There are a couple of other topics unveiled when googling for gitlab docker container srv permissions - I’m not sure if related but maybe they’ll help in your research.
I’m testing further but the permission fixing command seems to have fixed it. I was able to upgrade from 12.8.5 to 12.8.9 and will upgrade to 12.9, 12.10, and 13 shortly. Thank you for your assistance here.
