Hello,
Yesterday I manage to upgrade my Omnibus Gitlab EE install from 13.9.4 to 13.10.0 on Debian 10.
Install seems successful but Web interface is down, Gitaly is ok.
After some researches I found the problem => some assets are wrongly named in Gitlab installation (missing themes files fires a 500, other files just broke some function). I recovered a little part by renaming some files in assets directory.
I tried :
- Many times to reconfigure and restart
- To reinstall package gitlab-ee
- To execute command “gitlab:assets:clean gitlab:assets:compile” has advised (but discouraged) by troubleshooting but it returns me that yarn is missing (but yarn is installed)
Can someone help me please ?
Here is a gitlab check (seems OK) :
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/7 ... yes
4/8 ... yes
7/9 ... yes
7/10 ... yes
7/11 ... yes
7/12 ... yes
8/13 ... yes
8/14 ... yes
8/15 ... yes
7/16 ... yes
7/17 ... yes
9/18 ... yes
9/19 ... yes
9/20 ... yes
9/21 ... yes
9/22 ... yes
9/23 ... yes
11/24 ... yes
11/25 ... yes
12/26 ... yes
12/27 ... yes
13/28 ... yes
12/29 ... yes
13/30 ... yes
13/31 ... yes
13/32 ... yes
15/33 ... yes
16/35 ... yes
16/36 ... yes
9/37 ... yes
9/38 ... yes
6/39 ... yes
9/40 ... yes
9/41 ... yes
19/42 ... yes
19/43 ... yes
12/44 ... yes
3/45 ... yes
20/46 ... yes
23/47 ... yes
23/48 ... yes
23/49 ... yes
23/50 ... yes
20/51 ... yes
20/52 ... yes
29/53 ... yes
14/54 ... yes
10/55 ... yes
29/56 ... yes
12/57 ... yes
6/58 ... yes
37/59 ... yes
14/60 ... yes
29/61 ... yes
10/62 ... yes
12/63 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 22
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
some assets are wrongly named in Gitlab installation (missing themes files fires a 500, other files just broke some function). I recovered a little part by renaming some files in assets directory.
Can you share the error message in your gitlab logs or browser console that indicated there are missing assets?
If you run sudo gitlab-ctl tail in a terminal on the GitLab server while trying to load a page in the web UI, what errors are captured there?
Depending on what you find there, I might suggest reinstalling GitLab entirely.
sudo apt-get install --reinstall gitlab-ee
Thanks for your reponse.
I corrected by renaming some themes files that fired errors in logs (errors was " ActionView::Template::Error: No such file or directory @ rb_sysopen ***") and now I got errors in console like those :
GET https://gitlab-***.com/assets/highlight/themes/white-b3993639b265e6a0de95d667365bc0dc4a707b70202945298c5715dc0d1f6159.css
[HTTP/2 404 Not Found 75ms]
GET https://gitlab-***.com/assets/webpack/runtime.cf2586c0.bundle.js
[HTTP/2 404 Not Found 105ms]
GET https://gitlab-***.com/assets/webpack/main.8311ebd7.chunk.js
[HTTP/2 404 Not Found 103ms]
GET https://gitlab-***.com/assets/webpack/pages.root.404483db.chunk.js
[HTTP/2 404 Not Found 101ms]
Also I get this kind of strange errors with “gitlab-ctl tail”, I don’t know what it means, maybe related with a very high CPU consumption (> 90 % most of the time) ?
==> /var/log/gitlab/puma/current <==
2021-03-27_13:59:34.83404 bundler: failed to load command: puma (/opt/gitlab/embedded/bin/puma)
2021-03-27_13:59:34.83434 RuntimeError: There is already a server bound to: /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
2021-03-27_13:59:34.83435 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/binder.rb:376:in `add_unix_listener'
2021-03-27_13:59:34.83436 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/binder.rb:213:in `block in parse'
2021-03-27_13:59:34.83437 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/binder.rb:152:in `each'
2021-03-27_13:59:34.83437 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/binder.rb:152:in `parse'
2021-03-27_13:59:34.83438 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/runner.rb:144:in `load_and_bind'
2021-03-27_13:59:34.83439 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/cluster.rb:342:in `run'
2021-03-27_13:59:34.83439 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/launcher.rb:182:in `run'
2021-03-27_13:59:34.83440 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/lib/puma/cli.rb:80:in `run'
2021-03-27_13:59:34.83440 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/puma-5.1.1/bin/puma:10:in `<top (required)>'
2021-03-27_13:59:34.83441 /opt/gitlab/embedded/bin/puma:23:in `load'
2021-03-27_13:59:34.83442 /opt/gitlab/embedded/bin/puma:23:in `<top (required)>'
2021-03-27_13:59:36.11664 {"timestamp":"2021-03-27T13:59:36.116Z","pid":31934,"message":"Puma starting in cluster mode..."}
2021-03-27_13:59:36.11694 {"timestamp":"2021-03-27T13:59:36.116Z","pid":31934,"message":"* Puma version: 5.1.1 (ruby 2.7.2-p137) (\"At Your Service\")"}
2021-03-27_13:59:36.12110 {"timestamp":"2021-03-27T13:59:36.116Z","pid":31934,"message":"* Min threads: 4"}
2021-03-27_13:59:36.12110 {"timestamp":"2021-03-27T13:59:36.116Z","pid":31934,"message":"* Max threads: 4"}
2021-03-27_13:59:36.12111 {"timestamp":"2021-03-27T13:59:36.117Z","pid":31934,"message":"* Environment: production"}
2021-03-27_13:59:36.12111 {"timestamp":"2021-03-27T13:59:36.117Z","pid":31934,"message":"* Master PID: 31934"}
2021-03-27_13:59:36.12112 {"timestamp":"2021-03-27T13:59:36.117Z","pid":31934,"message":"* Workers: 2"}
2021-03-27_13:59:36.12112 {"timestamp":"2021-03-27T13:59:36.117Z","pid":31934,"message":"* Restarts: (✔) hot (✖) phased"}
2021-03-27_13:59:36.12113 {"timestamp":"2021-03-27T13:59:36.117Z","pid":31934,"message":"* Preloading application"}
My server is a 2 CPU and 4 GB of RAM, I got ~ 15 active users (most of them with only a Git usage).
I do not know what to think about but in “Admin Area”, Gitlab is indicating 13.9.4-ee as version number. Maybe WebSite is still in the last version ? Packages on Debian are in 13.10.0 version.
Hello, I tried the apt-get install --reinstall gitlab-ee command as advised, install is successful but I retrieve the exact same state.
Install indicates “up to date” on some folders, maybe I need to delete something to force it to replace files ?
@gitlab-greg Can you advise about this please ?
Thank you !
Hi @ftassin,
We just released GitLab version 13.10.2 which fixes a number of regressions in GitLab 13.10. I suggest upgrading to 13.10.2 to see if this resolves your problem.
If not, I suggest clearing the redis cache on your instance.
Also, GitLab officially recommends a minimum of 4 CPU cores for best performance. Running GitLab on a machine with 4 cores instead of 2 would bring a noticeable performance improvement.
Thanks for your response @gitlab-greg
I did the 13.10.2 upgrade (launched before you advise) and it is exactly the same.
Also clearing the redis cache (+ reconfigure) did not do the trick…
I found two issues that seem relevant to the error message you’re seeing
To help troubleshoot further, can you provide sanitized output of the following commands:
grep -Ev "^\s*$|^\s*#" /etc/gitlab/gitlab.rb | grep -E "puma|unicorn|workhorse"
gitlab-rake db:migrate:status | grep -Ev "\ down\ "
gitlab-ctl status
sudo ps -ax | grep -E "puma|unicorn|workhorse"
Hello @gitlab-greg ,
As asked here are the results :
grep -Ev “^\s*$|^\s*#” /etc/gitlab/gitlab.rb | grep -E “puma|unicorn|workhorse” :
unicorn['enable'] = false
puma['enable'] = true
(I already searched this way, and explicitely enabled puma and disabled unicorn)
gitlab-rake db:migrate:status | grep -Ev “\ down\ " : Are you sure about this ? It returns hundred of lines (maybe thousand). There is no “down” line”.
gitlab-ctl status :
run: alertmanager: (pid 29591) 346865s; run: log: (pid 537) 10285692s
run: gitaly: (pid 29603) 346865s; run: log: (pid 531) 10285693s
run: gitlab-exporter: (pid 29618) 346864s; run: log: (pid 543) 10285693s
run: gitlab-workhorse: (pid 29620) 346864s; run: log: (pid 536) 10285693s
run: grafana: (pid 29638) 346863s; run: log: (pid 532) 10285693s
run: logrotate: (pid 9211) 1197s; run: log: (pid 540) 10285693s
run: nginx: (pid 29652) 346862s; run: log: (pid 544) 10285693s
run: node-exporter: (pid 29660) 346862s; run: log: (pid 539) 10285693s
run: postgres-exporter: (pid 29670) 346861s; run: log: (pid 534) 10285693s
run: postgresql: (pid 29695) 346861s; run: log: (pid 535) 10285693s
run: prometheus: (pid 29697) 346860s; run: log: (pid 541) 10285693s
run: puma: (pid 11866) 90s; run: log: (pid 9167) 977521s
run: redis: (pid 29720) 346859s; run: log: (pid 526) 10285693s
run: redis-exporter: (pid 29729) 346859s; run: log: (pid 538) 10285693s
run: sidekiq: (pid 29817) 346854s; run: log: (pid 555) 10285693s
ps -ax | grep -E “puma|unicorn|workhorse” :
518 ? Ss 0:00 runsv gitlab-workhorse
536 ? S 5:11 svlogd /var/log/gitlab/gitlab-workhorse
7221 ? Sl 144:51 puma: cluster worker 1: 24069 [gitlab-puma-worker]
9166 ? Ss 0:16 runsv puma
9167 ? S 0:04 svlogd -tt /var/log/gitlab/puma
12277 ? Rsl 0:29 puma 5.1.1 (unix:///var/opt/gitlab/gitlab-rails/sockets/gitlab.socket,tcp://127.0.0.1:8080) [gitlab-puma-worker]
12343 pts/0 S+ 0:00 grep -E puma|unicorn|workhorse
29620 ? Ssl 5:05 /opt/gitlab/embedded/bin/gitlab-workhorse -listenNetwork unix -listenUmask 0 -listenAddr /var/opt/gitlab/gitlab-workhorse/sockets/socket -authBackend http://localhost:8080 -authSocket /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket -documentRoot /opt/gitlab/embedded/service/gitlab-rails/public -pprofListenAddr -prometheusListenAddr localhost:9229 -secretPath /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret -logFormat json -config config.toml
I resolved my problem by uninstalling, removing everything in /opt/gitlab and reinstalling the whole Gitlab package.
hi ftassin,
I have a similar problem.
When you remove /opt/gitlab and reinstalled the whole gitlab package how does it pickup the old gitlab data?
Best Regards,
Kevin.