Upgrade to 14.5.0 caused a lot of issues: cannot push changes, cannot view projects

GitLab 14.5.0 Upgrade Issue

I updated to GitLab 14.5.0 and I have had a lot of issues with GitLab. I have a section that describes these issues below, a section that lists some things I’ve noticed that may be helpful, and another section that lists the things I’ve tried. My gut tells me this is an issue with the PostgreSQL database. I searched and found a lot of things online indicating others were having similar issues, however I didn’t find any resolutions where I looked. Thank you for any insight you can provide.

Issues

  1. In the web GUI, many repositories display error code 500 when visited.
  2. In the web GUI, many repositories mention, “An error occurred while fetching folder content.”
  3. Cannot push to any of the repositories created before 14.5.0 (but can for ones that were created that are newer). See "Detail: git push" section below.

Detail: git push

> git push
time="2021-11-25T08:48:53-07:00" level=info msg="SSL_CERT_DIR is configured" ssl_cert_dir=/opt/gitlab/embedded/ssl/certs/
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Delta compression using up to 12 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 790 bytes | 790.00 KiB/s, done.
Total 8 (delta 6), reused 0 (delta 0), pack-reused 0
remote: 
remote: ========================================================================
remote: 
remote: The git server, Gitaly, is not available at this time. Please contact your administrator.
remote: 
remote: ========================================================================
remote: 
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

Observations

GitLab

# gitlab-rake gitlab:env:info

System information
System:         Raspbian 10
Current User:   git
Using RVM:      no
Ruby Version:   2.7.4p191
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.0.16
Git Version:    2.33.1.
Sidekiq Version:6.2.2
Go Version:     go1.11.6 linux/arm

GitLab information
Version:        14.5.0
Revision:       f8796c0836e
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://my.url.org
HTTP Clone URL: https://my.url.org/some-group/some-project.git
SSH Clone URL:  git@my.url.org:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        13.22.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

System

I am currently running Raspberry Pi OS Buster 10 (5.10.63-v7l+) on a Raspberry Pi 4B w/ 8 GB RAM.

/var/log/gitlab/postgresql/current

FATAL:  terminating connection due to administrator command
LOG:  using stale statistics instead of current ones because stats collector is not responding

Things Tried

  1. Reconfigured GitLab CE (sudo gitlab-ctl reconfigure); this appeared to be successful (it didn’t fix the issue… just the command was successful).
  2. Reinstalled GitLab CE (sudo apt-get --reinstall install gitlab-ce); this reinstall appeared to be successful (it didn’t fix the issue… just the command was successful).
  3. Downgraded back go GitLab CE 14.4.2. This was successful with some resistance. Specifically, the initial gitlab-backup didn’t work (returned exit code 1). I re-ran it and then it seemed to work (at least it returned exit code 0). This is how I am currently operating.
  4. I noticed there were errors within /var/log/gitlab/postgresql/current that made it seem like user gitlab needed a masking to root (I forget the exact error and I didn’t write it down). The sleuthing led me to edit /var/opt/gitlab/postgresql/data/pg_ident.conf and add gitlab root gitlab at the bottom. I believe the error message was no match in usermap "gitlab" for user "gitlab" authenticated as "root".

Initial Unsuccessful Restore to 14.4.2

# gitlab-backup restore BACKUP=1637591959_2021_11_22_14.4.2
Restoring PostgreSQL database gitlabhq_production ... ERROR:  must be owner of extension pg_trgm
ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension btree_gist
ERROR:  must be owner of extension pg_trgm
<snip>
ALTER TABLE                                                                                  
[DONE]                                                                                       
2021-11-25 09:31:16 -0700 -- done                                                            
rake aborted!                                                                                
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column application_settings.con$
ent_validation_endpoint_url does not exist                                                   
LINE 1: ...application_settings"."suggest_pipeline_enabled", "applicati...                   
                                                             ^                               
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_prox$
.rb:103:in `block in read_using_load_balancer'                                               
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.r$
:53:in `read'                                                                                
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:102:in `read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:47:in `select_all'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:19:in `
current_without_cache'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:63:in `
rescue in current'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:51:in `
current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:60:in `uncached_appl
ication_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_applic
ation_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in curr
ent_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_appli
cation_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missin
g'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:97:in `ensure_write_to_
authorized_keys_is_enabled'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:72:in `setup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:48:in `block (3 levels)
 in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (3 levels
) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::UndefinedColumn: ERROR:  column application_settings.content_validation_endpoint_url does
 not exist
LINE 1: ...application_settings"."suggest_pipeline_enabled", "applicati...
                                                             ^
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:103:in `block in read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb
:53:in `read'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:102:in `read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:47:in `select_all'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:19:in `
current_without_cache'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:63:in `
rescue in current'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:51:in `
current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:60:in `uncached_appl
ication_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_applic
ation_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in curr
ent_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_appli
cation_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missin
g'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:97:in `ensure_[53/1959]
authorized_keys_is_enabled'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:72:in `setup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:48:in `block (3 levels)
 in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (3 levels
) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column application_settings.cont
ent_validation_endpoint_url does not exist
LINE 1: ...application_settings"."suggest_pipeline_enabled", "applicati...
                                                             ^
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:103:in `block in read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb
:53:in `read'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:102:in `read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:47:in `select_all'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:19:in `
current_without_cache'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:55:in `
current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:60:in `uncached_appl
ication_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_applic
ation_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in curr
ent_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_appli
cation_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missin
g'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:97:in `ensure_write_to_
authorized_keys_is_enabled'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:72:in `setup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:48:in `block (3 levels)
 in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (3 levels
) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::UndefinedColumn: ERROR:  column application_settings.content_validation_endpoint_url does
 not exist
LINE 1: ...application_settings"."suggest_pipeline_enabled", "applicati...
                                                             ^
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:103:in `block in read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb
:53:in `read'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:102:in `read_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy
.rb:47:in `select_all'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:19:in `
current_without_cache'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cacheable_attributes.rb:55:in `
current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:60:in `uncached_appl
ication_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:42:in `ensure_applic
ation_settings!'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `block in curr
ent_application_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/null_request_store.rb:34:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/safe_request_store.rb:12:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:15:in `current_appli
cation_settings'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/current_settings.rb:32:in `method_missin
g'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:97:in `ensure_write_to_
authorized_keys_is_enabled'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:72:in `setup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/shell.rake:48:in `block (3 levels)
 in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:87:in `block (3 levels
) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:shell:setup
(See full trace by running task with --trace)
Transfering ownership of /var/opt/gitlab/gitlab-rails/shared/registry to registry
# echo $?
1

Kind regards,
Anthony

1 Like

Thanks for the report

In the 14.5 release gitaly is currently broken on the raspberry pi: Accessing the repository home results in a 500 error (#3938) · Issues · GitLab.org / gitaly · GitLab

we are working to have it fixed in a patch, hopefully next week

3 Likes

Hi,

SSL_CERT_DIR in the git push log is a side effect from a regression in 14.5.0. This is discussed in SSL_CERT_DIR info message on git pull/push after update to 14.5 and tracked in this issue with a workaround.

The issue linked by @twk3 has a workaround with a binary too, thanks @stanhu :slight_smile:

Since you were not able to downgrade, maybe look into applying the workarounds and let the patch release upgrade override the files later.

Cheers,
Michael

1 Like

Thanks for commenting and sharing this information. It is very helpful.

Hello Michael:

Thank you for your comment. The first time I tried the downgrade command it didn’t work (and showed the error message above). Then, I tried it again, and it seemed to work (and returned exit code 0). So, I am currently on 14.4.2. Is there something I should do/run to ensure that the downgrade did work properly? Also, there may have been a few commits that were made (e.g., sleuthing experiments) that do not show up after the downgrade. I assume that data is buried in the git repository structure that GitLab maintains, however it is not seen. Will those things be cleaned up by some process or are they gonna tag along forever? For example, I created a new group and a new repo to see if things would work (and things worked a lot better for those new items than the old ones). However, that new group and new repo do not show up after the downgrade. Any thoughts you have on this would be greatly appreciated. Thank you for your help with this.

Kind regards,
Anthony

Hi,

That’s good to know, then I recommend to stay there until future bugfix releases are out.

If there was no backup/restore which would have included these in the data sets, I would say they do not exist anymore. I would guess that the downgrade used an older backup created from before the upgrade procedure.

For verifying the health of a GitLab instance, I use a few things from a monitoring perspective. Checking the health status, the gitlab:check rails command, and the database migrations status.

Cheers,
Michael

Thanks @iqgrande for reporting this. My local GitLab CE broke as well on RasPi, I was able to downgrade to previous working GitLab v14.4.2. Also thank you @dnsmichi and GitLab team for working on the fix, really appreciate it.

1 Like

I am seeing exactly the same symptoms on my CE instance on a Pi after upgrading to 14.5.0. I just tried a restore from backup for 14.5.0 with no results, but these posts has saved me the headache of reverting to 14.4 - I’ll wait for the fix. Many thanks to @twk3 and @dnsmichi for the clear guidance.

1 Like

14.5.1 for Raspberry Pi2 has been published: gitlab/raspberry-pi2 - Packages · packages.gitlab.com

3 Likes

This fix worked fine. Thank you for the help.

1 Like

I just upgraded from 14.4.2 to 14.5.1 and all appears to be working for me, too. Thank you @stanhu and the GitLab team for addressing this!

2 Likes