Unable to fork projects after upgrade from 14.1 to 14.2.1

d.mortimer · August 25, 2021, 1:01pm

Hi,

This morning we upgraded to 14.2.1-ce via RPM repo (CO7) and now we find we are unable to fork any project. I haven’t had any reports of forks being impossible on 14.1 so I’m assuming this is a new issue.
When the fork widget is clicked on a projects home page you get taken to a new page with a Fork project icon and descriptive text but nothing else appears (see image).

This is the same for every project I’ve looked at despite them allowing forks for members of the project. I’ve also created an empty test project and the same behaviour is observed.
This is confirmed as an issue on Safari, Firefox and Chrome on macOS and Firefox on Centos 7.

iwalker · August 25, 2021, 1:48pm

I get this on mine, also 14.2.1 which seems to show that it’s working:

root@gitlab:~# dpkg -l | grep -i gitlab
ii  gitlab-ce                                     14.2.1-ce.0                  amd64        GitLab Community Edition (including NGINX, Postgres, Redis)

mine was upgraded from 14.1.3 to 14.2.0 and then 14.2.1 since I do my upgrades regularly when each new release comes out. Not sure what version of 14.1.x you upgraded from?

Would need a lot more information from you to be able to even find out why you have a problem. Exactly the version you upgraded from, how you did the upgrade, have the migrations finished, were there errors during the upgrade process, check /var/log/gitlab log files for any errors etc.

Maybe you need to clear your browser cache, or check in private/incognito mode to rule out extensions causing you problems. I’m using Firefox on Linux, no issues.

You can try some things like for example:

gitlab-ctl restart
systemctl restart gitlab-runsvdir

or even reboot the server just in case of a sticky process. You can also do:

gitlab-ctl reconfigure
gitlab-ctl restart

as well. This is usually displayed in the output after upgrade in case of weird things happening of what you can try.

d.mortimer · August 25, 2021, 2:12pm

Jul 08 12:17:17 Updated: gitlab-ce-14.0.4-ce.0.el7.x86_64
Aug 09 14:15:05 Updated: gitlab-ce-14.1.2-ce.0.el7.x86_64
Aug 25 12:40:26 Updated: gitlab-ce-14.2.1-ce.0.el7.x86_64

Upgrade was via ‘yum update’
There are no listed background migrations and the update process appeared to go through without error, but did seem to take a fairly long time (ca 5 mins) to get back to an operational state.
Happy to provide logs if you indicate which would be of most use. Logging at /v/l/gl/reconfigure for example:

# Logfile created on 2021-08-25 12:41:04 +0100 by logger.rb/v1.4.2
[2021-08-25T12:41:04+01:00] INFO: Started Chef Infra Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded
  One version per cookbook

[2021-08-25T12:41:05+01:00] INFO: *** Chef Infra Client 15.14.0 ***
[2021-08-25T12:41:05+01:00] INFO: Platform: x86_64-linux
[2021-08-25T12:41:05+01:00] INFO: Chef-client pid: 11625
[2021-08-25T12:41:32+01:00] INFO: Setting the run_list to ["recipe[gitlab::config]", "recipe[postgresql::bin]"] from CLI options
[2021-08-25T12:41:32+01:00] INFO: Run List is [recipe[gitlab::config], recipe[postgresql::bin]]
[2021-08-25T12:41:32+01:00] INFO: Run List expands to [gitlab::config, postgresql::bin]
[2021-08-25T12:41:32+01:00] INFO: Starting Chef Infra Client Run for *
[2021-08-25T12:41:32+01:00] INFO: Running start handlers
[2021-08-25T12:41:32+01:00] INFO: Start handlers complete.
[2021-08-25T12:41:33+01:00] INFO: Loading cookbooks [gitlab@0.0.1, postgresql@0.1.0, package@0.1.0, logrotate@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.3, acme@4.1.3, crond@0.1.0]
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/attributes/default.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/libraries/helpers/gitlab_rails_env_helper.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/libraries/postgresql.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/libraries/patroni.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/libraries/gitlab_rails.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-workhorse.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/recipes/gitlab-rails.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/database_geo.yml.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/database.yml.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-pages-http.conf.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-registry-http.conf.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/puma.rb.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/nginx-gitlab-mattermost-http.conf.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/gitlab/templates/default/workhorse-config.toml.erb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/postgresql/recipes/bin.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/package/libraries/config/gitlab.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/package/libraries/helpers/gitlab_cluster_helper.rb in the cache.
[2021-08-25T12:41:33+01:00] INFO: Storing updated cookbooks/package/libraries/helpers/settings_helper.rb in the cache.
[2021-08-25T12:41:36+01:00] INFO: Chef Infra Client Run complete in 3.513921064 seconds
[2021-08-25T12:41:36+01:00] INFO: Running report handlers
[2021-08-25T12:41:36+01:00] INFO: Report handlers complete
[2021-08-25T12:41:36+01:00] WARN: This release of Chef Infra Client became end of life (EOL) on May 1st 2021. Please update to a supported release to receive new features, bug fixes, and security updates.

So other than the Chef warning, the reconfiguration appears to run fine, and has been run again just in case with no change, but different log output:

# Logfile created on 2021-08-25 12:41:49 +0100 by logger.rb/v1.4.2
[2021-08-25T12:41:49+01:00] INFO: Started Chef Infra Zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded
  One version per cookbook

[2021-08-25T12:41:49+01:00] INFO: *** Chef Infra Client 15.14.0 ***
[2021-08-25T12:41:49+01:00] INFO: Platform: x86_64-linux
[2021-08-25T12:41:49+01:00] INFO: Chef-client pid: 11903
[2021-08-25T12:42:16+01:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2021-08-25T12:42:17+01:00] INFO: Run List is [recipe[gitlab]]
[2021-08-25T12:42:17+01:00] INFO: Run List expands to [gitlab]
[2021-08-25T12:42:17+01:00] INFO: Starting Chef Infra Client Run for *
[2021-08-25T12:42:17+01:00] INFO: Running start handlers
[2021-08-25T12:42:17+01:00] INFO: Start handlers complete.
[2021-08-25T12:42:17+01:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, logrotate@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, gitlab-kas@0.1.0, gitlab-pages@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@5.1.3, acme@4.1.3, crond@0.1.0]
[2021-08-25T12:42:20+01:00] WARN: Selected systemd because /run/systemd/system/ exists
[2021-08-25T12:42:22+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/database.yml.chef-20210825124222.464396
[2021-08-25T12:42:22+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/database.yml.chef-20200501090145.348852
[2021-08-25T12:42:22+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/database.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/database.yml
[2021-08-25T12:42:22+01:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2021-08-25T12:42:22+01:00] WARN: only_if block for templatesymlink[Create a gitlab_kas_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2021-08-25T12:42:22+01:00] INFO: link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_kas_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_kas_secret] created
[2021-08-25T12:42:22+01:00] INFO: file[/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql] owner changed to 1996
[2021-08-25T12:42:23+01:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/VERSION.chef-20210825124223.020171
[2021-08-25T12:42:23+01:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/VERSION.chef-20210608090938.680217
[2021-08-25T12:42:23+01:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] updated file contents /var/opt/gitlab/gitlab-rails/VERSION
[2021-08-25T12:42:23+01:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/REVISION.chef-20210825124223.062564
[2021-08-25T12:42:23+01:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/REVISION.chef-20210608090938.725447
[2021-08-25T12:42:23+01:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] updated file contents /var/opt/gitlab/gitlab-rails/REVISION
[2021-08-25T12:42:26+01:00] INFO: execute[Guard resource] ran successfully
[2021-08-25T12:42:28+01:00] INFO: execute[Guard resource] ran successfully
[2021-08-25T12:42:31+01:00] INFO: execute[Guard resource] ran successfully
[2021-08-25T12:42:31+01:00] INFO: ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] called
[2021-08-25T12:42:32+01:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitaly/VERSION.chef-20210825124232.141576
[2021-08-25T12:42:32+01:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitaly/VERSION.chef-20210608090947.716379
[2021-08-25T12:42:32+01:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] updated file contents /var/opt/gitlab/gitaly/VERSION
[2021-08-25T12:42:53+01:00] INFO: bash[migrate gitlab-rails database] ran successfully
[2021-08-25T12:42:53+01:00] INFO: rails_migration[gitlab-rails] sending run action to execute[clear the gitlab-rails cache] (immediate)
[2021-08-25T12:43:10+01:00] INFO: execute[clear the gitlab-rails cache] ran successfully
[2021-08-25T12:43:10+01:00] INFO: rails_migration[gitlab-rails] sending run action to ruby_block[check remote PG version] (immediate)
[2021-08-25T12:43:10+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/puma.rb] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/puma.rb.chef-20210825124310.933598
[2021-08-25T12:43:10+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/puma.rb] updated file contents /var/opt/gitlab/gitlab-rails/etc/puma.rb
[2021-08-25T12:43:11+01:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-workhorse/VERSION.chef-20210825124311.422205
[2021-08-25T12:43:11+01:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-workhorse/VERSION.chef-20210608091030.212548
[2021-08-25T12:43:11+01:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] updated file contents /var/opt/gitlab/gitlab-workhorse/VERSION
[2021-08-25T12:43:11+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/gitlab-http.conf.chef-20210825124311.509386
[2021-08-25T12:43:11+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/gitlab-http.conf.chef-20200922154735.563791
[2021-08-25T12:43:11+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] updated file contents /var/opt/gitlab/nginx/conf/gitlab-http.conf
[2021-08-25T12:43:11+01:00] INFO: ruby_block[re-populate GitLab Pages configuration options] called
[2021-08-25T12:43:11+01:00] INFO: file[/var/opt/gitlab/gitlab-pages/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-pages/VERSION.chef-20210825124311.787965
[2021-08-25T12:43:11+01:00] INFO: file[/var/opt/gitlab/gitlab-pages/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-pages/VERSION.chef-20210419151731.783558
[2021-08-25T12:43:11+01:00] INFO: file[/var/opt/gitlab/gitlab-pages/VERSION] updated file contents /var/opt/gitlab/gitlab-pages/VERSION
[2021-08-25T12:43:12+01:00] INFO: ruby_block[populate Grafana configuration options] called
[2021-08-25T12:43:12+01:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed)
[2021-08-25T12:43:12+01:00] INFO: runit_service[gitaly] signalled (HUP)
[2021-08-25T12:43:12+01:00] INFO: runit_service[gitaly] sent hup
[2021-08-25T12:43:12+01:00] INFO: version_file[Create version file for Workhorse] sending restart action to runit_service[gitlab-workhorse] (delayed)
[2021-08-25T12:43:13+01:00] INFO: version_file[Create version file for Gitlab Pages] sending restart action to runit_service[gitlab-pages] (delayed)
[2021-08-25T12:43:13+01:00] INFO: Chef Infra Client Run complete in 56.914807456 seconds
[2021-08-25T12:43:13+01:00] INFO: Running report handlers
[2021-08-25T12:43:13+01:00] INFO: Report handlers complete
[2021-08-25T12:43:13+01:00] WARN: This release of Chef Infra Client became end of life (EOL) on May 1st 2021. Please update to a supported release to receive new features, bug fixes, and security updates.

Here’s production.log for the click on the Fork button:

Started GET "/duncan/test/-/forks/new" for 163.1.84.240 at 2021-08-25 15:04:26 +0100
Processing by Projects::ForksController#new as HTML
  Parameters: {"namespace_id"=>"duncan", "project_id"=>"test"}
  Rendered layout layouts/project.html.haml (Duration: 65.9ms | Allocations: 48041)
Completed 200 OK in 94ms (Views: 63.6ms | ActiveRecord: 7.0ms | Elasticsearch: 0.0ms | Allocations: 61580)
Started GET "/namespace/test/-/forks/new.json" for X.Y.Z.A at 2021-08-25 15:04:26 +0100
Processing by Projects::ForksController#new as JSON
  Parameters: {"namespace_id"=>"namespace", "project_id"=>"test"}
Completed 200 OK in 70ms (Views: 2.7ms | ActiveRecord: 9.4ms | Elasticsearch: 0.0ms | Allocations: 34961)

Nothing appears in exceptions_json.log for this event.
Migration log for the upgrade doesn’t show errors:

# zcat migrations.log-2021-08-25.gz | grep -i failed
[root@git gitlab-rails]# zcat migrations.log-2021-08-25.gz | grep -i error
{"severity":"INFO","time":"2021-08-25T11:42:51.103Z","correlation_id":null,"method":"with_lock_retries","class":"CreateErrorTrackingClientKeys","message":"Lock timeout is set","current_iteration":1,"lock_timeout_in_ms":100}
{"severity":"INFO","time":"2021-08-25T11:42:51.132Z","correlation_id":null,"method":"with_lock_retries","class":"CreateErrorTrackingClientKeys","message":"Migration finished","current_iteration":1,"lock_timeout_in_ms":100}

d.mortimer · August 25, 2021, 3:15pm

We have a second GitLab CE install still on 14.1.2 which behaves as expected. They don’t share the same content/update history, so this is probably an irrelevant comparison. Will be holding off on updating this one for a while.

d.mortimer · August 27, 2021, 8:32am

Hi,
I’ve tried rebooting the server and no change.

d.mortimer · September 1, 2021, 3:32pm

The only error message I can locate in any logs are in the postgresql logs and occur on each run of gitlab-ctl reconfigure:

2021-09-01_15:29:34.13575 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2021-09-01_15:29:34.13577 FATAL: Peer authentication failed for user “gitlab”
2021-09-01_15:29:34.13578 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2021-09-01_15:29:34.19539 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2021-09-01_15:29:34.19541 FATAL: Peer authentication failed for user “gitlab”
2021-09-01_15:29:34.19542 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”
2021-09-01_15:29:34.24712 LOG: no match in usermap “gitlab” for user “gitlab” authenticated as “root”
2021-09-01_15:29:34.24715 FATAL: Peer authentication failed for user “gitlab”
2021-09-01_15:29:34.24715 DETAIL: Connection matched pg_hba.conf line 70: “local all all peer map=gitlab”

I updated to 14.2.2 today and no change.

d.mortimer · September 7, 2021, 2:51pm

We have been able to access the old-style fork screen by

irb(main):002:0> Feature.disable(:fork_project_form)
=> true