Unable to pull or push after upgrading to 12.10 and 12.10.1

Unable to pull or push after upgrading to 12.10 and 12.10.1

After upgrading to 12.10 we cannot push nor pull in our repositories, we receive messages like this:

git pull
warning: redirecting to https://xxxxxx.com/yyy.git
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500
fatal: expected flush after ref listing

We have a self-managed instance, it’s happening since the 12.10 upgrade. Some hours after the upgrade we upgraded to 12.10.1 because the server found an update. The web application works as expected, I can login, list repositories, list commits, etc

This is the result of gitlab-rake check:

gitlab-rake gitlab:check

Checking GitLab subtasks …

Checking GitLab Shell …

GitLab Shell: … GitLab Shell version >= 12.2.0 ? … OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell … Finished

Checking Gitaly …

Gitaly: … default … OK

Checking Gitaly … Finished

Checking Sidekiq …

Sidekiq: … Running? … yes
Number of Sidekiq processes … 1

Checking Sidekiq … Finished

Checking Incoming Email …

Incoming Email: … Reply by email is disabled in config/gitlab.yml

Checking Incoming Email … Finished

Checking LDAP …

LDAP: … LDAP is disabled in config/gitlab.yml

Checking LDAP … Finished

Checking GitLab App …

Git configured correctly? … yes
Database config exists? … yes
All migrations up? … yes
Database contains orphaned GroupMembers? … no
GitLab config exists? … yes
GitLab config up to date? … yes
Log directory writable? … yes
Tmp directory writable? … yes
Uploads directory exists? … yes
Uploads directory has correct permissions? … yes
Uploads directory tmp has correct permissions? … yes
Init script exists? … skipped (omnibus-gitlab has no init script)
Init script up-to-date? … skipped (omnibus-gitlab has no init script)
Projects have namespace: …

HERE there is a list of our repositories and all have “YES” after the name.

Redis version >= 4.0.0? … yes
Ruby version >= 2.5.3 ? … yes (2.6.5)
Git version >= 2.22.0 ? … yes (2.26.2)
Git user has default SSH configuration? … yes
Active users: … XXXXXX
Is authorized keys file accessible? … yes

Checking GitLab App … Finished

Checking GitLab subtasks … Finished

Thank you so much for any advice, Miguel

I’ve investigated a little bit in the logs and found this:

1. Completed 500 Internal Server Error in 107ms (ActiveRecord: 7.9ms | Elasticsearch: 0.0ms | Allocations: 7958)
2. ActiveRecord::StatementInvalid (PG::SyntaxError: ERROR: error de sintaxis en o cerca de «ON»
3. LINE 3: ON CONFLICT (project_id, date) DO UPDATE SET fetch_count = p…
4. ^
5. ):
6. app/services/projects/fetch_statistics_increment_service.rb:18:in `execute’
7. app/controllers/repositories/git_http_controller.rb:85:in `update_fetch_statistics’
8. app/controllers/repositories/git_http_controller.rb:26:in `git_upload_pack’
9. app/controllers/application_controller.rb:479:in `set_current_admin’
10. lib/gitlab/i18n.rb:55:in `with_locale’
11. lib/gitlab/i18n.rb:61:in `with_user_locale’
12. app/controllers/application_controller.rb:464:in `set_locale’
13. lib/gitlab/error_tracking.rb:48:in `with_context’
14. app/controllers/application_controller.rb:559:in `sentry_context’
15. lib/gitlab/application_context.rb:52:in `block in use’
16. lib/gitlab/application_context.rb:52:in `use’
17. lib/gitlab/application_context.rb:20:in `with_context’
18. app/controllers/application_controller.rb:455:in `set_current_context’
19. lib/gitlab/request_profiler/middleware.rb:17:in `call’
20. lib/gitlab/middleware/go.rb:20:in `call’
21. lib/gitlab/etag_caching/middleware.rb:13:in `call’
22. lib/gitlab/middleware/multipart.rb:122:in `call’
23. lib/gitlab/middleware/read_only/controller.rb:53:in `call’
24. lib/gitlab/middleware/read_only.rb:18:in `call’
25. lib/gitlab/middleware/same_site_cookies.rb:27:in `call’
26. lib/gitlab/middleware/basic_health_check.rb:25:in `call’
27. lib/gitlab/middleware/request_context.rb:23:in `call’
28. config/initializers/fix_local_cache_middleware.rb:9:in `call’
29. lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call’
30. lib/gitlab/middleware/release_env.rb:12:in `call’

Looks like 12.10 requires postgresql 11 and we have 9.4, is it possible to go back? Can I work on the query and resolve the problem?

Thanks!

More info, we can push and pull again after commenting these lines of code in the file:
/opt/gitlab/embedded/service/gitlab-rails/app/services/projects/fetch_statistics_increment_service.rb

image829×143 4.46 KB

Looks like the error is related to that query, I did not upgrade to Postgresql 11.

Any advice for the future maintenance of the product?

Thanks again for the hard work.

PostgreSQL 9.4 is not supported. Either upgrade to PostgreSQL 9.6 or 11.

I have the same problem with postgres 11.7