It looks like the upgrade failed because of a timeout waiting for Grafana to restart. Is there something I can do to retry the upgrade?

Here’s the tail of the upgrade output where the errors occur:

Recipe: gitlab::prometheus
  * execute[reload prometheus] action run
    - execute /opt/gitlab/bin/gitlab-ctl hup prometheus
Recipe: <Dynamically Defined Resource>
  * service[grafana] action restart
    
    ================================================================================
    Error executing action `restart` on resource 'service[grafana]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana ----
    STDOUT: timeout: run: /opt/gitlab/service/grafana: (pid 12957) 51s, got TERM
    STDERR: 
    ---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana ----
    Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana returned 1
    
    Compiled Resource:
    ------------------
    # Declared in 
    
    service("grafana") do
      provider Chef::Provider::Service::Simple
      action [:nothing]
      default_guard_interpreter :default
      service_name "grafana"
      enabled nil
      running nil
      masked nil
      pattern "grafana"
      start_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv start /opt/gitlab/service/grafana"
      stop_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv stop /opt/gitlab/service/grafana"
      status_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv status /opt/gitlab/service/grafana"
      restart_command "/opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana"
      supports {:restart=>true, :reload=>true, :status=>true}
    end
    
    System Info:
    ------------
    chef_version=13.6.4
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client
    

Running handlers:
There was an error running gitlab-ctl reconfigure:

service[grafana] (dynamically defined) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana ----
STDOUT: timeout: run: /opt/gitlab/service/grafana: (pid 12957) 51s, got TERM
STDERR: 
---- End output of /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana ----
Ran /opt/gitlab/embedded/bin/chpst -u root:root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/grafana returned 1

Running handlers complete
Chef Client failed. 78 resources updated in 08 minutes 29 seconds
Ensuring PostgreSQL is updated:Traceback (most recent call last):
	8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
	7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
	6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
	5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
	4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
	3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:64:in `block in load_file'
	2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:105:in `roles'
	1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:64:in `get_node_attributes'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:52:in `parse_json_file': Attributes not found in /opt/gitlab/embedded/nodes/mara.alphasoft.com.json, has reconfigure been run yet? (GitlabCtl::Errors::NodeError)
Ensuring PostgreSQL is updated: NOT OK
Error ensuring PostgreSQL is updated. Please check the logs

I’ve just faced the same issue

Try running gitlab-ctl reconfigure manually. Mine fails altering epics table:

-> 0.0035s-- foreign_keys(“design_management_designs”)
-> 0.0036s-- foreign_keys(“design_management_designs_versions”)
-> 0.0035s
– foreign_keys(“design_management_designs_versions”)
-> 0.0034s
– foreign_keys(“draft_notes”)
-> 0.0035s
– foreign_keys(“draft_notes”)
-> 0.0035s
– foreign_keys(“elasticsearch_indexed_namespaces”)
-> 0.0044s
– foreign_keys(“elasticsearch_indexed_projects”)
-> 0.0039s
– foreign_keys(“epic_issues”)
-> 0.0038s
– foreign_keys(“epic_issues”)
-> 0.0037s
– foreign_keys(“epic_metrics”)
-> 0.0079s
– foreign_keys(“epics”)
-> 0.0037s
– transaction_open?()
-> 0.0000s
– foreign_keys(“epics”)
-> 0.0036s
– execute(“ALTER TABLE epics\nADD CONSTRAINT fk_25b99c1be3\nFOREIGN KEY (parent_id)\nREFERENCES epics (id)\nON DELETE CASCADE\nNOT VALID;\n”)
STDERR:
---- End output of “bash” “/tmp/chef-script20190623-23590-2225s5” -—
Ran “bash” “/tmp/chef-script20190623-23590-2225s5” returned 1

Hmm, different symptoms here. After a restart and successful gitlab-ctl reconfigure, GitLab seems to be running except for alert manager.

gitlab-ctl status says:

down: alertmanager: 0s, normally up, want up; run: log: (pid 1278) 31213s
run: gitaly: (pid 1283) 31213s; run: log: (pid 1266) 31213s
run: gitlab-monitor: (pid 1254) 31213s; run: log: (pid 1238) 31213s
run: gitlab-workhorse: (pid 1263) 31213s; run: log: (pid 1235) 31213s
run: grafana: (pid 1285) 31213s; run: log: (pid 1277) 31213s
run: logrotate: (pid 30308) 2371s; run: log: (pid 1250) 31213s
run: mailroom: (pid 1299) 31213s; run: log: (pid 1288) 31213s
run: nginx: (pid 5319) 18286s; run: log: (pid 1236) 31213s
run: node-exporter: (pid 1260) 31213s; run: log: (pid 1253) 31213s
run: postgres-exporter: (pid 1290) 31213s; run: log: (pid 1265) 31213s
run: postgresql: (pid 1255) 31213s; run: log: (pid 1247) 31213s
run: prometheus: (pid 1282) 31213s; run: log: (pid 1261) 31213s
run: redis: (pid 1284) 31213s; run: log: (pid 1264) 31213s
run: redis-exporter: (pid 1258) 31214s; run: log: (pid 1252) 31214s
run: registry: (pid 1281) 31214s; run: log: (pid 1271) 31214s
run: sidekiq: (pid 1259) 31214s; run: log: (pid 1248) 31214s
run: unicorn: (pid 1251) 31214s; run: log: (pid 1237) 31214s

this happens for me aswel. the database log yells at me with this:

2019-06-24_09:52:13.12611 ERROR: column “parent_id” referenced in foreign key constraint does not exist
2019-06-24_09:52:13.12612 STATEMENT: ALTER TABLE epics
2019-06-24_09:52:13.12612 ADD CONSTRAINT fk_25b99c1be3
2019-06-24_09:52:13.12612 FOREIGN KEY (parent_id)
2019-06-24_09:52:13.12613 REFERENCES epics (id)
2019-06-24_09:52:13.12613 ON DELETE CASCADE
2019-06-24_09:52:13.12613 NOT VALID;

Hello @Aoife @k.vd.rijt Thanks for posting the details about this problem.

Have you opened an issue regarding this at https://gitlab.com/gitlab-org/gitlab-ce/issues/?

no i haven’t yet… I wil do now!

Is it safe to upgrade to the 12.x release?

I managed to get it fixed. What went wrong for me is that yum did the upgrade from 11.4.0 straight to 12.0 and that broke it. You have to be carefull and upgrade to 11.11.3 before you go to 12.0

I wrote what I did to fix it here: https://gitlab.com/gitlab-org/gitlab-ce/issues/63660#note_184696228

It seems all the issues reported are from people that were not on 11.11.3 before they tried to upgrade to 12.0.0, so is it safe to upgrade if we’re on 11.11.3?

For me upgrade failing with same error from GitLab 11.11.3 (e3eeb779d72) to 12.0.0:

– execute(“ALTER TABLE epics\nADD CONSTRAINT fk_25b99c1be3\nFOREIGN KEY (parent_id)\nREFERENCES epics (id)\nON DELETE CASCADE\nNOT VALID;\n”)
STDERR:
---- End output of “bash” “/tmp/chef-script20190623-23590-2225s5” -—
Ran “bash” “/tmp/chef-script20190623-23590-2225s5” returned 1

I have 11.11.3 installed and have the same issue.

Same issue with 11.11.3 to 12.0.1. (but here I had other issues with custom prometheus paths, defined in gitlab.rb)

but no issue on other install from 11.11.1 to 12.0.1

No luck with upgrading to 12.0.1. Still have problems with grafana.

There is an issue at the gitlab bugtracker about this. The bug is caused by some “leftovers” from Enterprise Edition database tables.

The Omnibus install will leave an partial installed gitlab-ce package which installation can be restarted with sudo apt-get upgrade

I fixed it on our machine by manually editing / extending the PostgresSQL data base tables.

sudo gitlab-psql -d gitlabhq_production

ALTER TABLE epics ADD COLUMN parent_id integer;
ALTER TABLE epics ADD COLUMN closed_by_id integer;
ALTER TABLE geo_event_log ADD COLUMN cache_invalidation_event_id bigint;
ALTER TABLE geo_event_log ADD COLUMN reset_checksum_event_id bigint;
\d

( \d (2 chars) to quit the SQL console)

And then restart the upgrade process. Once the 2 tables are corrected installation will succeed.

same issue … trying to upgrade from 11.11.3. Upgrade failed waiting for Grafana to restart…