Greetings
I have tried several times to do what the installer suggested me, to downgrade to 11.11.3-EE and then to Upgrade to 12.x-CE so I did this process at least 5 times, sorry @gitlab, just this is a damn mad loophole. Here is first LOG from 11.x:
Gitlab-11
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] == 20181204031328 CreateApprovalRules: migrating ==============================
-- adapter_name()
-> 0.0000s
-- adapter_name()
-> 0.0000s
-- create_table(:approval_project_rules, {:id=>:bigserial})
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
: CREATE TABLE "approval_project_rules" ("id" bigserial NOT NULL PRIMARY KEY, "created_at" timestamptz NOT NULL, "updated_at" timestamptz NOT NULL, "project_id" integer NOT NULL, "approvals_required" smallint DEFAULT 0 NOT NULL, "name" character varying NOT NULL, CONSTRAINT "fk_rails_5fb4dd100b"
FOREIGN KEY ("project_id")
REFERENCES "projects" ("id")
ON DELETE CASCADE)
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
: CREATE TABLE "approval_project_rules" ("id" bigserial NOT NULL PRIMARY KEY, "created_at" timestamptz NOT NULL, "updated_at" timestamptz NOT NULL, "project_id" integer NOT NULL, "approvals_required" smallint DEFAULT 0 NOT NULL, "name" character varying NOT NULL, CONSTRAINT "fk_rails_5fb4dd100b"
FOREIGN KEY ("project_id")
REFERENCES "projects" ("id")
ON DELETE CASCADE)
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20190627-1729-au6rs2" ----
STDOUT: == 20181204031328 CreateApprovalRules: migrating ==============================
-- adapter_name()
-> 0.0000s
-- adapter_name()
-> 0.0000s
-- create_table(:approval_project_rules, {:id=>:bigserial})
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
: CREATE TABLE "approval_project_rules" ("id" bigserial NOT NULL PRIMARY KEY, "created_at" timestamptz NOT NULL, "updated_at" timestamptz NOT NULL, "project_id" integer NOT NULL, "approvals_required" smallint DEFAULT 0 NOT NULL, "name" character varying NOT NULL, CONSTRAINT "fk_rails_5fb4dd100b"
FOREIGN KEY ("project_id")
REFERENCES "projects" ("id")
ON DELETE CASCADE)
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
: CREATE TABLE "approval_project_rules" ("id" bigserial NOT NULL PRIMARY KEY, "created_at" timestamptz NOT NULL, "updated_at" timestamptz NOT NULL, "project_id" integer NOT NULL, "approvals_required" smallint DEFAULT 0 NOT NULL, "name" character varying NOT NULL, CONSTRAINT "fk_rails_5fb4dd100b"
FOREIGN KEY ("project_id")
REFERENCES "projects" ("id")
ON DELETE CASCADE)
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20190627-1729-au6rs2" ----
Ran "bash" "/tmp/chef-script20190627-1729-au6rs2" returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
53: bash "migrate gitlab-rails database" do
54: code <<-EOH
55: set -e
56: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
57: umask 077
58: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
59: STATUS=${PIPESTATUS[0]}
60: echo $STATUS > #{db_migrate_status_file}
61: exit $STATUS
62: EOH
63: environment env_variables unless env_variables.empty?
64: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
65: dependent_services.each do |svc|
66: notifies :restart, svc, :immediately
67: end
68: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
69: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
70: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:53:in `from_file'
bash("migrate gitlab-rails database") do
action [:run]
default_guard_interpreter :default
command nil
backup 5
returns 0
user nil
interpreter "bash"
declared_type :bash
cookbook_name "gitlab"
recipe_name "database_migrations"
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8c5c506955e\n exit $STATUS\n"
domain nil
not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8c5c506955e) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-8c5c506955e | grep -Fx 0)"
only_if { #code block }
end
System Info:
------------
chef_version=13.6.4
platform=ubuntu
platform_version=18.04
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 53) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20190627-1729-au6rs2" ----
STDOUT: == 20181204031328 CreateApprovalRules: migrating ==============================
-- adapter_name()
-> 0.0000s
-- adapter_name()
-> 0.0000s
-- create_table(:approval_project_rules, {:id=>:bigserial})
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
: CREATE TABLE "approval_project_rules" ("id" bigserial NOT NULL PRIMARY KEY, "created_at" timestamptz NOT NULL, "updated_at" timestamptz NOT NULL, "project_id" integer NOT NULL, "approvals_required" smallint DEFAULT 0 NOT NULL, "name" character varying NOT NULL, CONSTRAINT "fk_rails_5fb4dd100b"
FOREIGN KEY ("project_id")
REFERENCES "projects" ("id")
ON DELETE CASCADE)
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
: CREATE TABLE "approval_project_rules" ("id" bigserial NOT NULL PRIMARY KEY, "created_at" timestamptz NOT NULL, "updated_at" timestamptz NOT NULL, "project_id" integer NOT NULL, "approvals_required" smallint DEFAULT 0 NOT NULL, "name" character varying NOT NULL, CONSTRAINT "fk_rails_5fb4dd100b"
FOREIGN KEY ("project_id")
REFERENCES "projects" ("id")
ON DELETE CASCADE)
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::DuplicateTable: ERROR: relation "approval_project_rules" already exists
/opt/gitlab/embedded/service/gitlab-rails/ee/db/migrate/20181204031328_create_approval_rules.rb:9:in `change'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:56:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20190627-1729-au6rs2" ----
Ran "bash" "/tmp/chef-script20190627-1729-au6rs2" returned 1
Running handlers complete
Chef Client failed. 3 resources updated in 33 seconds
----
root@gitlab:/home/gitlab# gitlab-ctl status
run: alertmanager: (pid 681) 290s; run: log: (pid 1176) 2282s
run: gitaly: (pid 696) 289s; run: log: (pid 32565) 448s
run: gitlab-monitor: (pid 718) 289s; run: log: (pid 1171) 2282s
run: gitlab-workhorse: (pid 738) 289s; run: log: (pid 1154) 2282s
run: logrotate: (pid 748) 288s; run: log: (pid 1174) 2282s
finish: mailroom: (pid 2684) 12s; run: log: (pid 1186) 2282s
run: nginx: (pid 845) 283s; run: log: (pid 1177) 2282s
run: node-exporter: (pid 853) 282s; run: log: (pid 1175) 2282s
run: postgres-exporter: (pid 861) 282s; run: log: (pid 1184) 2282s
run: postgresql: (pid 871) 281s; run: log: (pid 1183) 2282s
down: prometheus: 1s, normally up, want up; run: log: (pid 1155) 2282s
run: redis: (pid 992) 250s; run: log: (pid 1169) 2282s
run: redis-exporter: (pid 999) 250s; run: log: (pid 1187) 2282s
run: registry: (pid 1008) 250s; run: log: (pid 1179) 2282s
run: sidekiq: (pid 1094) 241s; run: log: (pid 1197) 2282s
run: unicorn: (pid 1042) 248s; run: log: (pid 1173) 2282s
root@gitlab:/home/gitlab# gitlab-ctl restart
ok: run: alertmanager: (pid 2738) 0s
ok: run: gitaly: (pid 2767) 0s
ok: run: gitlab-monitor: (pid 2791) 1s
ok: run: gitlab-workhorse: (pid 2796) 0s
ok: run: logrotate: (pid 2823) 0s
ok: run: mailroom: (pid 2919) 0s
ok: run: nginx: (pid 2922) 1s
ok: run: node-exporter: (pid 2931) 0s
ok: run: postgres-exporter: (pid 2938) 1s
ok: run: postgresql: (pid 2950) 0s
ok: run: prometheus: (pid 3009) 0s
ok: run: redis: (pid 3011) 0s
ok: run: redis-exporter: (pid 3019) 0s
ok: run: registry: (pid 3026) 1s
ok: run: sidekiq: (pid 3042) 1s
ok: run: unicorn: (pid 3055) 0s
root@gitlab:/home/gitlab# gitlab-ctl status
run: alertmanager: (pid 2738) 78s; run: log: (pid 1176) 2368s
run: gitaly: (pid 2767) 77s; run: log: (pid 32565) 534s
run: gitlab-monitor: (pid 2791) 77s; run: log: (pid 1171) 2368s
run: gitlab-workhorse: (pid 2796) 76s; run: log: (pid 1154) 2368s
run: logrotate: (pid 2823) 76s; run: log: (pid 1174) 2368s
run: mailroom: (pid 3140) 5s; run: log: (pid 1186) 2368s
run: nginx: (pid 2922) 71s; run: log: (pid 1177) 2368s
run: node-exporter: (pid 2931) 71s; run: log: (pid 1175) 2369s
run: postgres-exporter: (pid 2938) 71s; run: log: (pid 1184) 2369s
run: postgresql: (pid 2950) 70s; run: log: (pid 1183) 2369s
down: prometheus: 1s, normally up, want up; run: log: (pid 1155) 2369s
run: redis: (pid 3011) 48s; run: log: (pid 1169) 2369s
run: redis-exporter: (pid 3019) 48s; run: log: (pid 1187) 2369s
run: registry: (pid 3026) 48s; run: log: (pid 1179) 2369s
run: sidekiq: (pid 3042) 46s; run: log: (pid 1197) 2369s
run: unicorn: (pid 3055) 44s; run: log: (pid 1173) 2369s
root@gitlab:/home/gitlab# gitlab-ctl restart
------
Here is the second one from 12.x-EE
root@gitlab:/etc/gitlab# apt install gitlab-ee
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
gitlab-ee
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/724 MB of archives.
After this operation, 29.8 MB disk space will be freed.
(Reading database ... 224684 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_12.0.2-ee.0_amd64.deb ...
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab.uhlhost.net.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_12.0.2-ee.0_amd64.deb (--unpack):
new gitlab-ee package pre-installation script subprocess returned error exit status 1
/opt/gitlab/embedded/nodes/gitlab.uhlhost.net.json
gitlab: GitLab now ships with a newer version of PostgreSQL (10.7), but it is not yet
gitlab: enabled by default. To upgrade, RUN THE FOLLOWING COMMAND:
sudo gitlab-ctl pg-upgrade
gitlab: Note: This command does not support Geo instances yet. So we don't
gitlab: recommend running this command on Geo nodes. It will be supported
gitlab: in GitLab 12.0.
gitlab: For more details, please see:
gitlab: https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server
gitlab:
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ee_12.0.2-ee.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@gitlab:/etc/gitlab# /opt/gitlab/embedded/nodes/gitlab.uhlhost.net.json
bash: /opt/gitlab/embedded/nodes/gitlab.uhlhost.net.json: Permission denied
root@gitlab:/etc/gitlab# rm -rf /opt/gitlab/embedded/nodes/gitlab.uhlhost.net.json
root@gitlab:/etc/gitlab# apt install gitlab-ee
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
gitlab-ee
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/724 MB of archives.
After this operation, 29.8 MB disk space will be freed.
(Reading database ... 224684 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_12.0.2-ee.0_amd64.deb ...
JSON file with existing configuration not found inside /opt/gitlab/embedded/nodes.
Skipping config check.
Unpacking gitlab-ee (12.0.2-ee.0) over (11.11.3-ee.0) ...
Setting up gitlab-ee (12.0.2-ee.0) ...
Checking PostgreSQL executables:Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab::config", "postgresql::bin"]
Synchronizing Cookbooks:
- gitlab (0.0.1)
- postgresql (0.1.0)
- redis (0.1.0)
- package (0.1.0)
- consul (0.1.0)
- registry (0.1.0)
- letsencrypt (0.1.0)
- mattermost (0.1.0)
- nginx (0.1.0)
- runit (4.3.0)
- gitaly (0.1.0)
- acme (3.1.0)
- crond (0.1.0)
- compat_resource (12.19.1)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 1 resources
Recipe: postgresql::bin
* ruby_block[Link postgresql bin files to the correct version] action run
- execute the ruby block Link postgresql bin files to the correct version
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 06 seconds
Checking PostgreSQL executables: OK
Found /etc/gitlab/skip-auto-reconfigure, exiting...
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / __/ / __/ / / __ `/ __ \
/ /_/ / / /_/ /___/ /_/ / /_/ /
\____/_/\__/_____/\__,_/_.___/
Upgrade complete! If your GitLab server is misbehaving try running
sudo gitlab-ctl restart
before anything else.
If you need to roll back to the previous version you can use the database
backup made during the upgrade (scroll up for the filename).
root@gitlab:/etc/gitlab#
Now my issue is that cannot find a way to fix these DB indexes, in 12-CE I have missing indexes, in 11.x I have duplicate indexes when it runs trough bash-db-migrate.
Current Status: Unhealthy
Migrations are pending. To resolve this issue, run: bin/rails db:migrate RAILS_ENV=production
