Upgrade from an 2+ years old installation of gitlab 6.9 from source with mysql to 8.13 with postgresql with omnibus with docker containers in one shot

ReSearchITEng · October 26, 2016, 3:45pm

This activity has been finished, and we are 5 days with the new system apparently without any major issue.
We did a trial in feb 2016, and now we did the actual migration (Oct 2016). We did it in ~4h plus additional time for finetunings and writing notes for this guide, giving a total of 8h (aka 1 person day).

All the issues our developers faced are not related to upgrade steps, but to:

few bugs in 8.13, like: egit plugin in eclipse Luna version does not work with ssh in 8.13, while it was working fine with 6.9.2. Work around: use https protocol instead of ssh).
while upacking the archive.zip in 6.9.2 there the folder name was the project_name.git while now it’s project_name-branch-gitcommitid.
people kept using the old host in their scripts, and the curl command did not have -L (follow) option.

#Challenge
Upgrade from an RH v6 installation of gitlab 6.9.2 from source with mysql to 8.13 with postgresql with omnibus with docker containers in one shot .
Of course, with min. downtime possible.
Old version: 6.9.2 e46b644 ; Target version: 8.13.0.ce0

#Prerequisites:

a new machine with running docker service (enabled on each machine restart)
basic docker understanding
basic sql understanding

#Notes:

biggest challenge is preparing the DB. Repository copy, authorized_keys, can be done without any issue.
To be a perfectionist you may want to copy also old machine’s ssh host keys and put them in: /storage/srv/gitlab/config
We use: /storage/srv/ as the folder where we keep all the git related files on the new machine. Of course you may want to replace this string with yours.
We’ll do the mysql to postgresql using the 8.13 version, while the db structure upgrade using the
oldest docker container we found: 8.8.1-ce.0

FYI, by using directly 8.13 we would get worst issues:

/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20160225090018_add_delete_at_to_issues.rb
mv /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20160122185421_add_pending_delete_to_project.rb

And one cannot use tricks like:

# mv /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20160225090018_add_delete_at_to_issues.rb /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140729152410_add_delete_at_to_issues.rb
# mv /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20160122185421_add_pending_delete_to_project.rb /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140729152413_add_pending_delete_to_project.rb

-because the classes were changed and no real solution on that path.

DB export for mysql_to_postgresql (activity on old machine)

On OLD MACHINE we will follow mainly 2nd part of mysql_to_postgresql guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/mysql_to_postgresql.md

make a mysqldump of the old DB with postgresql compatibility:

mysqldump --compatible=postgresql --default-character-set=utf8 -r gitlabhq_production.mysql -u git gitlabhq_production -p REPLACE_YOUR_DB_PASSWORD_HERE
git clone... (as per the link above)

Take the file above to the new machine and, as per the link above, do some more preparations for postgresql by using
Ensure you have python 2.7 or newer and run (as per the link above):

python mysql-postgresql-converter/db_converter.py gitlabhq_production.mysql /storage/srv/gitlab/data/database.psql

Create an empty postgresql db and import the data

This will be our gitlab db. We start with an empty one and afterwards on import the dump created above.

There might be easier ways to do it, but I prefed o create this empty db using gitlab commands inside an gitlab container. This procedure works for docker images labeled: 8.12.7-ce.0 and probably 8.13.0-ce.0 as well, but does not work in older ones. In older ones it always autopopulates the DB, and we cannot reach a completely empty DB.

docker stop gitlab_mig
docker rm gitlab_mig
cd /storage/srv/gitlab/
#rm -rf config/* logs/* data/*
docker -D run --detach --name gitlab_mig \
 --hostname gitlab.corp.dontbeevil.com \
 --publish 443:443 --publish 80:80 --publish 22:22 \
 --restart always \
 --env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.corp.dontbeevil.com/'; gitlab_rails['lfs_enabled'] = true;" \
 --volume /storage/srv/gitlab/config:/etc/gitlab \
 --volume /storage/srv/gitlab/logs:/var/log/gitlab \
 --volume /storage/srv/gitlab/data:/var/opt/gitlab \
 gitlab/gitlab-ce:8.12.7-ce.0 #8.9.5-ce.0 #8.3.10-ce.2 (in older ones, git)

docker exec -ti gitlab_mig bash

sleep 80
gitlab-ctl status # wait to have all up
gitlab-ctl stop gitlab-workhorse #as per docs, leave this one up; I think should not matter much
gitlab-ctl stop logrotate
gitlab-ctl stop nginx
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
gitlab-ctl start redis
gitlab-ctl start postgresql

gitlab-rake db:drop
gitlab-ctl reconfigure #recreates empty db; if gitlab image version is older, it will create all DB, and we don't want that!

su - git
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
\d+
# It should return empty !!! DO not proceed if not empty! Try different gitlab docker image version or create the empty DB manually!
#type CTRL+D to exit psql

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -f /var/opt/gitlab/database.psql -U gitlab
# There should be no errors, only NOTICEs
#type CTRL+D to exit psql

exit #exit git user

gitlab-ctl stop redis
gitlab-ctl stop postgresql

exit #exit root and container in same time.

docker stop gitlab_mig #stop this temp. container
docker rm gitlab_mig #remove it, we don't need the container any longer; we only care of /storage/srv/gitlab/data/postgresql which we mounted outside of container.

cd /storage/srv/gitlab/
rm -rf config/* #we don't want anything that was generated by this container besides the data/postgresql
rm -rf logs/*
mv data/postgresql .
rm -rf data/*  #This is req. otherwise redis does not come up with old data here...
mv postgresql data/
cp -r data/postgresql /storage/srv/gitlab/postgresql.datamigrated_structurependingtobeupdated.backup #should we need it by chance

Start migration

At this stage we can start an gitlab (for now the older docker version we can find) with the DB we just created:

docker -D run --detach --name gitlab_mig \
 --hostname gitlab.corp.dontbeevil.com \
 --publish 443:443 --publish 80:80 --publish 22:22 \
 --restart always \
 --env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.corp.dontbeevil.com/'; gitlab_rails['lfs_enabled'] = true;" \
 --volume /storage/srv/gitlab/config:/etc/gitlab \
 --volume /storage/srv/gitlab/logs:/var/log/gitlab \
 --volume /storage/srv/gitlab/data:/var/opt/gitlab \
 gitlab/gitlab-ce:8.3.10-ce.2  #the oldest I could find on the dockerhub list of tagged containers.

START db structure upgrade !!!##############

There are some ~6 issues gitlab has and we’ll apply some workarounds. Hopefully they will be fixed in future.
Back in Feb 2016 when we did a trial for this migration there were more issues which aparently were fixed till now (Oct 2016).

docker exec -ti gitlab_mig bash
su - git
gitlab-rake  --trace db:migrate RAILS_ENV=production

20131112114325
First error is on dbpatch 20131112114325 related to broadcast_messages table.
We don’t consider it worth manually fix the structure to the keep the data, so we’ll drop it and allow gitlab to recreate it:

su - git
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
DROP TABLE broadcast_messages;
#CTRL+D

resume the db upgrade:

su - git
gitlab-rake  --trace db:migrate RAILS_ENV=production

20140122112253 & 20140122112253
Expected error: 20140122112253 #PG::DuplicateTable: ERROR: relation “merge_request_diffs” already exists

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
update merge_request_diffs set state = 'collected' where state is NULL;

#with root (in containter):
vi /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140304005354*
# and remove the add_index line.

#with root (in containter):
vi /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140122112253*
#Clear all and put only:
class CreateMergeRequestDiffs < ActiveRecord::Migration
  def up
    change_column_default :merge_request_diffs, :state, 'collected'
  end
end

Resume the upgrade

su - git
gitlab-rake  --trace db:migrate RAILS_ENV=production

20140209025651 PG::DuplicateTable: ERROR: relation “emails” already exists

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
ALTER TABLE emails ALTER COLUMN email TYPE character varying;
#CTRL+D

#with root: make empty Class: as the current table is identical with what upgrade is trying to do:
vi /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140209025651_create_emails.rb
class CreateEmails < ActiveRecord::Migration
end

20140625115202
Resume upgrade and you’ll get the error below

su - git
gitlab-rake --trace db:migrate RAILS_ENV=production

#20140625115202 users_star_projects already exists # In our situation we don't have anything in this table (\dt+ users_star_projects shows 0 bytes, select * from users_star_projects; returns 0 rows), , so we dropped it
# if you have data in it, you may want to play with: #vi /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140625115202*
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
DROP TABLE users_star_projects;
#CTRL+D

20140729134820
Resume upgrade and you’ll get the error below

gitlab-rake  --trace db:migrate RAILS_ENV=production

#20140729134820  PG::DuplicateTable: ERROR:  relation "labels" already exists; # In our situation we don't have anything here (\dt+ labels shows 0 bytes, select * from labels; returns 0 rows), , so we drop it
#vi /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140729134820*
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
DROP TABLE labels;
#CTRL+D

gitlab-rake  --trace db:migrate RAILS_ENV=production

#20140729140420 PG::DuplicateTable: ERROR:  relation "label_links" already exists
# In our case is empty, so we drop it
DROP TABLE label_links;
#CTRL+D

20140729152420
Resume upgrade and you’ll get the error below

gitlab-rake  --trace db:migrate RAILS_ENV=production

#20140729152420 MigrateTaggableLabels: migrating
#undefined local variable or method `template' for #<Label:0x0000000803b918>/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activemodel-4.2.4/lib/active_model/attribute_methods.rb:433:in `method_missing'
#ERROR:  ->>> 20140729152420 MigrateTaggableLabels: migrating

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
alter table labels add column template boolean default false;
#CTRL+D

Going further we have identified a bunch of errors when gitlab tries to create tables which already exist.
The important part is that all but one (application_settings) of those tables are empty, so we can safely drop them and allow gitlab to recreate them.

gitlab-rake  --trace db:migrate RAILS_ENV=production

#20140914113604 PG::DuplicateTable: ERROR:  relation "members" already exists
#In our case it's empty, so we drop it.
DROP TABLE members;
#CTRL+D

gitlab-rake  --trace db:migrate RAILS_ENV=production

#20141118150935 PG::DuplicateTable: ERROR:  relation "audit_events" already exists
#In our case it's empty, so we drop it.
DROP TABLE audit_events;
#CTRL+D

#20141121161704 PG::DuplicateTable: ERROR:  relation "identities" already exists   # CREATE TABLE "identities"
#In our case it's empty, so we drop it.
DROP TABLE identities;
#CTRL+D

#20141216155758 PG::DuplicateTable: ERROR:  relation "oauth_applications" already exists
DROP TABLE oauth_applications;

DROP TABLE oauth_access_grants;
oauth_access_tokens

#20150108073740

#20150108073740

git@gitlab:~$ grep -i create_table /opt/gitlab/embedded/service/gitlab-rails/db/migrate/2015*
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150108073740_create_application_settings.rb:    create_table :application_settings do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150313012111_create_subscriptions_table.rb:    create_table :subscriptions do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150417121913_create_project_import_data.rb:    create_table :project_import_data do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150806104937_create_abuse_reports.rb:    create_table :abuse_reports do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150818213832_add_sent_notifications.rb:    create_table :sent_notifications do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_application_settings", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_builds", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_commits", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_events", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_jobs", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_projects", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_runner_projects", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_runners", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_services", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_sessions", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_trigger_requests", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_triggers", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_variables", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150826001931_add_ci_tables.rb:    create_table "ci_web_hooks", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150914215247_add_ci_tags.rb:    create_table "ci_taggings", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150914215247_add_ci_tags.rb:    create_table "ci_tags", force: true do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20151103134857_create_lfs_objects.rb:    create_table :lfs_objects do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20151103134958_create_lfs_objects_projects.rb:    create_table :lfs_objects_projects do |t|
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20151105094515_create_releases.rb:    create_table :releases do |t|

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
select * FROM application_settings   # IT HAS 1 ROW (by default);  all the rest are empty!
subscriptions
project_import_data
abuse_reports
sent_notifications
ci_application_settings
ci_builds
ci_commits
ci_events
ci_jobs
ci_projects
ci_runner_projects
ci_runners
ci_services
ci_sessions
ci_trigger_requests
ci_triggers
ci_variables
ci_web_hooks
ci_taggings
ci_tags
lfs_objects
lfs_objects_projects
releases

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
alter table application_settings rename to application_settings_BCK;
DROP TABLE subscriptions;
DROP TABLE project_import_data;
DROP TABLE abuse_reports;
DROP TABLE sent_notifications;
DROP TABLE ci_application_settings;
DROP TABLE ci_builds;
DROP TABLE ci_commits;
DROP TABLE ci_events;
DROP TABLE ci_jobs;
DROP TABLE ci_projects;
DROP TABLE ci_runner_projects;
DROP TABLE ci_runners;
DROP TABLE ci_services;
DROP TABLE ci_sessions;
DROP TABLE ci_trigger_requests;
DROP TABLE ci_triggers;
DROP TABLE ci_variables;
DROP TABLE ci_web_hooks;
DROP TABLE ci_taggings;
DROP TABLE ci_tags;
DROP TABLE lfs_objects;
DROP TABLE lfs_objects_projects;
DROP TABLE releases;

gitlab-rake  --trace db:migrate RAILS_ENV=production

#20150902001023 ERROR ->>>>    20150902001023 AddTemplateToLabel: migrating
/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
alter table labels drop column template;

gitlab-rake  --trace db:migrate RAILS_ENV=production

/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production -U gitlab
alter table application_settings rename to application_settings_empty;
alter table application_settings_BCK rename to application_settings;


docker -D run --detach --name gitlab \
 --hostname gitlab.corp.dontbeevil.com \
 --publish 443:443 --publish 80:80 --publish 22:22 \
 --restart always \
 --env GITLAB_OMNIBUS_CONFIG="external_url 'http://gitlab.corp.dontbeevil.com/'; gitlab_rails['lfs_enabled'] = true; mattermost_external_url 'http://mattermost.corp.dontbeevil.com';" \
 --volume /storage/srv/gitlab/config:/etc/gitlab \
 --volume /storage/srv/gitlab/logs:/var/log/gitlab \
 --volume /storage/srv/gitlab/data:/var/opt/gitlab \
 --volume /etc/localtime:/etc/localtime \
 gitlab/gitlab-ce:8.12.7-ce.0


#from host:
# Put your repos
chown -R 998:998 /storage/srv/gitlab/data/git-data/repositories/*
chown 998:998 /storage/srv/gitlab/data

# SSL CERTS
#/storage/srv/gitlab/config/ssl
#name them:
#/etc/gitlab/ssl/#{node['fqdn']}.crt"
#/etc/gitlab/ssl/#{node['fqdn']}.key
#chmod 700 the files under ssl
#ideally put in the same folder ca.crt  with full chain.

# Put your host ssh keys from old host (ssh_host_* - find them under /etc/....) to new machine, under /storage/srv/gitlab/config
#ensure proper perms (e.g. all private keys are chmod 600 )

docker exec -ti gitlab bash

su - git
/opt/gitlab/embedded/service/gitlab-shell/bin/create-hooks /var/opt/gitlab/git-data/repositories # most probably your links changed, this will update them
#e.g.    repositories/ansible-roles/scanner.wiki.git/hooks -> ...

gitlab-rake gitlab:check

Forward trafic

Forward traffic from old machine to new one (till everyone updates their code/tools)
Old machine will forward the requests to new machine (should you require)

On old machine: define IP TABLES port fowarding and rules.

echo 1 >/proc/sys/net/ipv4/ip_forward
sysctl net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" >> etc/sysctl.conf #to save for next restart
sysctl -a | grep ip_forward

iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 22 -j DNAT --to-destination 10.234.176.30
iptables -t nat -A POSTROUTING -o eth1 -p tcp --dport 22 -d 10.234.176.30 -j SNAT --to-source 10.232.167.150

iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j DNAT --to-destination 10.234.176.30
iptables -t nat -A POSTROUTING -o eth1 -p tcp --dport 80 -d 10.234.176.30 -j SNAT --to-source 10.232.167.150

iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j DNAT --to-destination 10.234.176.30
iptables -t nat -A POSTROUTING -o eth1 -p tcp --dport 443 -d 10.234.176.30 -j SNAT --to-source 10.232.167.150

iptables -t nat -A PREROUTING -i eth1 -p udp --dport 22 -j DNAT --to-destination 10.234.176.30
iptables -t nat -A POSTROUTING -o eth1 -p udp --dport 22 -d 10.234.176.30 -j SNAT --to-source 10.232.167.150

iptables -t nat -A PREROUTING -i eth1 -p udp --dport 80 -j DNAT --to-destination 10.234.176.30
iptables -t nat -A POSTROUTING -o eth1 -p udp --dport 80 -d 10.234.176.30 -j SNAT --to-source 10.232.167.150

iptables -t nat -A PREROUTING -i eth1 -p udp --dport 443 -j DNAT --to-destination 10.234.176.30
iptables -t nat -A POSTROUTING -o eth1 -p udp --dport 443 -d 10.234.176.30 -j SNAT --to-source 10.232.167.150

/etc/init.d/iptables save   #Save them the rules for reboot
/etc/init.d/iptables status

Start/upgrade script

The best way to start or upgrade it would be the follow script:

#/bin/bash
VERSION=${1:-'8.13.0-ce.0'}
GITLAB_FILEMAX=1000000
[[ $(cat /proc/sys/fs/file-max) -lt ${GITLAB_FILEMAX} ]] && echo $GITLAB_FILEMAX > /proc/sys/fs/file-max

# https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1217 #mattermost docker containers
# # --sysctl vm.overcommit_memory=1 \
docker run --detach --name gitlab \
 --hostname gitlab.corp.dontbeevil.com \
 --sysctl net.core.somaxconn=1024 \
 --ulimit sigpending=62793 \
 --ulimit nproc=131072 \
 --ulimit nofile=60000 \
 --ulimit core=0 \
 --publish 443:443 --publish 80:80 --publish 22:22 --publish 8060:8060 \
 --restart always \
 --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.corp.dontbeevil.com/'; gitlab_rails['lfs_enabled'] = true; mattermost_external_url 'http://mattermost.corp.dontbeevil.com';" \
 --volume /storage/srv/gitlab/config:/etc/gitlab:z \
 --volume /storage/srv/gitlab/logs:/var/log/gitlab:z \
 --volume /storage/srv/gitlab/data:/var/opt/gitlab:z \
 --volume /etc/localtime:/etc/localtime \
 gitlab/gitlab-ce:${VERSION}

Note: this script should not be required at host machine reboot, as docker will take care of restarting the containers.

Upgrade from an 2+ years old installation of gitlab **6.9 from source with mysql** to **8.13 with postgresql with omnibus with docker containers** in **one shot**