Resize /opt/gitlab

Hi all,

Running GitLab 7.10.0 a651951 on Centos 6.6.

We’ve run into a problem where /opt/gitlab is using most of the available space, so we are looking to migrate.

My plan to do this was as follows:

# take a backup
/home/user/bin/gitlab-backup.sh
# mount new storage at /opt/gitlab-tmp
mount /dev/vg2/lv0 /opt/gitlab-tmp
# stop gitlab
gitlab-ctl stop
# copy data from /opt/gitlab to /opt/gitlab-tmp 
rsync -avz /opt/gitlab/ /opt/gitlab-tmp/
# unmount temporary mount
umount /opt/gitlab-tmp
# delete temporary mount dir
rmdir /opt/gitlab-tmp
# move existing directory out of the way
mv /opt/gitlab /opt/gitlab.old
# make new dir 
mkdir /opt/gitlab
# uncomment fstab entry for the new storage
sed -e '/\/dev\/vg2\/lv0/ s/^#//' -i /tmp/fstab
# mount new storage
mount /opt/gitlab
# start gitlab
gitlab-ctl start
# remove "old" if everything tests ok
rm -rf /opt/gitlab.old

However when i came to implementing this, i found there were a number of processes running, holding files on /opt/gitlab open.

[root@gitlabsrv01 ~]# gitlab-ctl status
run: logrotate: (pid 905) 95s; run: log: (pid 904) 95s
run: nginx: (pid 903) 95s; run: log: (pid 902) 95s
run: redis: (pid 901) 95s; run: log: (pid 900) 95s
run: sidekiq: (pid 899) 95s; run: log: (pid 898) 95s
run: unicorn: (pid 897) 95s; run: log: (pid 896) 95s
[root@gitlabsrv01 ~]# gitlab-ctl stop
ok: down: logrotate: 0s, normally up
ok: down: nginx: 0s, normally up
ok: down: redis: 0s, normally up
ok: down: sidekiq: 1s, normally up
ok: down: unicorn: 1s, normally up
[root@gitlabsrv01 ~]# lsof | grep '/opt/gitlab' | awk '{print $2}' | sort | uniq
878
891
892
893
894
895
896
898
900
902
904
912
[root@gitlabsrv01 ~]# ps -ef | egrep '878|891|892|893|894|895|896|898|900|902|904|912'
root       878     1  0 10:06 ?        00:00:00 runsvdir -P /opt/gitlab/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
root       891   878  0 10:06 ?        00:00:00 runsv unicorn
root       892   878  0 10:06 ?        00:00:00 runsv nginx
root       893   878  0 10:06 ?        00:00:00 runsv sidekiq
root       894   878  0 10:06 ?        00:00:00 runsv logrotate
root       895   878  0 10:06 ?        00:00:00 runsv redis
root       896   891  0 10:06 ?        00:00:00 svlogd -tt /var/log/gitlab/unicorn
root       898   893  0 10:06 ?        00:00:00 svlogd -tt /var/log/gitlab/sidekiq
root       900   895  0 10:06 ?        00:00:00 svlogd -tt /var/log/gitlab/redis
root       902   892  0 10:06 ?        00:00:00 svlogd -tt /var/log/gitlab/nginx
root       904   894  0 10:06 ?        00:00:00 svlogd -tt /var/log/gitlab/logrotate
root       912     1  0 10:06 ?        00:00:00 sleep 600
root     16318 16214  0 10:09 pts/0    00:00:00 egrep 878|891|892|893|894|895|896|898|900|902|904|912
[root@gitlabsrv01 ~]# gitlab-ctl start
ok: run: logrotate: (pid 16327) 0s
ok: run: nginx: (pid 16330) 0s
ok: run: redis: (pid 16332) 0s
ok: run: sidekiq: (pid 16334) 0s
ok: run: unicorn: (pid 16336) 0s
[root@gitlabsrv01 ~]# gitlab-ctl status
run: logrotate: (pid 16327) 4s; run: log: (pid 904) 197s
run: nginx: (pid 16330) 4s; run: log: (pid 902) 197s
run: redis: (pid 16332) 4s; run: log: (pid 900) 197s
run: sidekiq: (pid 16334) 4s; run: log: (pid 898) 197s
run: unicorn: (pid 16336) 4s; run: log: (pid 896) 197s
[root@gitlabsrv01 ~]#

How can i stop these services ?

(also open to any other advise re: process/procedure etc)

gitlab-ctl stop should stop all the services. If that doesn’t work, try stopping them one by one with gitlab-ctl stop unicorn, etc.

Are you sure it is /opt/gitlab that uses most of the space? GitLab stores git data in /var/opt/gitlab/git-data and there is a configuration option for that.

See https://gitlab.com/gitlab-org/omnibus-gitlab/blob/629def0a7a26e7c2326566f0758d4a27857b52a3/README.md#storing-git-data-in-an-alternative-directory and https://gitlab.com/gitlab-org/omnibus-gitlab/blob/7-10-stable/files/gitlab-config-template/gitlab.rb.template#L116

Hi,

yeah, i am sure it is /opt/gitlab. The volume it sits on is only 2gb, and 960mb of that is being taken up by /opt/gitlab.

The git data is already stored elsewhere (/u01)

Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/vg0-lv0  2.0G  1.7G  236M  88% /
tmpfs                1.9G     0  1.9G   0% /dev/shm
/dev/sda1            248M  126M  110M  54% /boot
/dev/mapper/vg0-lv3  2.0G  287M  1.6G  15% /home
/dev/mapper/vg0-lv4  4.0G 1022M  2.8G  27% /usr
/dev/mapper/vg0-lv2  4.0G  988M  2.8G  26% /var
/dev/mapper/vg1-lv0   20G  7.1G   12G  38% /u01

the gitlab-ctl commands stop the services, however the runsvdir,runsv,svlogd continue to run, keeping files open on /opt/gitlab. I am assuming this is a supervisor process that will monitor and stop/start/restart services when they fail.

I am just not sure how to stop the supervisor processes.