Gitlab Activa\Passive

Hi how can w configure drbd in gitlab for an Active\Passive HA setup

There isn’t a GitLab-provided doc on it. Is there a doc for active/passive configuration?

can you help me to do that

I’m using NFS instead of DRBD. On both nodes I used the mount options listed in the HA NFS doc, and on the passive node I added noauto,ro.

On active node:

  1. Mount /var/opt/gitlab
  2. Install GitLab
  3. Configure /etc/gitlab/gitlab.rb and run ‘gitlab-ctl reconfigure’
  4. Test service is up and running as desired

On passive node:

  1. Install GitLab
  2. Copy /etc/gitlab/* from active node
  3. Copy /etc/ssh/ssh_host_* from active node
  4. Run ‘gitlab-ctl reconfigure’
  5. Run ‘systemctl stop gitlab-runsvdir’
  6. Run ‘systemctl disable gitlab-runsvdir’

Note that I did not mount the storage on the passive node. When I need to failover I will do the following. Copying the ssh host keys between nodes is that developers won’t receive any SSH man-in-the-middle errors after the failover.

On Active node:

  1. Run ‘systemctl stop gitlab-runsvdir’
  2. Run ‘systemctl disable gitlab-runsvdir’
  3. Unmount /var/opt/gitlab
  4. Add ‘noauto,ro’ mount options in /etc/fstab

On Passive node:

  1. Remove ‘noauto,ro’ from mount options in /etc/fstab
  2. Mount /var/opt/gitlab
  3. Run ‘systemctl start gitlab-runsvdir’
  4. Run ‘systemctl enable gitlab-runsvdir’

Now the node that was passive is now active. You can also configure virtual network interface to bind a dedicated IP address to the active node, or use a load balancer that will only send traffic to the active node, so that the server hostname/IP doesn’t have to change after failover.

Hi thanks for the information and its really great,

But as of our need we cannot afford a single point of failure, Here NFS Will be a single point of failure right?

And other think we need 99.9 % up-time and we don’t want to run a manual task to switch this to the passive server.

i am trying to use pacemaker and drbd so the failover will be managed.

Thank you for your help! :slight_smile:

I have a question - when you write “/var/opt/gitlab” you mean ALL the directory OR like in https://docs.gitlab.com/ee/administration/high_availability/nfs.html
only:
git_data_dirs({“default” => “/gitlab-data/git-data”})
user[‘home’] = ‘/gitlab-data/home’
gitlab_rails[‘uploads_directory’] = ‘/gitlab-data/uploads’
gitlab_rails[‘shared_path’] = ‘/gitlab-data/shared’
gitlab_ci[‘builds_directory’] = ‘/gitlab-data/builds’

because I did like the link, and when I changed Passive to Active it doesn’t save the users so I cant login to check if everything works correctly.
and if you mount all the directory how you configure it at gitlab.rb and /etc/fstab?

are you using Active Directory users or locally? if locally that’s mean you should mount /etc/gitlab as well?

thank you!

Right, the link you posted for HA nfs is for active/active. For active/passive you do actually want the entire /var/opt/gitlab mounted. The reason is that the SQL database lives under /var/opt/gitlab. So that has to migrate to the new server during failover too. In an active/active configuration the SQL database is on a whole other server and all the rails instances communicate with SQL over the wire.

1 Like

thank you!

I used the same mount options in fstab as what they recommended in the active/active (except we’re using nfs v3 because there was too much extra overhead to setup v4)

server:share /var/opt/gitlab nfs defaults,nfsvers=3,soft,rsize=1048576,wsize=1048576,noatime,lookupcache=positive 0 2

There’s no related configuration changes in gitlab.rb, because the storage is all mounted to the default location.

1 Like