Installing gitlab on Openshift 3.11 - PVs get bounded incorrectly

Hello

After using Gitlab for the past few years personally, I’v now managed to convince my work to install Gitlab.

However, installation is proving to be very complicated!

I’m trying to install Gitlab on Openshift 3.11.

The immediate problem I’m facing is that the Persistent Volumes are being bound to the incorrect PVC.

For example, if I run oc get pvc I get the following - note that gitlab-prometheus-server is bound to gitlab-minio:

NAME                        STATUS    VOLUME                      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
gitlab-minio                Pending   gitlab-minio                0                                        1h
gitlab-postgresql           Bound     gitlab-prometheus-server    10Gi       RWO                           1h
gitlab-prometheus-server    Bound     gitlab-minio                10Gi       RWO                           1h
gitlab-redis                Pending   gitlab-redis                0                                        1h
repo-data-gitlab-gitaly-0   Bound     repo-data-gitlab-gitaly-0   50Gi       RWO                           1h

I’ve tried adding to my values.yaml file similar to what is on examples/storage/use_manual_volumes.yml · master · GitLab.org / charts / GitLab Chart · GitLab, but with the correct volume names, eg:

gitlab:
  gitaly:
    persistence:
      volumeName: repo-data-gitlab-gitaly-0
    
postgresql:
  persistence:
    volumeName: gitlab-postgresql
minio:
  persistence:
    volumeName: gitlab-minio
redis:
  persistence:
    volumeName: gitlab-redis

However this hasnt made any difference.

I’ve created an in-depth SO post that outlines the issue in more detail at kubernetes helm - Installing gitlab - PVs get bounded incorrectly, externalIP causes error - Stack Overflow

How can I get PVs to bind to the correct, and corresponding, PVCs?

Other issues

Postgres

I can see that Postgres is having some permissions issues:
oc logs gitlab-postgresql-65bd954977-wcvkk -c gitlab-postgresql returns

chown: changing ownership of '/var/lib/postgresql/data/pgdata': Permission denied

This is after giving 777 the persistent volume folder on the host.

Prometheus

Similarly, oc logs gitlab-prometheus-server-75585684b4-wnznq -c prometheus-server returns:

...
level=error ts=2019-11-26T00:34:04.344Z caller=main.go:731 err="opening storage failed: lock DB directory: open /data/lock: permission denied"

gitlay

The logs from oc logs gitlab-gitaly-0 show:

...
time="2019-11-26T01:53:41Z" level=fatal msg="load config" config_path=/etc/gitaly/config.toml error="mkdir /home/git/repositories/+gitaly: permission denied"

UPDATE

I managed to solve the PV/PVC mis-binding issue by using claimRef in the persistent volume, eg (ansible code):

  - name: "Create PersistentVolume for each gitlab component"
    k8s:
      name: "{{ item[0] | lower }}"
      state: present
      definition:
        apiVersion: v1
        kind: PersistentVolume
        metadata:
          namespace: kube-system
          name: "{{ item[0] | lower }}"
          labels:
            app: "{{ item[1] }}"
        spec:
          accessModes:
          - ReadWriteOnce
          capacity:
            storage: "{{ item[2] }}"
          hostPath:
            path: "/{{ansible_env.PV_HOST_DIRECTORY}}/{{  item[0] | lower }}"
          persistentVolumeReclaimPolicy: Retain
          claimRef:
            namespace: kube-system
            name: "{{ item[0] | lower }}"
    loop:
        - [ 'gitlab-minio', 'minio', '10Gi' ]
        - [ 'gitlab-postgresql', 'postgresql', '10Gi']
        - [ 'gitlab-prometheus-server', 'prometheus', '10Gi']
        - [ 'repo-data-gitlab-gitaly-0', 'gitlab', '50Gi']
        - [ 'gitlab-redis', 'gitlab-redis', '5Gi']

However, the permission issues still remain.

Thanks