Struggling on Openshift Installation

Hi, following the gitlab advices to use latest helm charts I’m using following commands to install gitlab-ce

The whole process looks quite ok, after following the procedures for helm installation in my openshift cluster some of the pods don’t come up; especially the postgresql pod does not come up with error “Create Container Config Error” (maybe the reason for the other ones). Any hint what i can do?

My Environment:

kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1+a0ce1bc657", GitCommit:"a0ce1bc", GitTreeState:"clean", BuildDate:"2018-06-24T01:54:00Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1+a0ce1bc657", GitCommit:"a0ce1bc", GitTreeState:"clean", BuildDate:"2018-06-24T01:54:00Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"}

[root@ocn01de ~]# oc version
oc v3.9.0+71543b2-33
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO


openshift v3.9.0+71543b2-33
kubernetes v1.9.1+a0ce1bc657

Steps that i made:

helm repo add gitlab https://charts.gitlab.io/
helm repo update
helm upgrade --install gitlab-ce gitlab/gitlab --timeout 600 --set global.hosts.domain=be1.io --set global.hosts.externalIP=207.180.198.115 --set certmanager-issuer.email=martin@be1.io --set global.edition=ce

Console Output:

Release "gitlab-ce" does not exist. Installing it now.
NAME:   gitlab-ce
LAST DEPLOYED: Fri Jan  4 17:13:39 2019
NAMESPACE: gitlab-ce
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                                      DATA  AGE
gitlab-ce-certmanager-issuer-certmanager  2     3s
gitlab-ce-gitlab-runner                   3     3s
gitlab-ce-gitaly                          3     3s
gitlab-ce-gitlab-shell                    2     3s
gitlab-ce-nginx-ingress-tcp               1     3s
gitlab-ce-migrations                      4     3s
gitlab-ce-sidekiq-all-in-1                1     3s
gitlab-ce-sidekiq                         6     3s
gitlab-ce-task-runner                     4     3s
gitlab-ce-unicorn                         7     3s
gitlab-ce-workhorse-config                3     3s
gitlab-ce-unicorn-tests                   1     3s
gitlab-ce-minio-config-cm                 3     3s
gitlab-ce-nginx-ingress-controller        7     3s
gitlab-ce-postgresql                      0     3s
gitlab-ce-prometheus-server               3     3s
gitlab-ce-redis                           2     3s
gitlab-ce-registry                        2     3s

==> v1beta1/CustomResourceDefinition
NAME                               AGE
certificates.certmanager.k8s.io    3s
clusterissuers.certmanager.k8s.io  3s
issuers.certmanager.k8s.io         3s

==> v1beta1/ClusterRoleBinding
NAME                                     AGE
gitlab-ce-certmanager                    3s
gitlab-ce-prometheus-alertmanager        3s
gitlab-ce-prometheus-kube-state-metrics  3s
gitlab-ce-prometheus-node-exporter       3s
gitlab-ce-prometheus-server              3s

==> v1beta1/Role
NAME                     AGE
gitlab-ce-gitlab-runner  3s

==> v1/Job
NAME                              COMPLETIONS  DURATION  AGE
gitlab-ce-issuer.1                0/1          1s        1s
gitlab-ce-migrations.1            0/1          1s        1s
gitlab-ce-minio-create-buckets.1  0/1          1s        1s

==> v2beta1/HorizontalPodAutoscaler
NAME                        REFERENCE                              TARGETS         MINPODS  MAXPODS  REPLICAS  AGE
gitlab-ce-gitlab-shell      Deployment/gitlab-ce-gitlab-shell      <unknown>/75%   2        10       0         1s
gitlab-ce-sidekiq-all-in-1  Deployment/gitlab-ce-sidekiq-all-in-1  <unknown>/350m  1        10       0         1s
gitlab-ce-unicorn           Deployment/gitlab-ce-unicorn           <unknown>/1     2        10       0         1s
gitlab-ce-registry          Deployment/gitlab-ce-registry          <unknown>/75%   2        10       0         1s

==> v1/PersistentVolumeClaim
NAME                         STATUS  VOLUME  CAPACITY  ACCESS MODES  STORAGECLASS  AGE
gitlab-ce-minio              Bound   vol125  500Gi     RWO,RWX       3s
gitlab-ce-postgresql         Bound   vol183  500Gi     RWO,RWX       3s
gitlab-ce-prometheus-server  Bound   vol62   500Gi     RWO,RWX       3s
gitlab-ce-redis              Bound   vol185  500Gi     RWO,RWX       3s

==> v1beta1/ClusterRole
NAME                                     AGE
gitlab-ce-certmanager                    3s
gitlab-ce-prometheus-kube-state-metrics  3s
gitlab-ce-prometheus-server              3s

==> v1/Service
NAME                                        TYPE          CLUSTER-IP      EXTERNAL-IP                  PORT(S)                                  AGE
gitlab-ce-gitaly                            ClusterIP     None            <none>                       8075/TCP,9236/TCP                        3s
gitlab-ce-gitlab-shell                      ClusterIP     172.30.206.116  <none>                       22/TCP                                   3s
gitlab-ce-unicorn                           ClusterIP     172.30.148.67   <none>                       8080/TCP,8181/TCP                        2s
gitlab-ce-minio-svc                         ClusterIP     172.30.250.77   <none>                       9000/TCP                                 2s
gitlab-ce-nginx-ingress-controller-metrics  ClusterIP     172.30.22.173   <none>                       9913/TCP                                 2s
gitlab-ce-nginx-ingress-controller          LoadBalancer  172.30.130.110  172.29.244.50,172.29.244.50  80:32476/TCP,443:31965/TCP,22:32468/TCP  2s
gitlab-ce-nginx-ingress-controller-stats    ClusterIP     172.30.145.142  <none>                       18080/TCP                                2s
gitlab-ce-nginx-ingress-default-backend     ClusterIP     172.30.107.228  <none>                       80/TCP                                   2s
gitlab-ce-postgresql                        ClusterIP     172.30.41.84    <none>                       5432/TCP                                 2s
gitlab-ce-prometheus-server                 ClusterIP     172.30.242.211  <none>                       80/TCP                                   2s
gitlab-ce-redis                             ClusterIP     172.30.221.6    <none>                       6379/TCP,9121/TCP                        2s
gitlab-ce-registry                          ClusterIP     172.30.119.207  <none>                       5000/TCP                                 2s

==> v1beta1/Deployment
NAME                         DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
gitlab-ce-certmanager        1        1        1           0          2s
gitlab-ce-gitlab-runner      1        1        1           0          2s
gitlab-ce-postgresql         1        1        1           0          2s
gitlab-ce-prometheus-server  1        1        1           0          1s

==> v1beta2/StatefulSet
NAME              DESIRED  CURRENT  AGE
gitlab-ce-gitaly  1        1        1s

==> v1/Role
NAME                          AGE
gitlab-ce-certmanager-issuer  3s
gitlab-ce-nginx-ingress       3s

==> v1beta1/RoleBinding
NAME                     AGE
gitlab-ce-gitlab-runner  3s

==> v1beta2/Deployment
NAME                                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
gitlab-ce-gitlab-shell                   1        1        1           0          2s
gitlab-ce-sidekiq-all-in-1               1        1        1           0          2s
gitlab-ce-task-runner                    1        1        1           0          2s
gitlab-ce-unicorn                        1        1        1           0          2s
gitlab-ce-minio                          1        1        1           0          2s
gitlab-ce-nginx-ingress-controller       3        0        0           0          2s
gitlab-ce-nginx-ingress-default-backend  2        2        2           0          2s
gitlab-ce-redis                          1        1        1           0          1s
gitlab-ce-registry                       1        1        1           0          1s

==> v1beta1/PodDisruptionBudget
NAME                                     MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
gitlab-ce-gitaly                         N/A            1                0                    3s
gitlab-ce-gitlab-shell                   N/A            1                0                    3s
gitlab-ce-sidekiq                        N/A            1                0                    3s
gitlab-ce-unicorn                        N/A            1                0                    3s
gitlab-ce-minio-v1                       N/A            1                0                    3s
gitlab-ce-nginx-ingress-controller       2              N/A              0                    3s
gitlab-ce-nginx-ingress-default-backend  1              N/A              0                    3s
gitlab-ce-redis-v1                       N/A            1                0                    3s
gitlab-ce-registry-v1                    N/A            1                0                    3s

==> v1/ServiceAccount
NAME                                     SECRETS  AGE
gitlab-ce-certmanager-issuer             2        3s
gitlab-ce-certmanager                    2        3s
gitlab-ce-gitlab-runner                  2        3s
gitlab-ce-nginx-ingress                  2        3s
gitlab-ce-prometheus-alertmanager        2        3s
gitlab-ce-prometheus-kube-state-metrics  2        3s
gitlab-ce-prometheus-node-exporter       2        3s
gitlab-ce-prometheus-server              2        3s

==> v1/RoleBinding
NAME                          AGE
gitlab-ce-certmanager-issuer  3s
gitlab-ce-nginx-ingress       3s

==> v1beta1/Ingress
NAME                HOSTS            ADDRESS  PORTS  AGE
gitlab-ce-unicorn   gitlab.be1.io    80, 443  1s
gitlab-ce-minio     minio.be1.io     80, 443  1s
gitlab-ce-registry  registry.be1.io  80, 443  1s

==> v1/Pod(related)
NAME                                                      READY  STATUS             RESTARTS  AGE
gitlab-ce-certmanager-67549bf84c-g8hnn                    0/1    ContainerCreating  0         2s
gitlab-ce-gitlab-runner-58c964b556-zjwct                  0/1    Init:0/1           0         2s
gitlab-ce-gitlab-shell-7d4c6b7554-m5fq4                   0/1    Init:0/2           0         2s
gitlab-ce-sidekiq-all-in-1-5b4844dc6-c4kwn                0/1    Init:0/3           0         2s
gitlab-ce-task-runner-5d7d7cb65f-m8jpj                    0/1    Init:0/2           0         2s
gitlab-ce-unicorn-7f4646c448-66f5h                        0/2    Pending            0         2s
gitlab-ce-minio-5dd4f85c65-n8bzn                          0/1    Pending            0         2s
gitlab-ce-nginx-ingress-default-backend-7955548dbf-9tclb  0/1    Pending            0         1s
gitlab-ce-nginx-ingress-default-backend-7955548dbf-xm4wc  0/1    ContainerCreating  0         2s
gitlab-ce-postgresql-cc95bcc7b-mk6s4                      0/2    Pending            0         1s
gitlab-ce-prometheus-server-58d77dcbc7-5bq44              0/2    Pending            0         1s
gitlab-ce-redis-5dc6554754-g62zc                          0/2    Pending            0         1s
gitlab-ce-registry-856449f5db-5khf9                       0/1    Pending            0         1s
gitlab-ce-gitaly-0                                        0/1    Pending            0         1s
gitlab-ce-issuer.1-gxqbv                                  0/1    Pending            0         1s
gitlab-ce-migrations.1-8cn8n                              0/1    Pending            0         1s
gitlab-ce-minio-create-buckets.1-nh2wm                    0/1    Pending            0         1s


svc/gitlab-ce-postgresql - 172.30.41.84:5432 -> postgresql
  deployment/gitlab-ce-postgresql deploys postgres:9.6.8,wrouesnel/postgres_exporter:v0.1.1
    deployment #1 running for 16 minutes - 0/1 pods

No Errors, The Result is that some of the pods start ok, others fail - especially the postgresql pod:

Container gitlab-ce-postgresql

State:

Waiting (CreateContainerConfigError)

Ready:

false

Restart Count:

0

Let me extend the status

[root@ocn01de ~]# oc status -v
In project gitlab-ce on server 

svc/gitlab-ce-gitaly (headless) ports 8075, 9236
  statefulset/gitlab-ce-gitaly manages registry.gitlab.com/gitlab-org/build/cng/gitaly:v1.7.1
    created 16 minutes ago - 1 pod

svc/gitlab-ce-gitlab-shell - 172.30.206.116:22 -> 2222
  deployment/gitlab-ce-gitlab-shell deploys registry.gitlab.com/gitlab-org/build/cng/gitlab-shell:v8.4.3
    deployment #1 running for 16 minutes - 2 pods

svc/gitlab-ce-minio-svc - 172.30.250.77:9000
  deployment/gitlab-ce-minio deploys minio/minio:RELEASE.2017-12-28T01-21-00Z
    deployment #1 running for 16 minutes - 1 pod

svc/gitlab-ce-nginx-ingress-controller - 172.30.130.110 ports 80->http, 443->https, 22->gitlab-shell
  deployment/gitlab-ce-nginx-ingress-controller deploys quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
    deployment #1 running for 16 minutes - 0/3 pods growing to 3

svc/gitlab-ce-nginx-ingress-controller-metrics - 172.30.22.173:9913 -> metrics
  deployment/gitlab-ce-nginx-ingress-controller deploys quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
    deployment #1 running for 16 minutes - 0/3 pods growing to 3

svc/gitlab-ce-nginx-ingress-controller-stats - 172.30.145.142:18080 -> stats
  deployment/gitlab-ce-nginx-ingress-controller deploys quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.20.0
    deployment #1 running for 16 minutes - 0/3 pods growing to 3

svc/gitlab-ce-nginx-ingress-default-backend - 172.30.107.228:80 -> http
  deployment/gitlab-ce-nginx-ingress-default-backend deploys k8s.gcr.io/defaultbackend:1.4
    deployment #1 running for 16 minutes - 2 pods

svc/gitlab-ce-postgresql - 172.30.41.84:5432 -> postgresql
  deployment/gitlab-ce-postgresql deploys postgres:9.6.8,wrouesnel/postgres_exporter:v0.1.1
    deployment #1 running for 16 minutes - 0/1 pods

svc/gitlab-ce-prometheus-server - 172.30.242.211:80 -> 9090
  deployment/gitlab-ce-prometheus-server deploys jimmidyson/configmap-reload:v0.1,prom/prometheus:v2.2.1
    deployment #1 running for 16 minutes - 1 pod

svc/gitlab-ce-redis - 172.30.221.6 ports 6379, 9121->metrics
  deployment/gitlab-ce-redis deploys redis:3.2.12,oliver006/redis_exporter:latest
    deployment #1 running for 16 minutes - 1 pod

svc/gitlab-ce-registry - 172.30.119.207:5000
  deployment/gitlab-ce-registry deploys registry:2.6.2
    deployment #1 running for 16 minutes - 2 pods

svc/gitlab-ce-unicorn - 172.30.148.67 ports 8080, 8181
  deployment/gitlab-ce-unicorn deploys registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce:v11.6.2,registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce:v11.6.2
    deployment #1 running for 16 minutes - 0/2 pods

deployment/gitlab-ce-certmanager deploys quay.io/jetstack/cert-manager-controller:v0.4.0
  deployment #1 running for 16 minutes - 1 pod

deployment/gitlab-ce-gitlab-runner deploys gitlab/gitlab-runner:alpine-v11.5.0
  deployment #1 running for 16 minutes - 0/1 pods (warning: 8 restarts)

deployment/gitlab-ce-sidekiq-all-in-1 deploys registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v11.6.2
  deployment #1 running for 16 minutes - 0/1 pods

deployment/gitlab-ce-task-runner deploys registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce:v11.6.2
  deployment #1 running for 16 minutes - 1 pod

pod/gitlab-ce-migrations.1-8cn8n runs registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce:v11.6.2

Errors:
  * pod/gitlab-ce-migrations.1-8cn8n is crash-looping

    The container is starting and exiting repeatedly. This usually means the container is unable
    to start, misconfigured, or limited by security restrictions. Check the container logs with

      oc logs gitlab-ce-migrations.1-8cn8n -c migrations

  * hpa/gitlab-ce-gitlab-shell is attempting to scale Deployment/gitlab-ce-gitlab-shell, which doesn't exist
  * hpa/gitlab-ce-registry is attempting to scale Deployment/gitlab-ce-registry, which doesn't exist
  * hpa/gitlab-ce-sidekiq-all-in-1 is attempting to scale Deployment/gitlab-ce-sidekiq-all-in-1, which doesn't exist
  * hpa/gitlab-ce-unicorn is attempting to scale Deployment/gitlab-ce-unicorn, which doesn't exist

Warnings:
  * pod/gitlab-ce-gitlab-runner-58c964b556-zjwct has restarted within the last 10 minutes

Info:
  * deployment/gitlab-ce-certmanager has no liveness probe to verify pods are still running.
    try: oc set probe deployment/gitlab-ce-certmanager --liveness ...
  * deployment/gitlab-ce-redis has no liveness probe to verify pods are still running.
    try: oc set probe deployment/gitlab-ce-redis --liveness ...
  * deployment/gitlab-ce-registry has no liveness probe to verify pods are still running.
    try: oc set probe deployment/gitlab-ce-registry --liveness ...
  * deployment/gitlab-ce-task-runner has no liveness probe to verify pods are still running.
    try: oc set probe deployment/gitlab-ce-task-runner --liveness ...

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

And the behavour is the same when i use this options:
–set gitlab.migrations.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce
–set gitlab.sidekiq.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce
–set gitlab.unicorn.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce
–set gitlab.unicorn.workhorse.image=registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce
–set gitlab.task-runner.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce

May be the issue is the UID used by some pods. OpenShift is not supporting fixed UID by default. Have a look here.

When you check the pod logs in the Web UI you should see related error messages.

Thanks for the hint, I had the right anyuid flag but the wrong project name :wink: - corrected and that fixed the issue for about 80% of the containers.

for my painpoint the postgresql still no solution

po/gitlab-ce-postgresql-cc95bcc7b-rvv56 1/2 CreateContainerConfigError 0 3m

[root@ocn01de ~]# oc describe pod gitlab-ce-postgresql-cc95bcc7b-rvv56
Name:           gitlab-ce-postgresql-cc95bcc7b-rvv56
Namespace:      gitlab-ce
Node:           ocn01de.be1.io/207.180.198.115
Start Time:     Fri, 04 Jan 2019 19:47:27 +0100
Labels:         app=postgresql
                pod-template-hash=775167736
                release=gitlab-ce
Annotations:    openshift.io/scc=anyuid
Status:         Pending
IP:             10.128.1.254
Controlled By:  ReplicaSet/gitlab-ce-postgresql-cc95bcc7b
Containers:
  gitlab-ce-postgresql:
    Container ID:   
    Image:          postgres:9.6.8
    Image ID:       
    Port:           5432/TCP
    State:          Waiting
      Reason:       CreateContainerConfigError
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   exec [sh -c exec pg_isready --host $POD_IP] delay=120s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [sh -c exec pg_isready --host $POD_IP] delay=5s timeout=3s period=5s #success=1 #failure=3
    Environment:
      POSTGRES_USER:           gitlab
      PGUSER:                  gitlab
      POSTGRES_DB:             gitlabhq_production
      POSTGRES_INITDB_ARGS:    
      PGDATA:                  /var/lib/postgresql/data/pgdata
      POSTGRES_PASSWORD_FILE:  /conf/postgres-password
      POD_IP:                   (v1:status.podIP)
    Mounts:
      /conf from password-file (ro)
      /var/lib/postgresql/data/pgdata from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dk7mt (ro)
  metrics:
    Container ID:   docker://81070521c591fc69baa75b749d644016fd9454d63e841c09a50e42877562302e
    Image:          wrouesnel/postgres_exporter:v0.1.1
    Image ID:       docker-pullable://docker.io/wrouesnel/postgres_exporter@sha256:d8bc6221112d77b2d7b7746b729f848b0db60823eb385355636943934c09d822
    Port:           9187/TCP
    State:          Running
      Started:      Fri, 04 Jan 2019 19:47:31 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  256Mi
    Environment:
      DATA_SOURCE_NAME:  postgresql://gitlab@127.0.0.1:5432?sslmode=disable
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dk7mt (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  gitlab-ce-postgresql
    ReadOnly:   false
  password-file:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  gitlab-ce-postgresql-password
    Optional:    false
  default-token-dk7mt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dk7mt
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason  Age                  From                     Message
  ----     ------  ----                 ----                     -------
  Warning  Failed  5m (x1241 over 4h)   kubelet, ocn01de.be1.io  Error: lstat /mnt/data/vol183: no such file or directory
  Normal   Pulled  46s (x1264 over 4h)  kubelet, ocn01de.be1.io  Container image "postgres:9.6.8" already present on machine
[root@ocn01de ~]# 

it seems that it cannot mount the volume - but the volume exists…
Warning Failed 5m (x1241 over 4h) kubelet, ocn01de.be1.io Error: lstat /mnt/data/vol183: no such file or directory

[root@ocn01de ~]# oc describe pv vol183
Name:            vol183
Labels:          <none>
Annotations:     pv.kubernetes.io/bound-by-controller=yes
StorageClass:    
Status:          Bound
Claim:           gitlab-ce/gitlab-ce-postgresql
Reclaim Policy:  Retain
Access Modes:    RWO,RWX
Capacity:        500Gi
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/data/vol183
    HostPathType:  
Events:            <none>

[root@ocn01de ~]# oc describe pvc gitlab-ce-postgresql
Name:          gitlab-ce-postgresql
Namespace:     gitlab-ce
StorageClass:  
Status:        Bound
Volume:        vol183
Labels:        app=postgresql
               chart=postgresql-0.12.0
               heritage=Tiller
               release=gitlab-ce
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
Finalizers:    []
Capacity:      500Gi
Access Modes:  RWO,RWX
Events:        <none>
[root@ocn01de ~]#