Docker compose works fine locally, but not in CI/CD

I have project that contains a kafka stack with all its configuration files in my self-hosted GitLab. This project when cloned to my laptop, or anybody else’s for that matter, comes up with a simple docker compose up -d without any issues. However, when using this stack in a different project’s URL, the stack does not start up. The error says no config found, but the config is mapped via a volume to the container, same as what happens when running the command locally.

Let’s start off with the error:

A relevant excerpt from my docker-compose.yml

zookeeper:
    image: MY_REGISTRY/MY_GROUP/cp_kafka_dev/cp-zookeeper:${CONFLUENT_DOCKER_TAG}
    networks:
      - kafka
    restart: always
    hostname: zookeeper
    container_name: zookeeper
    healthcheck:
      test: echo srvr | nc zookeeper 2181 || exit 1
      retries: 20
      interval: 10s
    volumes:
      - ./security:/etc/kafka/secrets
    environment:
      TZ: Africa/Johannesburg
    env_file:
      - ./env_files/zookeeper.env

My .env file which is in the same root location as docker-compose.yaml:

CONFLUENT=7.5.0
CONFLUENT_DOCKER_TAG=7.5.0
REPOSITORY=MY_LOCAL_REGISTRY_URL
CONNECTOR_VERSION=7.5.0
SSL_CIPHER_SUITES=TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
LOGLEVEL=WARN

The files in the ./security folder:

The files in ./env_files folder:
image

My dind service definition:

test-topology:
  image: $CI_REGISTRY/MY_GROUP/base-containers/docker:latest
  stage: test-topology
  rules:
    - if: $CI_COMMIT_BRANCH == "develop"
  tags:
    - docker
  services:
    - name: $CI_REGISTRY/MY_GROUP/base-containers/dind:latest
      alias: docker

These tow images are just ‘retagged’ images from upstream docker 20.19 and dind 20.19. I only use it this way to store it on my local gitlab.

I have done a debug in the pipeline to check if I am indeed able to list all the files during the CI run, and they are indeed in the locations specified. So I am really at a loss, and don’t even know where to look for issues anymore. Can anybody please help me?

EDIT: Could this be a dind volumes issue? Should I copy the files to a directory inside of /builds in order for the dind containers to have access?

TIA

I fixed it. The issue were volumes and the availability of the files. Based on the info from Docker volumes not mounted when using docker:dind (#41227) · Issues · GitLab.org / GitLab FOSS · GitLab I was able to create a shared folder space into which I clone the upstream projects, then make the files available to the service containers.

...
script:
    - mkdir -p ${SHARED_PATH}
    - cd ${SHARED_PATH} && git clone... 
...

Then I needed to register a new priviledged docker runner, and had to ensure the volumes definition inside this runner’s config looked like this:

volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/builds:/builds"]

Now it all runs.