Has Anyone Managed To Get Kafka Setup As A Service Within CI environment?

GitLab CI/CD
1

Has anyone managed to get Kafka setup as a service within CI environment? I am currently trying to use the lensesio/fast-data-dev docker image. Have also tried the Confluent images, see below…

I have the following CI job for running tests. Setting ADV_HOST to the kafka container on the same docker network via Gitlab CI services. The job hangs when trying to contact the kafka container.

GitLab CI Job Using lensesio/fast-data-dev

tests:
  stage: test
  variables:
    ADV_HOST: kafka
    DISABLE: azure-documentdb,blockchain,bloomberg,cassandra,coap,druid,elastic,elastic5,ftp,hazelcast,hbase,influxdb,jms,kudu,mongodb,mqtt,redis,rethink,voltdb,yahoo,hdfs,jdbc,elasticsearch,s3,twitter
    CONNECT_HEAP: 512m
    MINIO_BUCKET: images
    SAMPLEDATA: 0
    REST_PORT: 8082
    FORWARDLOGS: 0
    RUNTESTS: 0
    DISABLE_JMX: 1
    WEB_PORT: 0
    DISABLE: hive-1.1

  ##
  # Services
  # - Kafka
  # - Mosquitto (MQTT)
  # - Minio (S3)
  ##
  services:
  - name: lensesio/fast-data-dev:2.5.1-L0
    alias: kafka
  - name: dcs3spp/minio:version-1.0.2
    alias: minio
  - name: eclipse-mosquitto:1.6.9
    alias: mqtt


  script:
    - >
      dotnet test --no-restore
      --logger:trx
      --settings:Tests/coverlet.runsettings
      --collect:"XPlat Code Coverage"
      WebApp.sln

CI job hangs trying to contact kafka container

info: WebApp.Kafka.Admin.KafkaAdminService[0]
      Admin service trying to create Kafka Topic...
info: WebApp.Kafka.Admin.KafkaAdminService[0]
      Topic::eventbus, ReplicationCount::1, PartitionCount::3
info: WebApp.Kafka.Admin.KafkaAdminService[0]
      Bootstrap Servers::kafka:9092

Confluent Kafka docker images/

tests:
  stage: test
  variables:
    ZOOKEEPER_CLIENT_PORT: 2181
    ZOOKEEPER_TICK_TIME: 2000
    KAFKA_BROKER_ID: 1
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
    KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  ##
  # Services
  # - Kafka
  # - Mosquitto (MQTT)
  # - Minio (S3)
  ##
  services:
  - name: confluentinc/cp-zookeeper:5.1.0
    alias: zookeeper
  - name: confluentinc/cp-kafka:5.1.0
    alias: kafka
  - name: dcs3spp/minio:version-1.0.2
    alias: minio
  - name: eclipse-mosquitto:1.6.9
    alias: mqtt


  script:
    - >
      dotnet test --no-restore
      --logger:trx
      --settings:Tests/coverlet.runsettings
      --collect:"XPlat Code Coverage"
      WebApp.sln

Confluent Images Error log

2020-11-15T12:15:44.558992574Z [2020-11-15 12:15:44,536] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.558996094Z [2020-11-15 12:15:44,536] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.558999139Z [2020-11-15 12:15:44,542] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.559002735Z [2020-11-15 12:15:44,542] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.559006873Z [2020-11-15 12:15:44,542] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.559009945Z [2020-11-15 12:15:44,542] INFO Server environment:os.version=4.19.78-coreos (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.559012987Z [2020-11-15 12:15:44,542] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.559015990Z [2020-11-15 12:15:44,542] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.559019219Z [2020-11-15 12:15:44,544] INFO Server environment:user.dir=/ (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.646950748Z [2020-11-15 12:15:44,642] INFO tickTime set to 2000 (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.646965707Z [2020-11-15 12:15:44,642] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.646969289Z [2020-11-15 12:15:44,642] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
2020-11-15T12:15:44.748601072Z [2020-11-15 12:15:44,725] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
2020-11-15T12:15:44.780680437Z [2020-11-15 12:15:44,754] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
*********
Pulling docker image mcr.microsoft.com/dotnet/core/sdk:3.1-alpine ...
Using docker image sha256:b0c526e8732fdcf06a1cc277f04523b4d3f10a6554d2b9df855e683524ee7ddf for mcr.microsoft.com/dotnet/core/sdk:3.1-alpine with digest mcr.microsoft.com/dotnet/core/sdk@sha256:3982ac41d8777b78ad7a2efe4c9674338975ebf9a25eeceb943348e45edf91b1 ...
Preparing environment
00:01
ERROR: Job failed (system failure): prepare environment: Error response from daemon: Cannot link to a non running container: /runner-z3wu8uu--project-20752619-concurrent-0-5f56b4323d8253b2-confluentinc__cp-kafka-1 AS /runner-z3wu8uu--project-20752619-concurrent-0-5f56b4323d8253b2-predefined-0/kafka (docker.go:817:0s).

Is the error caused by the kafka container startup trying to connect to the kafka container that I set in `KAFKA_ADVERTISED_LISTENERS``` variable. For example, it is failing to connect to kafka:29092 because the container has not started yet?

2

Solved, eventually figured out. Hope this helps others encountering similar issue. I ended up using confluent/cp-kafka images.

Created a small repository here that demonstrates using ASP.NET WebApp that spawns Kafka Background Service to create a Kafka topic on startup.

---
version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - camnet
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_LOG4J_ROOT_LOGLEVEL: WARN

  kafka:
    image: confluentinc/cp-kafka:6.0.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    networks:
      - camnet
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_HEAP_OPTS: -Xmx512M -Xms512M
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_LOG4J_LOGGERS: "org.apache.zookeeper=WARN,org.apache.kafka=WARN,kafka=WARN,kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN"
  
  netclient-test:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: netclient-test
    image: dcs3spp/netclient
    networks:
      - camnet
    depends_on:
      - kafka
      - netclient-run
    entrypoint: []
    command:
      - bash
      - -c
      - |-
        echo 'Giving Kafka a bit of time to start up…'
        while ! nc -z kafka 9092;
        do
          sleep 1;
        done;

        echo 'Giving netclient-run a bit of time to start up…'
        while ! nc -z netclient-run 80;
        do
          sleep 1;
        done;

        echo .NET Client test container ready. Running test that uses WebApplicationFactory TestServer to start WebApp with KafkaAdmin background service
        echo This runs successfully in a local development environment on MacOS and Ubuntu Linux 16.04.
        echo This fails when running on a GitLab CI Server. It can be seen that the test server bootstraps the WebApp.....
        echo The KafkaAdmin background service blocks when requesting topic creation from the kafka service
        dotnet test --runtime linux-musl-x64 -c Release --no-restore  --nologo tests/KafkaAdmin.Kafka.IntegrationTests/

  netclient-run:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: netclient-run
    image: dcs3spp/netclient
    networks:
      - camnet
    depends_on:
      - kafka
    entrypoint: []
    command:
      - bash
      - -c
      - |-
        echo 'Giving Kafka a bit of time to start up…'
        while ! nc -z kafka 9092;
        do
          sleep 1;
        done;
        echo .NET Run Web App Ready. Starting WebApp that contains KafkaAdmin background service.
        dotnet run --runtime linux-musl-x64 -c Release --no-restore --project src/KafkaAdmin.WebApp/
    
networks:
  camnet: