CI/CD ASP.NET Docker-compose stack works locally, but blocks/hangs in GitLab CI job

GitLab CI/CD
,
1

Hi,

I have a docker-compose stack, containing the following services within the same network (please see listing below):

  • Kafka + Zookeeper
  • netclient-run: ASP.NET Core WebApp that spawns a Background/Hosted Service that requests topic creation from the Kafka service. This is done using the confluent-kafka-dotnet client.
  • netclient-test: ASP.NET Core xUnit test that uses WebApplicationFactory to start a TestServer instance and create a Http Client. The Test Server bootstraps the WebApp.

Upon startup of the live WebApp the background/hosted service runs and successfully makes a request to the kafka service. This is successful locally and on the GitLab CI Server.

However the test blocks indefinelty on the Gitlab CI Server when the background service makes a request to kafka to create the topic. The same test runs successfully on local development environment. I am using WebApplicationFactory to spin-up a TestServer and create a HttpClient.

Does anyone have any ideas as to why the test is blocking on GitLab CI server?

I have created a small GitLab project to illustrate the issue that I am encountering.

Output from GitLab CI Server

netclient-run     | .NET Run Web App Ready. Starting WebApp that contains KafkaAdmin background service.
netclient-test    | Giving netclient-run a bit of time to start up…
netclient-run     | warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
netclient-run     |       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
netclient-run     | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
netclient-run     |       No XML encryptor configured. Key {395ba0f4-cde9-49af-8fb4-fd16b9f05bae} may be persisted to storage in unencrypted form.
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Admin service trying to create Kafka Topic...
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Topic::eventbus, ReplicationCount::1, PartitionCount::3
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Bootstrap Servers::kafka:9092
netclient-run     | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-run     |       Admin service successfully created topic eventbus
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Now listening on: http://[::]:80
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Application started. Press Ctrl+C to shut down.
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Hosting environment: Docker
netclient-run     | info: Microsoft.Hosting.Lifetime[0]
netclient-run     |       Content root path: /KafkaAdmin/src/KafkaAdmin.WebApp


netclient-test    | .NET Client test container ready. Running test that uses WebApplicationFactory TestServer to start WebApp with KafkaAdmin background service
netclient-test    | This runs successfully in a local development environment on MacOS and Ubuntu Linux 16.04.
netclient-test    | This fails when running on a GitLab CI Server. It can be seen that the test server bootstraps the WebApp.....
netclient-test    | The KafkaAdmin background service blocks when requesting topic creation from the kafka service
netclient-test    | Test run for /KafkaAdmin/tests/KafkaAdmin.Kafka.IntegrationTests/bin/Release/netcoreapp3.1/linux-musl-x64/KafkaAdmin.Kafka.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
netclient-test    | Starting test execution, please wait...
netclient-test    | 
netclient-test    | A total of 1 test files matched the specified pattern.
netclient-test    | warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
netclient-test    |       Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
netclient-test    | warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
netclient-test    |       No XML encryptor configured. Key {2b234f03-01b4-472d-9621-db8e056db173} may be persisted to storage in unencrypted form.
netclient-test    | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-test    |       Admin service trying to create Kafka Topic...
netclient-test    | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-test    |       Topic::eventbus, ReplicationCount::1, PartitionCount::3
netclient-test    | info: KafkaAdmin.Kafka.KafkaAdminService[0]
netclient-test    |       Bootstrap Servers::kafka:9092

GitLab Pipeline

stages:
  - test

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

services:
  - docker:dind
stages:
- test


test:
  stage: test
  image: docker/compose:alpine-1.27.4
  before_script:
    - docker version
    - docker-compose version
    - docker info
    - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - cd docker
    - docker-compose -f docker-compose.yml build
    - docker-compose -f docker-compose.yml up --exit-code-from netclient-test --abort-on-container-exit

Docker Compose Environment

---
version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.0.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - camnet
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_LOG4J_ROOT_LOGLEVEL: WARN

  kafka:
    image: confluentinc/cp-kafka:6.0.0
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    networks:
      - camnet
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_NUM_PARTITIONS: 3
      KAFKA_HEAP_OPTS: -Xmx512M -Xms512M
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_LOG4J_LOGGERS: "org.apache.zookeeper=WARN,org.apache.kafka=WARN,kafka=WARN,kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN"
  
  netclient-test:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: netclient-test
    image: dcs3spp/netclient
    networks:
      - camnet
    depends_on:
      - kafka
      - netclient-run
    entrypoint: []
    command:
      - bash
      - -c
      - |-
        echo 'Giving Kafka a bit of time to start up…'
        while ! nc -z kafka 9092;
        do
          sleep 1;
        done;

        echo 'Giving netclient-run a bit of time to start up…'
        while ! nc -z netclient-run 80;
        do
          sleep 1;
        done;

        echo .NET Client test container ready. Running test that uses WebApplicationFactory TestServer to start WebApp with KafkaAdmin background service
        echo This runs successfully in a local development environment on MacOS and Ubuntu Linux 16.04.
        echo This fails when running on a GitLab CI Server. It can be seen that the test server bootstraps the WebApp.....
        echo The KafkaAdmin background service blocks when requesting topic creation from the kafka service
        dotnet test --runtime linux-musl-x64 -c Release --no-restore  --nologo tests/KafkaAdmin.Kafka.IntegrationTests/

  netclient-run:
    build:
      context: ../
      dockerfile: docker/Dockerfile
    container_name: netclient-run
    image: dcs3spp/netclient
    networks:
      - camnet
    depends_on:
      - kafka
    entrypoint: []
    command:
      - bash
      - -c
      - |-
        echo 'Giving Kafka a bit of time to start up…'
        while ! nc -z kafka 9092;
        do
          sleep 1;
        done;
        echo .NET Run Web App Ready. Starting WebApp that contains KafkaAdmin background service.
        dotnet run --runtime linux-musl-x64 -c Release --no-restore --project src/KafkaAdmin.WebApp/
    
networks:
  camnet:

xUnit Test Stub

namespace KafkaAdmin.Kafka.IntegrationTests
{
    public class ApiControllerTest
    {
        [Fact]
        public void ApiControllerTest_Ping_Returns_Pong()
        {
            var appFactory = new WebApplicationFactory<WebApp.Startup>();

            /** THIS CODE HANGS ON GITLAB.COM CI BUT RUNS LOCALLY **/
            using (var client = appFactory.CreateClient())
            {
                Console.WriteLine("WE ARE IN THE TEST HERE");
            }
        }
    }
}
2

Update - Fixed

After reading this issue raised on aspnetcore GitHub, I discovered that the problem was with the implementation of my IHostedService implementation. In particular, the StartAsync method was performing the task, running until the request completed.

By design this method is meant to be fire and forget, i.e. start task and then continue. Still confused as to why the live WebApp starts successfully. Why is this just an issue for the TestServer?

Anyway, this was solved by updating my Kafka Service implementation from:

using System;
using System.Threading;
using System.Threading.Tasks;

using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Confluent.Kafka;
using Confluent.Kafka.Admin;

using KafkaAdmin.Kafka.Config;


namespace KafkaAdmin.Kafka
{
    public delegate IAdminClient KafkaAdminFactory(KafkaConfig config);
    public class KafkaAdminService : IHostedService
    {
        private KafkaAdminFactory _Factory { get; set; }
        private ILogger<KafkaAdminService> _Logger { get; set; }
        private KafkaConfig _Config { get; set; }


        /// <summary>
        /// Retrieve KafkaConfig from appsettings
        /// </summary>
        /// <param name="config">Config POCO from appsettings file</param>
        /// <param name="clientFactory"><see cref="KafkaAdminFactory"/></param>
        /// <param name="logger">Logger instance</param>
        public KafkaAdminService(
            IOptions<KafkaConfig> config,
            KafkaAdminFactory clientFactory,
            ILogger<KafkaAdminService> logger)
        {
            if (clientFactory == null)
                throw new ArgumentNullException(nameof(clientFactory));

            if (config == null)
                throw new ArgumentNullException(nameof(config));

            _Config = config.Value ?? throw new ArgumentNullException(nameof(config));
            _Factory = clientFactory ?? throw new ArgumentNullException(nameof(clientFactory));
            _Logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }


        /// <summary>
        /// Create a Kafka topic if it does not already exist
        /// </summary>
        /// <param name="token">Cancellation token required by IHostedService</param>
        /// <exception name="CreateTopicsException">
        /// Thrown for exceptions encountered except duplicate topic
        /// </exception>
        public async Task StartAsync(CancellationToken token)
        {
            using (var client = _Factory(_Config))
            {
                await CreateTopicAsync(client);
            }
        }

        /// <summary>Dispatch request to Kafka Broker to create Kafka topic from config</summary>
        /// <param name="client">Kafka admin client</param>
        /// <exception cref="">Thrown for errors except topic already exists</exception>
        private async Task CreateTopicAsync(IAdminClient client)
        {
            try
            {
                _Logger.LogInformation("Admin service trying to create Kafka Topic...");
                _Logger.LogInformation($"Topic::{_Config.Topic.Name}, ReplicationCount::{_Config.Topic.ReplicationCount}, PartitionCount::{_Config.Topic.PartitionCount}");
                _Logger.LogInformation($"Bootstrap Servers::{_Config.Consumer.BootstrapServers}");

                await client.CreateTopicsAsync(new TopicSpecification[] {
                        new TopicSpecification {
                            Name = _Config.Topic.Name,
                            NumPartitions = _Config.Topic.PartitionCount,
                            ReplicationFactor = _Config.Topic.ReplicationCount
                        }
                    }, null);

                _Logger.LogInformation($"Admin service successfully created topic {_Config.Topic.Name}");
            }
            catch (CreateTopicsException e)
            {
                if (e.Results[0].Error.Code != ErrorCode.TopicAlreadyExists)
                {
                    _Logger.LogInformation($"An error occured creating topic {_Config.Topic.Name}: {e.Results[0].Error.Reason}");
                    throw e;
                }
                else
                {
                    _Logger.LogInformation($"Topic {_Config.Topic.Name} already exists");
                }
            }
        }

        /// <summary>No-op</summary>
        /// <param name="token">Cancellation token</param>
        public async Task StopAsync(CancellationToken token) => await Task.CompletedTask;
    }
}

to:

using System;
using System.Threading;
using System.Threading.Tasks;

using Confluent.Kafka;
using Confluent.Kafka.Admin;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

using KafkaAdmin.Kafka.Config;


namespace KafkaAdmin.Kafka
{
    public delegate IAdminClient KafkaAdminFactory(KafkaConfig config);

    /// <summary>Background Service to make a request from Kafka to create a topic</summary>
    public class KafkaAdminService : BackgroundService, IDisposable
    {
        private KafkaAdminFactory _Factory { get; set; }
        private ILogger<KafkaAdminService> _Logger { get; set; }
        private KafkaConfig _Config { get; set; }


        /// <summary>
        /// Retrieve KafkaConfig from appsettings
        /// </summary>
        /// <param name="config">Config POCO from appsettings file</param>
        /// <param name="clientFactory"><see cref="KafkaAdminFactory"/></param>
        /// <param name="logger">Logger instance</param>
        public KafkaAdminService(
            IOptions<KafkaConfig> config,
            KafkaAdminFactory clientFactory,
            ILogger<KafkaAdminService> logger)
        {
            if (clientFactory == null)
                throw new ArgumentNullException(nameof(clientFactory));

            if (config == null)
                throw new ArgumentNullException(nameof(config));

            _Config = config.Value ?? throw new ArgumentNullException(nameof(config));
            _Factory = clientFactory ?? throw new ArgumentNullException(nameof(clientFactory));
            _Logger = logger ?? throw new ArgumentNullException(nameof(logger));
        }


        /// <summary>
        /// Create a Kafka topic if it does not already exist
        /// </summary>
        /// <param name="token">Cancellation token required by IHostedService</param>
        /// <exception name="CreateTopicsException">
        /// Thrown for exceptions encountered except duplicate topic
        /// </exception>
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            using (var client = _Factory(_Config))
            {
                try
                {
                    _Logger.LogInformation("Admin service trying to create Kafka Topic...");
                    _Logger.LogInformation($"Topic::{_Config.Topic.Name}, ReplicationCount::{_Config.Topic.ReplicationCount}, PartitionCount::{_Config.Topic.PartitionCount}");
                    _Logger.LogInformation($"Bootstrap Servers::{_Config.Consumer.BootstrapServers}");

                    await client.CreateTopicsAsync(new TopicSpecification[] {
                        new TopicSpecification {
                            Name = _Config.Topic.Name,
                            NumPartitions = _Config.Topic.PartitionCount,
                            ReplicationFactor = _Config.Topic.ReplicationCount
                        }
                    }, null);

                    _Logger.LogInformation($"Admin service successfully created topic {_Config.Topic.Name}");
                }
                catch (CreateTopicsException e)
                {
                    if (e.Results[0].Error.Code != ErrorCode.TopicAlreadyExists)
                    {
                        _Logger.LogInformation($"An error occured creating topic {_Config.Topic.Name}: {e.Results[0].Error.Reason}");
                        throw e;
                    }
                    else
                    {
                        _Logger.LogInformation($"Topic {_Config.Topic.Name} already exists");
                    }
                }
            }

            _Logger.LogInformation("Kafka Consumer thread started");

            await Task.CompletedTask;
        }


        /// <summary>
        /// Call base class dispose
        /// </summary>
        public override void Dispose()
        {
            base.Dispose();
        }
    }
}