Best practice for running tests with several docker containers (docker-compose, docker, api test and python tests)

GitLab CI/CD
1

Hey,

Context

We have a repository with several docker containers configured together as docker-compose services.

./
./service1
  Dockerfile
  [...]
./service2
  - Dockerfile
  - a_python_module_as_submodule/
    - setup.py
    - mypymodule/
      - __init__.py
      - [...]
docker-compose.yml
.gitlab-ci.yml
.gitmodules

What do we want to do?

  1. check that docker containers actually build
  2. perform tests using these services like:
    • some python unittest of module mypymodule in the service2 container
    • testing APIs etc…

How do we actually do it

After struggling with Gitlab CI for a while we managed to define 2 stages: the first, build that build the docker images and pushes it ; then, we may define several tests that pull those images and actually perform tests.

Note: some of those tests are run into the docker container corresponding to services, thus, it’s called by docker exec <CONTAINER> <TEST_COMMAND> e.g. docker exec service2 nose2 -v mypymodule

Here is the corresponding .gitlab-ci.yml:

image: docker/compose:1.27.4

variables:
  # When using dind service, you must instruct docker to talk with the
  # daemon started inside of the service. The daemon is available with
  # a network connection instead of the default /var/run/docker.sock socket.
  #
  # The 'docker' hostname is the alias of the service container as described at
  # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services
  #
  # If you're using GitLab Runner 12.7 or earlier with the Kubernetes executor and Kubernetes 1.6 or earlier,
  # the variable must be set to tcp://localhost:2375 because of how the
  # Kubernetes executor connects services to the job container
  # DOCKER_HOST: tcp://localhost:2375
  #
  DOCKER_HOST: tcp://docker:2375
  #
  # This instructs Docker not to start over TLS.
  DOCKER_TLS_CERTDIR: ""

  SERVICE2_CONTAINER: "service2"
  SERVICE2_IMAGE: "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$SERVICE2_CONTAINER:latest"
  SERVICE2_CONDITION: "SERVICE 2 Started and listenning on port"
  SERVICE1_CONTAINER: "service1"
  SERVICE1_IMAGE: "$CI_REGISTRY/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$SERVICE1_CONTAINER:latest"
  SERVICE1_CONDITION: "SERVICE 1 Started and listenning on port"

services:
  - name: docker:19.03.12-dind

before_script:
  - docker info
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build

  script:
    - docker-compose build service1 service2
    - docker-compose --env-file .env.test up -d service1 service2
    - docker tag $SERVICE1_CONTAINER:latest $SERVICE1_IMAGE
    - docker tag $SERVICE2_CONTAINER:latest $SERVICE2_IMAGE
    - docker push $SERVICE1_IMAGE
    - docker push $SERVICE2_IMAGE 

test:
  services:
    - name: docker:19.03.12-dind

  script:
    - docker run --env-file ./.env.test --name $SERVICE2_CONTAINER -d registry.gitlab.com/pltrdy/<repo>/$SERVICE2_CONTAINER
    - docker run --env-file ./.env.test --name $SERVICE1_CONTAINER -d registry.gitlab.com/pltrdy/<repo>/$SERVICE1_CONTAINER
    # Waiting for service to start
    - > 
      bash -c '
      until [[ $(docker logs $SERVICE1_CONTAINER 2>&1) == *"$LT_CONDITION"* ]];
      do
      echo "waiting on $SERVICE1_CONTAINER to boot...";
      sleep 4;
      done; echo "$SERVICE1_CONTAINER started!"'
    - echo "LT started"
    ###
    # Waiting for service to start
    - > 
      bash -c '
      until [[ $(docker logs $SERVICE2_CONTAINER 2>&1) == *"SERVICE2_CONDITION"* ]];
      do
      echo "waiting on $SERVICE2_CONTAINER to boot...";
      sleep 4;
      done; echo "$SERVICE2_CONTAINER started!"'
    ###
    # Test python package mypymodule from service2
    - docker exec $SERVICE2_CONTAINER bash -c 'echo "sample text to test" > /test.txt; cat /test.txt; mypymodule_bin /test.txt -model clf_rf6 -sc test'

Question: is that right / good practice?

This currently work, but I can see some drawbacks:

  • Pointless docker push/pull? with some docker images being quite big (~10G) it takes some time to push/pull and does not make a lot sense since we just builded it locally the stage before.
  • Pushing before actually testing It make sense to me to push the docker image so that we can pull it afterward for deployment etc. However, in our process we push it before actually testing it, resulting in potentially broken builds being pushed (therefore pointless or missleading).

I tried to look at how people are doing it but if found only two scenarios:

  • this precise scenario: maybe drawbacks are ignored? or less of a matter (smaller images?)
  • using created images a services as declared in the job in .gitlab-ci.yml however then:
    • services are running, but I cannot access it from docker i.e. performing docker run service2 ... for testing

Thank you for reading!

Would you have advices of best practices/examples of similar situations?

Thanks a lot