Deploy stage ends before Docker finishes starting

GitLab CI/CD
,
#1

Hi,

I’m setting up an AutoDevops pipeline on gitlab.com (Simplicite Software / gitlab-cicd · GitLab) with an additional automated UI testing (with Selenium) stage (uitest) between the staging stage and the production stage.

The thing is, my tests need the app to be fully started so they can work properly, so I don’t want to consider that a deploy job is finished before the app has finished warming up. As it can take some time, so I’ve added a HEALTHCHECK command to my Dockerfile

HEALTHCHECK --interval=30s --timeout=1s --start-period=300s CMD curl -sf http://localhost:8080/ping || exit 1

When I run my container locally, I have the correct states returned by the docker daemon:

> docker ps
CONTAINER ID   CREATED          STATUS                             
8d4d4d07ec6b   17 seconds ago   Up 16 seconds (health: starting)   
> docker ps
CONTAINER ID   CREATED         STATUS
8d4d4d07ec6b   7 minutes ago   Up 7 minutes (healthy)

I was expecting the staging stage to implement a health verification loop at the and to enable the job :

  • not to finish before the container is out of the starting state,
  • to succeed if the container falls in the healthy state,
  • to fail if the container falls in the unhealthy state,

but apparently gitlab doesn’t care about the container health.

I might have misconfigured something, but as a troubleshooting step, to exclude other factors that could make my uitest fail, I’ve added a curl healthcheck at the beginning of the script that runs the tests, which should systematically be positive: either the staging job fails and the uitest job should not run, or the staging job succeeds and the healthcheck can’t be negative.

TL;DR

The following would help me:

  1. confirmation that gitlab doesn’t check the container health before finishing a deploy job (I couldn’t find any such step in the Deploy template
  2. advice on how to customize the staging job to implement the described health verification loop.

Thank you for reading :+1:

#2

Anywhere else I can get help about this issue?

#3

I did find an unperfect temporary alternative by editing .gitlab/auto-deploy-values.yaml as per AutoDeploy FAQ. It’s unperfect because:

  • it doesn’t reuse the container health mechanism
  • not all of k8s probe configuration options are available, espacially, it’s missing failureThreshold, which forces to use a delay instead of retries.

Alternative options would be to use a custom helm template (or contribute to gitlab’s template)

livenessProbe:
  path: /ping
  initialDelaySeconds: 180

readinessProbe:
  path: /ping
  initialDelaySeconds: 180
#4

Backlink: Add startup probes to protect slow starting containers (!137) · Merge requests · GitLab.org / cluster-integration / auto-deploy-image · GitLab