Connection refused on mysql service

Hello

I’m running a self hosted gitlab instance on a debian 9 server.
gitlab version : CE 13.1.4 (same on both my computer and the server)
gitlab-runner version : 13.1.1 (same on both my computer and the server)
docker version: Docker Engine Community 19.03.12 (same on both my computer and the server)

I’m having issues connecting to my services with docker only on the gitlab instance, not locally on my computer.
The runner on my local computer is not registered in the gitlab instance.

I made a specific job to demonstrate it is not a PHP or Doctrine related problem.
I also tried with the mysql image and with several versions the result is the same.
Obviously the environment variables are the same on the gitlab instance and in the command I use to run the job.

gitlab-ci.yml

test:mysql:
  stage: test
  image: mariadb:latest
  variables:
    MYSQL_DATABASE: "${RDS_DB_NAME}"
    MYSQL_ROOT_PASSWORD: "${RDS_PASSWORD}"
  services:
    - mariadb:latest
  script:
    - echo "mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME"
    - echo "SELECT 'OK';" | mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME

Job output :

Executing "step_script" stage of the job script
 $ echo "mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME"
 mysql --user=root --password=password --host=mariadb symfony
 $ echo "SELECT 'OK';" | mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME
 ERROR 2002 (HY000): Can't connect to MySQL server on 'mariadb' (115)

However when running the job on my local computer (Ubuntu) I do not encounter any issue :
gitlab-runner exec docker test:mysql --env RDS_DB_NAME=symfony [...]

Executing "step_script" stage of the job script
$ echo "mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME"
mysql --user=root --password=password --host=mariadb symfony
$ echo "SELECT 'OK';" | mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME
OK
OK

I’m out of ideas to figure out what’s going on.
I guess it’s Docker related but the versions are the same and I do not have any specific configuration.

Do you guys have any ideas ?
Thank you !

Hi,

maybe the MariaDB service takes a while to start and a sleep is necessary in the CI job?

Cheers,
Michael

I tried with MySQL and had the same issue.
I’ll try this anyway and let you know, thanks.

EDIT:
Still the error I’m afraid with a 60 seconds sleep:

Executing "step_script" stage of the job script
$ sleep 60
$ echo "mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME"
mysql --user=root --password=password --host=mariadb symfony
$ echo "SELECT 'OK';" | mysql --user=$RDS_USERNAME --password=$RDS_PASSWORD --host=$RDS_HOSTNAME $RDS_DB_NAME
ERROR 2002 (HY000): Can't connect to MySQL server on 'mariadb' (115)
ERROR: Job failed: exit code 1

Hello,

I faced the same issue and the workaround was to increase the duration of the sleep from 1 to 5 minutes.
So I suggest you try with sleep 300.

Regards.
Romain.