Gitlab CI fails with Error : 2002 Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock

dptks19 · April 21, 2018, 12:41pm

I am running a gitlab CI for my django project with Mysql DB.

And I am getting following error :

django.db.utils.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)"

My requirement.txt file is as follows :

Django==2.0.4
django-extensions==2.0.6
mysqlclient==1.3.12

Here is My CI Config file:

# This file is a template, and might need editing before it works on your project.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python
image: python:latest

# Pick zero or more services to be used on all builds.
# Only needed when using a docker container to run your tests in.
# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-a-service
services:
  - mysql:latest
 # - postgres:latest

variables:
  # POSTGRES_DB: database_name
  MYSQL_DB: database_name
  MYSQL_ALLOW_EMPTY_PASSWORD: "1"

# This folder is cached between builds
# http://docs.gitlab.com/ce/ci/yaml/README.html#cache
cache:
  paths:
  - ~/.cache/pip/

# This is a basic example for a gem or script which doesn't use
# services such as redis or postgres
before_script:
  - python -V
  # Print out python version for debugging
  # Uncomment next line if your Django app needs a JS runtime:
  # - apt-get update -q && apt-get install nodejs -yqq
  - pip install -r requirements.txt

# To get Django tests to work you may need to create a settings file using
# the following DATABASES:
#
# DATABASES = {
#     'default': {
#        'ENGINE': 'django.db.backends.postgresql_psycopg2',
#        'NAME': 'ci',
#        'USER': 'postgres',
#        'PASSWORD': 'postgres',
#        'HOST': 'postgres',
#        'PORT': '5432',
#    },
# }
#
# and then adding `--settings app.settings.ci` (or similar) to the test command

test:
  variables:
    # DATABASE_URL: "postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB"
    MYSQL_ALLOW_EMPTY_PASSWORD: "1"
    DATABASE_URL: mysql://root@mysql:3306
  script:
  - python manage.py test

And the whole output is:

Running with gitlab-runner 10.7.0-rc1 (a4699306)
  on docker-auto-scale 4e4528ca
Using Docker executor with image python:latest ...
Starting service mysql:latest ...
Pulling docker image mysql:latest ...
Using docker image sha256:8d65ec712c69a27e2b9064f2fef307849775687d270e9ab5b79fd17dcd31b16e for mysql:latest ...
Waiting for services to be up and running...
Pulling docker image python:latest ...
Using docker image sha256:6bf7a4fa2d455fae4ce7f38b0b6db33fc5c5ea537c252b0bab8a6338176cb81b for python:latest ...
Running on runner-4e4528ca-project-4802656-concurrent-0 via runner-4e4528ca-srm-1524305095-4dc1df06...
Cloning repository...
Cloning into '/builds/dptks19/..._devel'...
Checking out 3062aac8 as master...
Skipping Git submodules setup
Checking cache for default...
FATAL: file does not exist                         
Failed to extract cache
$ python -V
Python 3.6.5
$ pip install -r requirements.txt
Collecting Django==2.0.4 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/89/f9/94c20658f0cdecc2b6607811e2c0bb042408a51f589e5ad0cb0eac3236a1/Django-2.0.4-py3-none-any.whl (7.1MB)
Collecting django-extensions==2.0.6 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/1e/0e/1a5184f7adbd2d28c8329cffd3806c036901d2a6e790c058fc70259bd4da/django_extensions-2.0.6-py2.py3-none-any.whl (217kB)
Collecting mysqlclient==1.3.12 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/6f/86/bad31f1c1bb0cc99e88ca2adb7cb5c71f7a6540c1bb001480513de76a931/mysqlclient-1.3.12.tar.gz (89kB)
Collecting pytz (from Django==2.0.4->-r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl (510kB)
Collecting six>=1.2 (from django-extensions==2.0.6->-r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Building wheels for collected packages: mysqlclient
  Running setup.py bdist_wheel for mysqlclient: started
  Running setup.py bdist_wheel for mysqlclient: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/50/c7/31/81a516762c8e9324f2b1fdffc1e84b9f07224fe3707956f6e1
Successfully built mysqlclient
Installing collected packages: pytz, Django, six, django-extensions, mysqlclient
Successfully installed Django-2.0.4 django-extensions-2.0.6 mysqlclient-1.3.12 pytz-2018.4 six-1.11.0
$ python manage.py test
Creating test database for alias 'default'...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 236, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 17, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/test.py", line 26, in run_from_argv
    super().run_from_argv(argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/test.py", line 59, in handle
    failures = test_runner.run_tests(test_labels)
  File "/usr/local/lib/python3.6/site-packages/django/test/runner.py", line 601, in run_tests
    old_config = self.setup_databases()
  File "/usr/local/lib/python3.6/site-packages/django/test/runner.py", line 548, in setup_databases
    self.parallel, **kwargs
  File "/usr/local/lib/python3.6/site-packages/django/test/utils.py", line 176, in setup_databases
    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/creation.py", line 54, in create_test_db
    self._create_test_db(verbosity, autoclobber, keepdb)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/creation.py", line 164, in _create_test_db
    with self._nodb_connection.cursor() as cursor:
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 236, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")
ERROR: Job failed: exit code 1

I am trying to run my django project with mysql, locally it is running fine, but when I push any of my code it failes in CI stage.

Any one, what goes wrong?

dptks19 · April 23, 2018, 4:16am

Do I need anything to add/update from the page : https://docs.gitlab.com/ee/ci/services/mysql.html

halfer · April 23, 2018, 3:12pm

I would guess the above is the problem. You need to connect via TCP, not a socket. You can only do the latter, I believe, if the database server is on the same machine. I would expect the MySQL service to be implemented as a Docker container, and thus a socket wouldn’t work.

I expect you might have to adjust your connection string.

dptks19 · April 24, 2018, 2:55am

I am using gitlab’s shared runners for CI. So how I can connect to mysql service as you said. Can you help me that?

halfer · April 24, 2018, 12:19pm

Not without doing it myself, no. I don’t actually use GitLab for CI. My guess is that you’d need to change the database connection string, or the Python database driver configuration, to force it to use TCP.

There seem to be some results here.

dptks19 · April 24, 2018, 1:07pm

I 'll try and let post here the result so, other can also get help if needed… Thanks.

nat.vachira · April 25, 2018, 2:52am

I could not manage to use Gitlab MySQL services as well. I found the complains in the forum long time ago and the solution there was not simple.

The work around was mentioned above, connected it through TCP. In my case, I decided to use docker. I have MySQL container running and connect it through docker network. If you already have a database running (maybe local in your test server), I believe that will also solve your problem.

ps. If you go on with docker solution, you will need a script to make sure Django is able to connect to MySQL or wait until MySQL initialization completed.

dptks19 · April 25, 2018, 2:49pm

So do I need to consider to have either any online server or my local server to be connected to gitlab CI to run mysql smoothly; just because of gitlab CI server(runners) might not have properly running mysql service and database created?

halfer · April 25, 2018, 6:51pm

No. You should always set up the database servers you need inside your CI infrastructure. You just need to persist to get it working. I am at risk of stating the obvious, but you could run some web searches for “GitLab Python MySQL connection” etc.

bart.goliszek · June 27, 2018, 9:27pm

Don’t use mysql:latest cause it refers to mysql 8.0 which brakes most currently used login methods.
You should be fine with mysql:5.7