Unable to reach hadoop service in GitLab CI: sucessful in local instance (gitlab-runner) but not in gitlab.com

runner
ci
docker
hadoop

#1

I have a hadoop environment with three docker containers:

  • hadoop namenode/datanode
  • spark master
  • spark worker

The environment is properly set as I can run any spark job on it. On top of that when I execute the gitlab-runner command, it ends with the expected result (job success). However in the official Gitlab CI site it ends with an error. The error is:

Call From hadoop/172.19.0.2 to hadoop:8020 failed on connection exception: java.net.ConnectException: Connection refused

The last simplification of the file looks like this one:

hadoop_and_spark_build_job:
  image: docker:git
  stage: build
  services:
  - docker:dind
  before_script:
  - cd environment
  - apk add bash py-pip
  - pip install docker-compose
  - docker login --username=$CI_GIT_USER --password=$CI_GIT_TOKEN registry.gitlab.com
  script:
  - docker-compose up -d sbd-worker # Pulls and run the hadoop and master containers opening the proper ports and launching the scripts of spark and hadoop
  - docker exec -i hadoop bash -c "pwd && ping -c 2 hadoop && ping -c 2 master && ping -c 2 sbd-worker"
  - docker exec -i hadoop bash -c "hadoop dfsadmin -report" # <<<<<<<<< Fails here <<<<< 
  - docker exec -i master bash -c "hadoop dfsadmin -report && mkdir -p /tmp"

I guess that the failure is related with some configuration from gitlab.com but I am really lost hereā€¦ any assistance would be appreciated. The sentence that fails is the last but one (it has the comment <<<<<<<<< Fails here <<<<< )

The command I use to check it locally is:

gitlab-runner exec docker --docker-privileged hadoop_and_spark_build_job

Thank you in advance.


#2

Well, I realized that the problems really reside in connecting the different services as they take a massive time to be up when they are in gitlab.com compared with the local execution.
Launching hadoop took up to 60 seconds compared to the 5 seconds that takes in my laptop with gitlab-runner.

I used a script to wait for the TCP port called wait-for-it.sh. The github repo is here.

In case someone run into the same problem, I have created a small repo with some files where I have tested this environment and there are docker-compose.yml and .gitlab-ci.yml files. I hope it can save you time.


#3

Thanks for sharing how you solved it!