CI/CD Inter-Service Networking

On the CI/CD documentation on using service, it is clear that the service can be accessed using the alias as the hostname.

However there is no documentation (or I cannot find) on inter-service networking. On the example below, the test code is able to access the database via dbhost and middleware via middlewarehost.

However the middleware cannot connect to the database. I’ve tried using the alias which is dbhost and localhost. Any body know how to connect services together?

I’m getting cannot resolve host when using the alias dbhost from the middleware service.

run-test:
  image: maven:3.3.9-jdk-8
  stage: test
  only:
  - master
  services:
  - name: database
    alias: dbhost
  - name: middleware
    alias: middlewarehost
  variables:
    DBPROFILE: integration-test
    MAX_ATTEMPTS: 30
    SPRING_PROFILES_ACTIVE: docker
  script:
    - mvn test -Dtest=**/**IT
  tags:
  - docker

I was able to replicate the problem in my local setup.

using command : gitlab-runner exec docker run-test.

I connected to the middleware container and I tried connecting using dbhost, container name, localhost, DBHOST (alias in caps), but nothing worked. I then tried the ip-address and it connected fine.

As a quick fix, I just workaround the problem by using the database container’s ip-address. But the test is using the alias.

Lastly, everything tells me that I am doing the right things, alias is working, containers are in the same network (since they can talk to each other via IP), but I guess the hostname resolution is the problem between services. Any ideas?

Hi @gitlab.padawan ,
I encountered the same problem and it surprises me that this issue hasn’t been fixed yet. There is another topic that indicates that people are struggling with this for almost 2 years now. imho service interconnection is an important “feature” one needs for testing.

Just like you I worked around the issue by using ip addresses. It turns out the runner assigns ip addresses starting from 172.17.0.2 incrementally. In your case dbhost would be reachable via 172.17.0.2 and middleware via 172.17.0.3

I don’t know how stable this approach is, especially if the runner is busy and there are many jobs running simultaneously.

Here is a minimal gitlab-ci config to reproduce this behavior:

stages:
  - test

mysql-test:
  stage: test
  services:
    - name: debian
      alias: debian
      command:
        - "sleep"
        - "60"
    - name: debian
      alias: deb-client-1
      command:
        - "ping"
        - "debian"
    - name: debian
      alias: deb-client-2
      command:
        - "ping"
        - "172.17.0.2"
  image: debian
  script:
    - ping debian -c 20
    - sleep 60

The first client fails

Service container logs:
2019-09-26T14:01:47.086531547Z ping: debian: Name or service not known

The container using the address to reach the target succeeds

Service container logs:
2019-09-26T14:01:48.010415693Z PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
2019-09-26T14:01:48.010497873Z 64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.136 ms

and the main environment can resolve the address:

[0K[32;1m$ ping debian -c 20[0;m
PING debian (172.17.0.2) 56(84) bytes of data.
64 bytes from debian (172.17.0.2): icmp_seq=1 ttl=64 time=0.157 ms

There is a related open issue on gitlab.com with many workarounds

There is now a feature flag to enable inter-service communication. Simply add the FF_NETWORK_PER_BUILD variable with a non null value to the job’s variables and they should be able to communicate now.

The reference issue. initially linked by _mark.

Hi @Progdrasil,

I am trying to get inter-service networking to work. However, even with setting the FF_NETWORK_PER_BUILD variable, the service containers are not able to reach another service container.

Any ideas on what I might be doing wrong?

I am still getting this error when backend tries to ensure database is up and running:

2020-12-31T11:30:44.710149815Z Ncat: Could not resolve hostname "database": Name or service not known. QUITTING.`

Thank you for your time and help :slight_smile:

Here’s the code in case it is easier than the link:

end-to-end-test:
  <<: *base
  image:
    name: cypress/included:6.1.0
    entrypoint: [""]
  services:
    - name: $DEVELOPMENT_CONTAINER_IMAGE
      alias: frontend
      command: ["scripts/start_app.sh"]
    - name: $PRODUCTION_CONTAINER_IMAGE
      alias: app
    - name: registry.gitlab.com/porter4/backend/master:latest
      alias: backend
      command: ["scripts/start_app.sh"]
    - name: postgres:13.0
      alias: database
  variables:
    DATABASE_HOST: database
    DATABASE_PORT: 5432
    POSTGRES_DB: porter_pipeline
    POSTGRES_USER: porter_app
    POSTGRES_PASSWORD: password
    GIT_STRATEGY: fetch
    CYPRESS_baseUrl: http://frontend:8080/
    FF_NETWORK_PER_BUILD: 1 # allow service containers to reach each other
    # https://forum.gitlab.com/t/ci-cd-inter-service-networking/19903/5
  before_script:
    - cd /builds/porter4/frontend/
    - ls -la
  script:
    - echo "production container specific testing is yet to come"
    - scripts/install_dependencies.sh
    - curl frontend:8080
    - curl app:80
    #- curl database:5432
    #- curl backend:8000
    - cypress run
  after_script:
    - cp -r tests/e2e/ /builds/porter4/frontend/.
  artifacts:
    paths:
      - e2e/
    when: always
  allow_failure: true
  timeout: 10 minutes