Error due to gitlab-runner cd in inexisting build directory

Summary

I have difficulties to setup CI:

  • self-managed GitLab Community Edition v13.0.0
  • gitlab-runner v13.0.1 running on OS X v10.15.4
  • executor virtualbox v6.1
  • vm debian buster amd64

First the runner uses old (cached?) vm instead of current vm, and second I face an issue becaus the runner tries to cd into builds/“group-id”/“project-id”, but this folder has never been created…

Configuration

The content of my config.toml:

concurrent = 1
check_interval = 0
log_level = “debug”

[session_server]
session_timeout = 1800

[[runners]]
name = “alhazen.local”
url = [MASKED]
token = [MASKED]
executor = “virtualbox”
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[runners.ssh]
user = “gitlab-ci”
password = [MASKED]
identity_file = “/Users/afalaize/.ssh/id_rsa”
[runners.virtualbox]
base_name = “debian_amd64_gitlabci”
disable_snapshots = true

The content of my .gitlab-ci.yml:

variables:
CI_DEBUG_TRACE: “true”

stages:

  • test

test:
stage: test
script: echo “Running tests”

The runner log from gitlab web interface:

e[0KRunning with gitlab-runner 13.0.1 (21cb397c)
e[0;me[0K on alhazen.local jXByepys
e[0;msection_start:1592152741:prepare_executor
e[0Ke[0Ke[36;1mPreparing the “virtualbox” executore[0;m
e[0;me[0KUsing VirtualBox version 6.1.10r138449 executor…
e[0;me[0KCreating new VM…
e[0;me[0KWaiting VM to become responsive…
e[0;me[0KStarting SSH command…
e[0;msection_end:1592152762:prepare_executor
e[0Ksection_start:1592152762:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;m
e[0;m+ set -eo pipefail

  • set +o noclobber
  • eval 'echo “Running on $(hostname) via alhazen.local…”
  • :
    +++ hostname
    ++ echo ‘Running on gitlabci-debian-amd64 via alhazen.local…’
    Running on gitlabci-debian-amd64 via alhazen.local…
  • exit 0
    ++ ‘[’ 1 = 1 ‘]’
    ++ ‘[’ -x /usr/bin/clear_console ‘]’
    ++ /usr/bin/clear_console -q
    section_end:1592152763:prepare_script
    e[0Ksection_start:1592152763:upload_artifacts_on_failure
    e[0Ke[0Ke[36;1mUploading artifacts for failed jobe[0;m
    e[0;m+ set -eo pipefail
  • set +o noclobber
  • eval ‘export FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION='\''false'\'' export FF_NETWORK_PER_BUILD=’’‘false’’’
    export FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY='\''true'\'' export FF_USE_DIRECT_DOWNLOAD=’’‘false’’’
    export CI_RUNNER_SHORT_TOKEN='\''jXByepys'\'' export CI_BUILDS_DIR=’’‘builds’’’
    export CI_PROJECT_DIR='\''builds/m2n_codes_collecte/lasie_rom'\'' export CI_CONCURRENT_ID=0 export CI_CONCURRENT_PROJECT_ID=0 export CI_SERVER=’’‘yes’’’
    mkdir -p “builds/m2n_codes_collecte/lasie_rom.tmp”
    echo -n $’’’-----BEGIN CERTIFICATE-----\n[MASKED]\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\n[MASKED]\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\n[MASKED]\n-----END CERTIFICATE-----’’’ > “$PWD/builds/m2n_codes_collecte/lasie_rom.tmp/CI_SERVER_TLS_CA_FILE”
    export CI_SERVER_TLS_CA_FILE="PWD/builds/m2n_codes_collecte/lasie_rom.tmp/CI_SERVER_TLS_CA_FILE" export CI_PIPELINE_ID=13024 export CI_PIPELINE_URL=’’‘https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom/pipelines/13024’’’
    export CI_JOB_ID=46762
    export CI_JOB_URL='\''https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom/-/jobs/46762'\'' export CI_JOB_TOKEN=’’’[MASKED]’’’
    export CI_BUILD_ID=46762
    export CI_BUILD_TOKEN='\''[MASKED]'\'' export CI_REGISTRY_USER=’’‘gitlab-ci-token’’’
    export CI_REGISTRY_PASSWORD='\''[MASKED]'\'' export CI_REPOSITORY_URL=’’‘https://gitlab-ci-token:[MASKED]@gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom.git’’’
    export CI_JOB_JWT='\''[MASKED]'\'' export CI_JOB_NAME=’’‘test’’’
    export CI_JOB_STAGE='\''test'\'' export CI_NODE_TOTAL=1 export CI_BUILD_NAME=’’‘test’’’
    export CI_BUILD_STAGE='\''test'\'' export CI=’’‘true’’’
    export GITLAB_CI='\''true'\'' export CI_SERVER_URL=’’‘https://gitlab.univ-lr.fr’’’
    export CI_SERVER_HOST='\''gitlab.univ-lr.fr'\'' export CI_SERVER_PORT=443 export CI_SERVER_PROTOCOL=’’‘https’’’
    export CI_SERVER_NAME='\''GitLab'\'' export CI_SERVER_VERSION=13.0.0 export CI_SERVER_VERSION_MAJOR=13 export CI_SERVER_VERSION_MINOR=0 export CI_SERVER_VERSION_PATCH=0 export CI_SERVER_REVISION=’’‘6743e883eae’’’
    export GITLAB_FEATURES=’’’’’’
    export CI_PROJECT_ID=210
    export CI_PROJECT_NAME='\''lasie_rom'\'' export CI_PROJECT_TITLE=’’‘lasie_rom’’’
    export CI_PROJECT_PATH='\''m2n_codes_collecte/lasie_rom'\'' export CI_PROJECT_PATH_SLUG=’’‘m2n-codes-collecte-lasie-rom’’’
    export CI_PROJECT_NAMESPACE='\''m2n_codes_collecte'\'' export CI_PROJECT_URL=’’‘https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom’’’
    export CI_PROJECT_VISIBILITY='\''private'\'' export CI_PROJECT_REPOSITORY_LANGUAGES=’’‘python,tex,jupyter notebook,glsl’’’
    export CI_DEFAULT_BRANCH='\''master'\'' export CI_PAGES_DOMAIN=’’‘example.com’’’
    export CI_PAGES_URL='\''http://m2n_codes_collecte.example.com/lasie_rom'\'' export CI_API_V4_URL=’’‘https://gitlab.univ-lr.fr/api/v4’’’
    export CI_PIPELINE_IID=27
    export CI_PIPELINE_SOURCE='\''push'\'' export CI_CONFIG_PATH=’’’.gitlab-ci.yml’’’
    export CI_COMMIT_SHA='\''991f06312f7d30dde6faa9d2fb90beaff45a8c34'\'' export CI_COMMIT_SHORT_SHA=’’‘991f0631’’’
    export CI_COMMIT_BEFORE_SHA='\''f6f3a9781b4af2ccd4592cd22b73932eb653daaa'\'' export CI_COMMIT_REF_NAME=’’‘temp’’’
    export CI_COMMIT_REF_SLUG='\''temp'\'' export CI_COMMIT_BRANCH=’’‘temp’’’
    export CI_COMMIT_MESSAGE='\''Update .gitlab-ci.yml'\'' export CI_COMMIT_TITLE=’’‘Update .gitlab-ci.yml’’’
    export CI_COMMIT_DESCRIPTION=’’’’’’
    export CI_COMMIT_REF_PROTECTED='\''false'\'' export CI_BUILD_REF=’’‘991f06312f7d30dde6faa9d2fb90beaff45a8c34’’’
    export CI_BUILD_BEFORE_SHA='\''f6f3a9781b4af2ccd4592cd22b73932eb653daaa'\'' export CI_BUILD_REF_NAME=’’‘temp’’’
    export CI_BUILD_REF_SLUG='\''temp'\'' export CI_RUNNER_ID=141 export CI_RUNNER_DESCRIPTION=’’‘alhazen.local’’’
    export CI_RUNNER_TAGS=’’’’’’
    export CI_DEBUG_TRACE='\''true'\'' export GITLAB_USER_ID=147 export GITLAB_USER_EMAIL=’’‘antoine.falaize@univ-lr.fr’’’
    export GITLAB_USER_LOGIN='\''afalaize'\'' export GITLAB_USER_NAME=’’‘afalaize’’’
    export CI_DISPOSABLE_ENVIRONMENT='\''true'\'' export CI_RUNNER_VERSION=13.0.1 export CI_RUNNER_REVISION=’’‘21cb397c’’’
    export CI_RUNNER_EXECUTABLE_ARCH='\''darwin/amd64'\'' ’’‘cd’’’ “builds/m2n_codes_collecte/lasie_rom”

    ++ export FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
    ++ FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
    ++ export FF_NETWORK_PER_BUILD=false
    ++ FF_NETWORK_PER_BUILD=false
    ++ export FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
    ++ FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
    ++ export FF_USE_DIRECT_DOWNLOAD=false
    ++ FF_USE_DIRECT_DOWNLOAD=false
    ++ export CI_RUNNER_SHORT_TOKEN=jXByepys
    ++ CI_RUNNER_SHORT_TOKEN=jXByepys
    ++ export CI_BUILDS_DIR=builds
    ++ CI_BUILDS_DIR=builds
    ++ export CI_PROJECT_DIR=builds/m2n_codes_collecte/lasie_rom
    ++ CI_PROJECT_DIR=builds/m2n_codes_collecte/lasie_rom
    ++ export CI_CONCURRENT_ID=0
    ++ CI_CONCURRENT_ID=0
    ++ export CI_CONCURRENT_PROJECT_ID=0
    ++ CI_CONCURRENT_PROJECT_ID=0
    ++ export CI_SERVER=yes
    ++ CI_SERVER=yes
    ++ mkdir -p builds/m2n_codes_collecte/lasie_rom.tmp
    ++ echo -n ‘-----BEGIN CERTIFICATE-----
    [MASKED]
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    [MASKED]
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    [MASKED]
    -----END CERTIFICATE-----’
    ++ export CI_SERVER_TLS_CA_FILE=/home/gitlab-ci/builds/m2n_codes_collecte/lasie_rom.tmp/CI_SERVER_TLS_CA_FILE
    ++ CI_SERVER_TLS_CA_FILE=/home/gitlab-ci/builds/m2n_codes_collecte/lasie_rom.tmp/CI_SERVER_TLS_CA_FILE
    ++ export CI_PIPELINE_ID=13024
    ++ CI_PIPELINE_ID=13024
    ++ export CI_PIPELINE_URL=https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom/pipelines/13024
    ++ CI_PIPELINE_URL=https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom/pipelines/13024
    ++ export CI_JOB_ID=46762
    ++ CI_JOB_ID=46762
    ++ export CI_JOB_URL=https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom/-/jobs/46762
    ++ CI_JOB_URL=https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom/-/jobs/46762
    ++ export CI_JOB_TOKEN=[MASKED]
    ++ CI_JOB_TOKEN=[MASKED]
    ++ export CI_BUILD_ID=46762
    ++ CI_BUILD_ID=46762
    ++ export CI_BUILD_TOKEN=[MASKED]
    ++ CI_BUILD_TOKEN=[MASKED]
    ++ export CI_REGISTRY_USER=gitlab-ci-token
    ++ CI_REGISTRY_USER=gitlab-ci-token
    ++ export CI_REGISTRY_PASSWORD=[MASKED]
    ++ CI_REGISTRY_PASSWORD=[MASKED]
    ++ export CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom.git
    ++ CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom.git
    ++ export CI_JOB_JWT=[MASKED]
    ++ CI_JOB_JWT=[MASKED]
    ++ export CI_JOB_NAME=test
    ++ CI_JOB_NAME=test
    ++ export CI_JOB_STAGE=test
    ++ CI_JOB_STAGE=test
    ++ export CI_NODE_TOTAL=1
    ++ CI_NODE_TOTAL=1
    ++ export CI_BUILD_NAME=test
    ++ CI_BUILD_NAME=test
    ++ export CI_BUILD_STAGE=test
    ++ CI_BUILD_STAGE=test
    ++ export CI=true
    ++ CI=true
    ++ export GITLAB_CI=true
    ++ GITLAB_CI=true
    ++ export CI_SERVER_URL=https://gitlab.univ-lr.fr
    ++ CI_SERVER_URL=https://gitlab.univ-lr.fr
    ++ export CI_SERVER_HOST=gitlab.univ-lr.fr
    ++ CI_SERVER_HOST=gitlab.univ-lr.fr
    ++ export CI_SERVER_PORT=443
    ++ CI_SERVER_PORT=443
    ++ export CI_SERVER_PROTOCOL=https
    ++ CI_SERVER_PROTOCOL=https
    ++ export CI_SERVER_NAME=GitLab
    ++ CI_SERVER_NAME=GitLab
    ++ export CI_SERVER_VERSION=13.0.0
    ++ CI_SERVER_VERSION=13.0.0
    ++ export CI_SERVER_VERSION_MAJOR=13
    ++ CI_SERVER_VERSION_MAJOR=13
    ++ export CI_SERVER_VERSION_MINOR=0
    ++ CI_SERVER_VERSION_MINOR=0
    ++ export CI_SERVER_VERSION_PATCH=0
    ++ CI_SERVER_VERSION_PATCH=0
    ++ export CI_SERVER_REVISION=6743e883eae
    ++ CI_SERVER_REVISION=6743e883eae
    ++ export GITLAB_FEATURES=
    ++ GITLAB_FEATURES=
    ++ export CI_PROJECT_ID=210
    ++ CI_PROJECT_ID=210
    ++ export CI_PROJECT_NAME=lasie_rom
    ++ CI_PROJECT_NAME=lasie_rom
    ++ export CI_PROJECT_TITLE=lasie_rom
    ++ CI_PROJECT_TITLE=lasie_rom
    ++ export CI_PROJECT_PATH=m2n_codes_collecte/lasie_rom
    ++ CI_PROJECT_PATH=m2n_codes_collecte/lasie_rom
    ++ export CI_PROJECT_PATH_SLUG=m2n-codes-collecte-lasie-rom
    ++ CI_PROJECT_PATH_SLUG=m2n-codes-collecte-lasie-rom
    ++ export CI_PROJECT_NAMESPACE=m2n_codes_collecte
    ++ CI_PROJECT_NAMESPACE=m2n_codes_collecte
    ++ export CI_PROJECT_URL=https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom
    ++ CI_PROJECT_URL=https://gitlab.univ-lr.fr/m2n_codes_collecte/lasie_rom
    ++ export CI_PROJECT_VISIBILITY=private
    ++ CI_PROJECT_VISIBILITY=private
    ++ export ‘CI_PROJECT_REPOSITORY_LANGUAGES=python,tex,jupyter notebook,glsl’
    ++ CI_PROJECT_REPOSITORY_LANGUAGES=‘python,tex,jupyter notebook,glsl’
    ++ export CI_DEFAULT_BRANCH=master
    ++ CI_DEFAULT_BRANCH=master
    ++ export CI_PAGES_DOMAIN=example.com
    ++ CI_PAGES_DOMAIN=example.com
    ++ export CI_PAGES_URL=http://m2n_codes_collecte.example.com/lasie_rom
    ++ CI_PAGES_URL=http://m2n_codes_collecte.example.com/lasie_rom
    ++ export CI_API_V4_URL=https://gitlab.univ-lr.fr/api/v4
    ++ CI_API_V4_URL=https://gitlab.univ-lr.fr/api/v4
    ++ export CI_PIPELINE_IID=27
    ++ CI_PIPELINE_IID=27
    ++ export CI_PIPELINE_SOURCE=push
    ++ CI_PIPELINE_SOURCE=push
    ++ export CI_CONFIG_PATH=.gitlab-ci.yml
    ++ CI_CONFIG_PATH=.gitlab-ci.yml
    ++ export CI_COMMIT_SHA=991f06312f7d30dde6faa9d2fb90beaff45a8c34
    ++ CI_COMMIT_SHA=991f06312f7d30dde6faa9d2fb90beaff45a8c34
    ++ export CI_COMMIT_SHORT_SHA=991f0631
    ++ CI_COMMIT_SHORT_SHA=991f0631
    ++ export CI_COMMIT_BEFORE_SHA=f6f3a9781b4af2ccd4592cd22b73932eb653daaa
    ++ CI_COMMIT_BEFORE_SHA=f6f3a9781b4af2ccd4592cd22b73932eb653daaa
    ++ export CI_COMMIT_REF_NAME=temp
    ++ CI_COMMIT_REF_NAME=temp
    ++ export CI_COMMIT_REF_SLUG=temp
    ++ CI_COMMIT_REF_SLUG=temp
    ++ export CI_COMMIT_BRANCH=temp
    ++ CI_COMMIT_BRANCH=temp
    ++ export ‘CI_COMMIT_MESSAGE=Update .gitlab-ci.yml’
    ++ CI_COMMIT_MESSAGE=‘Update .gitlab-ci.yml’
    ++ export ‘CI_COMMIT_TITLE=Update .gitlab-ci.yml’
    ++ CI_COMMIT_TITLE=‘Update .gitlab-ci.yml’
    ++ export CI_COMMIT_DESCRIPTION=
    ++ CI_COMMIT_DESCRIPTION=
    ++ export CI_COMMIT_REF_PROTECTED=false
    ++ CI_COMMIT_REF_PROTECTED=false
    ++ export CI_BUILD_REF=991f06312f7d30dde6faa9d2fb90beaff45a8c34
    ++ CI_BUILD_REF=991f06312f7d30dde6faa9d2fb90beaff45a8c34
    ++ export CI_BUILD_BEFORE_SHA=f6f3a9781b4af2ccd4592cd22b73932eb653daaa
    ++ CI_BUILD_BEFORE_SHA=f6f3a9781b4af2ccd4592cd22b73932eb653daaa
    ++ export CI_BUILD_REF_NAME=temp
    ++ CI_BUILD_REF_NAME=temp
    ++ export CI_BUILD_REF_SLUG=temp
    ++ CI_BUILD_REF_SLUG=temp
    ++ export CI_RUNNER_ID=141
    ++ CI_RUNNER_ID=141
    ++ export CI_RUNNER_DESCRIPTION=alhazen.local
    ++ CI_RUNNER_DESCRIPTION=alhazen.local
    ++ export CI_RUNNER_TAGS=
    ++ CI_RUNNER_TAGS=
    ++ export CI_DEBUG_TRACE=true
    ++ CI_DEBUG_TRACE=true
    ++ export GITLAB_USER_ID=147
    ++ GITLAB_USER_ID=147
    ++ export GITLAB_USER_EMAIL=[MASKED]
    ++ GITLAB_USER_EMAIL=[MASKED]
    ++ export GITLAB_USER_LOGIN=[MASKED]
    ++ GITLAB_USER_LOGIN=[MASKED]
    ++ export GITLAB_USER_NAME=[MASKED]
    ++ GITLAB_USER_NAME=[MASKED]
    ++ export CI_DISPOSABLE_ENVIRONMENT=true
    ++ CI_DISPOSABLE_ENVIRONMENT=true
    ++ export CI_RUNNER_VERSION=13.0.1
    ++ CI_RUNNER_VERSION=13.0.1
    ++ export CI_RUNNER_REVISION=21cb397c
    ++ CI_RUNNER_REVISION=21cb397c
    ++ export CI_RUNNER_EXECUTABLE_ARCH=darwin/amd64
    ++ CI_RUNNER_EXECUTABLE_ARCH=darwin/amd64
    ++ cd builds/m2n_codes_collecte/lasie_rom
    bash: ligne 90 : cd: builds/m2n_codes_collecte/lasie_rom: Aucun fichier ou dossier de ce type
    section_end:1592152763:upload_artifacts_on_failure
    e[0Ke[31;1mERROR: Job failed: Process exited with status 1
    e[0;m

Questions

  1. When registring the runner, which identity_file should be given: that of the host (osx) or that of the guest (debian vm)?
  2. The executor uses an old vm (gitlabci-debian-amd64) instead of the vm which name has been given at runner registration (debian_amd64_gitlabci). I tried to reset runner cach on gitlab web interface, to delete runner from project, to reset runners references and create/register new runner but it still uses old vm. Why is it so and how to force using current vm and not old vm?
  3. Finally, the runner tries to cd into builds/“group-id”/“project-id”, but this folder has never been created. I tried to create this folder in the vm but since the old vm is used this has no effect. What is appening here and how to solve this error?

Thank you in advance for your time.

Best regards;
afalaize

May be related to this issue (but I’m using a different executor)

Hi,

Since the macOS runner starts the VM and then initiates the communication via SSH, the runner’s identity file needs to be used. The public key needs to be added to the VM’s authorized_keys during creation (or via one time manual start and ssh-copy-id (you can get the latter via Homebrew).

When you edit the configuration file, you need to restart the runner service to reflect these changes. You cannot do that via the web interface though, as this needs to be done on the macOS host.

Maybe the runner registration was not entirely successful and the image definition was wrong/missing. Another thought would be that the base_image_name does not exist and then a fallback to default happens.

Question: How is the runner service being installed?

Good find. This may affect the macOS runner too with their home.

The mentioned workaround should be applicable as well with the bash logout, adding that to your macOS user’s bash profile. https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4449#workaround Could also be a permission issue with Catalina though.

Cheers,
Michael

Thank you for your reply!

Since the macOS runner starts the VM and then initiates the communication via SSH, the runner’s identity file needs to be used. The public key needs to be added to the VM’s authorized_keys during creation (or via one time manual start and ssh-copy-id (you can get the latter via Homebrew).

Yes, I doubt (and did) that. Maybe the documentation could be improved by few words highlighting this fact.

When you edit the configuration file, you need to restart the runner service to reflect these changes. You cannot do that via the web interface though, as this needs to be done on the macOS host.
Maybe the runner registration was not entirely successful and the image definition was wrong/missing. Another thought would be that the base_image_name does not exist and then a fallback to default happens.

Ok, thank you for these starting points. I’ll give a try next week.

Question: How is the runner service being installed?

I used homebrew.

Good find. This may affect the macOS runner too with their home.

Ok. I understand I have to dig in more seriously to understand what is happening.

Thank you for your time. I’ll post my findings ASAP.
Best regards;
Antoine

1 Like

Hi,

You can find the documentation at https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/install/osx.md - please go ahead, fork and create a merge request with your suggested changes :slight_smile:

Waiting for your findings then.

Cheers,
Michael