Gitlab runner Concurrrency

How to Use GitLab
#1

Using gitlab runner 12.4.1.

I have a single gitlab runner instance running with concurrency set to 8 for parallel jobs. The settings look like this (after remove some sensitive fields):

concurrent = 8
check_interval = 0

[session_server]
session_timeout = 1800

[[runners]]
limit = 1
output_limit = 32768
executor = “shell”
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]

[[runners]]
limit = 1
output_limit = 32768
executor = “shell”
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]

[[runners]]
limit = 1
output_limit = 32768
executor = “shell”
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]

[[runners]]
limit = 1
output_limit = 32768
executor = “shell”
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]

[[runners]]
limit = 1
output_limit = 32768
executor = “shell”
[runners.custom_build_dir]
[runners.cache]
[runners.cache.s3]

My .gitlab-ci.yml file looks like this:

Change pip’s cache directory to be inside the project directory since we can only cache local items.

variables:
PIP_CACHE_DIR: “$CI_PROJECT_DIR/.cache/pip”
DEPLOY_DIR: “/nfs/pdx/disks/mpe_tvpv_002/gitlab-runner”

cache:
paths:
- .cache/pip
- venv/

Setup.

before_script:

  • export PATH=/nfs/pdx/disks/mpe_tvpv_002/sw_tools/anaconda3/bin:${PATH}
  • unset PYTHONHOME # Make sure this is unset.
  • unset PYTHONPATH # Make sure this is unset.
  • python --version # NOTE: For debugging, do not remove.
  • virtualenv venv
  • source venv/bin/activate # Switch to local package environment.
  • pip install tox
  • cd python

Pipeline stages.

stages:

  • build
  • quality
  • sast
  • test
  • deploy

Build and check the source distribution.

check:
stage: build
script:
- tox -e check

Run quality.

flake8:
stage: quality
script:
- tox -e flake8

pylint:
stage: quality
script:
- tox -e pylint

py_sort_imports:
stage: quality
script:
- tox -e py_sort_imports

bandit:
stage: sast
script:
- tox -e bandit

Run unit tests.

TODO: we need to enable testing with coverage + reporting for that also.

.unit_tests:
stage: test
artifacts:
paths:
- python/tests_report.html
expire_in: 5 days

unit_tests_master:
extends: .unit_tests
script:
- tox -e py37-nocov
only:
- master

unit_tests_branch:
extends: .unit_tests
script:
- tox -e py37-nocov – pytest -vv --ignore=src -m “not slow”
only:
- branches
except:
- master

Per-commit builds on master for staging only.

staging:
stage: deploy
script:
- python setup.py sdist --format=gztar
- python setup.py bdist_wheel
- mkdir -p $DEPLOY_DIR/CI_ENVIRONMENT_NAME - mv dist/(python setup.py --name)-$(python setup.py --version).tar.gz
$DEPLOY_DIR/CI_ENVIRONMENT_NAME/(python setup.py --name)-(python setup.py --version)-{CI_PIPELINE_ID}-{CI_COMMIT_SHA:0:8}.tar.gz - mv dist/(python setup.py --name)-$(python setup.py --version)-py2.py3-none-any.whl
$DEPLOY_DIR/CI_ENVIRONMENT_NAME/(python setup.py --name)-(python setup.py --version)-py2.py3-none-any-{CI_PIPELINE_ID}-${CI_COMMIT_SHA:0:8}.whl
environment:
name: staging
only:
- master

Production releases (built only manually).

production:
stage: deploy
script:
- pip install python-semantic-release # For auto semantic versioning when doing production release.
- mkdir -p $DEPLOY_DIR/$CI_ENVIRONMENT_NAME
- python setup.py publish
- python setup.py bdist_wheel -d $DEPLOY_DIR/$CI_ENVIRONMENT_NAME
- python setup.py changelog --released > $DEPLOY_DIR/CI_ENVIRONMENT_NAME/CHANGELOG.(python setup.py --version).md
environment:
name: production
when: manual
only:
- master

When my pipelines are running, I am seeing lots of spurious errors like these:

    1. warning: failed to remove venv/bin/.nfs00000000dd8af2590000089b: Device or resource busy
    1. ERROR: InvocationError for command /nfs/pdx/disks/mpe_tvpv_002/gitlab-runner/builds/GUekZxcz/0/pgupta6/personal-work-repo/python/.tox/flake8/bin/flake8 src tests setup.py (exited with code 1)
    1. Bad interpreter.

I think this is might be due to race conditions or something. Possibly because I have concurrency set to 8. Is there a way to fix this? I am not sure how to resolve. Often I end up having to manually restart a lot of jobs on the pipelne. Then they pass OK. Thanks for any ideas.