Dear all,
Currently i am in the process of automating testing of my python application which internally calls another application many times iteratively thereby optimizing my model parameters. This application within my python code is executed among multiple cores and therefore i was required to use ‘mpiexec’ commands with help from python os.system() funciton. This works OK if i run the code locally. Upon integration with Gitlab CI, mpiexec is not recognised by the runners. Freely available runners were chosen initially at the first instance. The execution log and the configuration file used were given here for your reference. I am pretty sure that the problem is related to the correct choice of runner specific to the docker image needed to build/test the application, howver being a newbie i am not able to figure out what could be done in this regard to select the right build environment for my application. Would be happy to hear from you any feedback/suggestions so that i can solve the first hurdle towards setting up my CI workflow.
Execution log:
Running with gitlab-runner 12.10.3 (2910048c)
on ci-test-x 9dc50af2
Preparing the “docker” executor
00:01
Using Docker executor with image python:latest …
Pulling docker image python:latest …
Using docker image sha256:5b3b4504ff1f7b859dbc5d7fb86f4afc644be62f99b8ced636fbca64c8a6c2de for python:latest …
Preparing environment
00:01
Running on runner-9dc50af2-project-5234-concurrent-0 via ci-test-x.apps.l…
Getting source from Git repository
Fetching changes with git depth set to 50…
Reinitialized existing Git repository in /builds/abalasub/moose_feaup/.git/
From https://git.ccfe.ac.uk/abalasub/moose_feaup
- [new ref] refs/pipelines/61686 → refs/pipelines/61686
bc6b72e…c4123f3 testing → origin/testing
Checking out c4123f31 as testing…
Removing venv/
Skipping Git submodules setup
Restoring cache
00:02
Downloading artifacts
00:01
Running before_script and script
00:24
$ python -V
Python 3.9.5
$ pip install virtualenv
Collecting virtualenv
Downloading virtualenv-20.4.7-py2.py3-none-any.whl (7.2 MB)
Collecting six<2,>=1.9.0
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting filelock<4,>=3.0.0
Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting distlib<1,>=0.3.1
Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
Collecting appdirs<2,>=1.4.3
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Installing collected packages: six, filelock, distlib, appdirs, virtualenv
Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 six-1.16.0 virtualenv-20.4.7
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
$ virtualenv venv
created virtual environment CPython3.9.5.final.0-64 in 367ms
creator CPython3Posix(dest=/builds/abalasub/moose_feaup/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ source venv/bin/activate
$ source venv/bin/activate
$ pip install pytest pytest-cov pandas matplotlib scipy
Collecting pytest
Downloading pytest-6.2.4-py3-none-any.whl (280 kB)
Collecting pytest-cov
Downloading pytest_cov-2.12.0-py2.py3-none-any.whl (20 kB)
Collecting pandas
Downloading pandas-1.2.4-cp39-cp39-manylinux1_x86_64.whl (9.7 MB)
Collecting matplotlib
Downloading matplotlib-3.4.2-cp39-cp39-manylinux1_x86_64.whl (10.3 MB)
Collecting scipy
Downloading scipy-1.6.3-cp39-cp39-manylinux1_x86_64.whl (27.3 MB)
Collecting attrs>=19.2.0
Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting toml
Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting pluggy<1.0.0a1,>=0.12
Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting packaging
Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
Collecting iniconfig
Downloading iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting py>=1.8.2
Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting coverage[toml]>=5.2.1
Downloading coverage-5.5-cp39-cp39-manylinux2010_x86_64.whl (243 kB)
Collecting numpy>=1.16.5
Downloading numpy-1.20.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
Collecting python-dateutil>=2.7.3
Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2017.3
Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting cycler>=0.10
Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.3.1-cp39-cp39-manylinux1_x86_64.whl (1.2 MB)
Collecting pyparsing>=2.2.1
Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pillow>=6.2.0
Downloading Pillow-8.2.0-cp39-cp39-manylinux1_x86_64.whl (3.0 MB)
Installing collected packages: pyparsing, toml, six, py, pluggy, packaging, iniconfig, coverage, attrs, pytz, python-dateutil, pytest, pillow, numpy, kiwisolver, cycler, scipy, pytest-cov, pandas, matplotlib
Successfully installed attrs-21.2.0 coverage-5.5 cycler-0.10.0 iniconfig-1.1.1 kiwisolver-1.3.1 matplotlib-3.4.2 numpy-1.20.3 packaging-20.9 pandas-1.2.4 pillow-8.2.0 pluggy-0.13.1 py-1.10.0 pyparsing-2.4.7 pytest-6.2.4 pytest-cov-2.12.0 python-dateutil-2.8.1 pytz-2021.1 scipy-1.6.3 six-1.16.0 toml-0.10.2
$ cd UnitTest
$ pytest -s -q test_soln_match.py
sh: 1: mpiexec: not found
Deleting <FEAUpdate.FEAUpdate object at 0x7f060d6b49a0>
Running after_script
00:02
Uploading artifacts for failed job
00:01
ERROR: Job failed: exit code 1
.gitlab-ci.yaml:
image: python:latest
before_script:
- python -V # Print out python version for debugging
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
test:
script:
- source venv/bin/activate
- pip install pytest pytest-cov pandas matplotlib scipy
- cd UnitTest
- pytest -s -q test_soln_match.py
artifacts:
paths:
- dist/*.whl
Kind regards,
Arun
