Pytest fails due to VTK import error when called by gitlab-ci, but not when called directly

GitLab CI/CD
1

We have a windows 10 machine running a gitlab runner version 13.4.1. Our repo has been configured with a gitlab-ci yml file that builds a new conda environment (including vtk version 8.1.0) and then runs a pytest unit tests in that environment.

Until a few days ago, this was working fine. Recently all our pipelines have been receiving some version of the same error that:

ImportError while importing test module 'C:\path\to\test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\path\to\virtualenv\lib\site-packages\vtk\vtkCommonCore.py:5: in <module>
    from .vtkCommonCorePython import *
E   ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:
C:\path\to\virtualenv\gitci-env\lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
Tests\test.py:4: in <module>
    import vtk
C:\path\to\virtualenv\lib\site-packages\vtk\__init__.py:7: in <module>
    from .vtkCommonCore import *
C:\path\to\virtualenv\lib\site-packages\vtk\vtkCommonCore.py:9: in <module>
    from vtkCommonCorePython import *
E   ModuleNotFoundError: No module named 'vtkCommonCorePython'
=========================== short test summary info ===========================
ERROR Tests\test.py

Other points worth noting:

  • This error does not occur if the virtual environment is invoked in a powershell and the unit tests run there. It can only be replicated through a gitlab-ci pipeline execution.
  • If a simple assert True unit test is run it will pass through gitlab-ci. If it is modified to import vtk, it will fail
  • Creating a named environment on the server and updating the yml file to use that existing environment rather than making a fresh one does not resolve the behaviour.
  • Machine restart didn’t resolve the issue

Does anyone have any insight on why this might happen or how to resolve it?
When logged into the machine that runs the gitlab runner, are there debug tools available for explore differences between gitlab-ci and my main environment?

2

I was able to work around this issue by taking an old conda environment I had previously made and updating gitlab-ci to use that environment. It is worth noting that gitlab-ci still will fail if it creates a fresh conda environment OR if I make a new named conda environment