Inside a pipeline, how to install Python package that has a dependency without any PyPI

GitLab CI/CD
1

One of my company’s teams doesn’t want to use GitLab’s package repository. Instead, they install Python packages directly from Git (see example below).

(test) me@machine:~$ pip install ourpackage@git+https://gitlab.ourcompany.com/ourpackage.git@main#egg=ourpackage
Collecting ourpackage@ git+https://gitlab.ourcompany.com/ourpackage.git@main#egg=ourpackage
  Cloning https://gitlab.ourcompany.com/ourpackage.git (to revision main) to /tmp/pip-install-pdutqfvy/ourpackage_d3dbe93f0469494daac950f94b133ce6
  Running command git clone --filter=blob:none --quiet https://gitlab.ourcompany.com/ourpackage.git /tmp/pip-install-pdutqfvy/ourpackage_d3dbe93f0469494daac950f94b133ce6
  Resolved https://gitlab.ourcompany.com/ourpackage.git to commit a2b578ccedf4cc81942152d29714ebedf0cc4604
  Preparing metadata (setup.py) ... done
Collecting ourpackagedependency
  Downloading ourpackagedependency-1.0.0-py2.py3-none-any.whl (100 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.5/100.5 KB 5.5 MB/s eta 0:00:00
Using legacy 'setup.py install' for ourpackage, since package 'wheel' is not installed.
Installing collected packages: ourpackagedependency, ourpackage
  Running setup.py install for ourpackage ... done
Successfully installed ourpackage-1.0 ourpackagedependency-2.0.2

We have another package, let’s call it anotherpackage, that has ourpackage as a dependency. Here’s a simplified setup.cfg for anotherpackage.

[metadata]
name = anotherpackage

[options]
include_package_data = True
install_requires =
    ourpackage @ git+https://gitlab.ourcompany.com/ourpackage.git@main#egg=ourpackage
package_dir =
    = src
packages = find_namespace:
zip_safe = False

[options.packages.find]
exclude =
    tests
where = src

You can do pip install . in the root dir of anotherpackage, and it works. The user gets a prompt for their GitLab credentials, and that’s okay for local development work. You can use git config credential.helper store to avoid the prompt.

However, none of this works in a GitLab pipeline that builds anotherpackage. I tried many different things, but no luck so far. Does anyone know how to make this work without using GitLab’s package repository?

2

How about going completely git with the dependency and include it as a git submodule?

As soon as you set

# all submodules need to be downloaded
variables:
    GIT_SUBMODULE_STRATEGY: recursive

In the yml you are good to go then.

1 Like