Releasing to GitLab Package Registry with python-specific wheels
Hi,
I try to use GitLab CI to automate the package release process and store the releases in GitLab Package Registry. I have followed the documentation:
https://docs.gitlab.com/ee/user/packages/pypi_repository/
and managed to setup the CI so it builds and pushes wheels to the Package Registry.
However, I have faced an issue when there are several pipelines building different versions of the package, specific to different python versions. When I have more than 1 version of wheel in a given release, it seems that the Package Registry index is not correctly matching the versions and only the last file (literally last, following the upload/push the order) uploaded to the registry is available/can be installed via pip.
Here’s the setup - extension of the mypypipackage example from GitLab docs:
- in my setup.py I am “baking” python into the wheel:
import sys
import setuptools
# Set the required version of python.
__python_version__ = "=={}.{}.*".format(sys.version_info.major, sys.version_info.minor)
# Set python tag if building a wheel.
if "bdist_wheel" in sys.argv:
if not any(arg.startswith('--python-tag') for arg in sys.argv):
sys.argv.extend(['--python-tag', "py{}{}".format(sys.version_info.major, sys.version_info.minor)])
setuptools.setup(
name="mypypipackage",
version="0.1.3",
author="tkornuta",
description="A small example package",
packages=setuptools.find_packages(),
classifiers=[
'Programming Language :: Python :: {}.{}'.format(sys.version_info.major, sys.version_info.minor),
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires=__python_version__,
)
- I have set up the GitLab CI to build three wheels, for three different python versions/containers:
python36:
image: "urm.nvidia.com/docker/python:3.6"
script:
- python -c "import sys; print('{}.{}.*'.format(sys.version_info.major, sys.version_info.minor))"
- pip install twine
- python setup.py bdist_wheel
- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi dist/*
python37:
image: "urm.nvidia.com/docker/python:3.7"
script:
- python -c "import sys; print('{}.{}.*'.format(sys.version_info.major, sys.version_info.minor))"
- pip install twine
- python setup.py bdist_wheel
- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi dist/*
python38:
image: "urm.nvidia.com/docker/python:3.8"
script:
- python -c "import sys; print('{}.{}.*'.format(sys.version_info.major, sys.version_info.minor))"
- pip install twine
- python setup.py bdist_wheel
- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi dist/*
- When Ci completes, I can clearly see three versions of the wheel in Package Registry for this release (0.1.3):
- When installing “the latest uploaded” wheel (i.e. for python 3.8) file it works correctly:
(p38) tkornuta@aaa:~$ pip install mypypipackage --index-url https://<personal_access_token_name>:<personal_access_token>@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple
Looking in indexes: https://<personal_access_token_name>:****@@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple
Collecting mypypipackage
Downloading https://bbb/mypypipackage-0.1.3-py38-none-any.whl (1.6 kB)
Installing collected packages: mypypipackage
Successfully installed mypypipackage-0.1.3
- But when I try to install wheel with a different python version, e.g. in venv with python 3.7, pip throws an error:
(p37) tkornuta@aaa$ pip install mypypipackage==0.1.3 --index-url https://<personal_access_token_name>:<personal_access_token>@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple
Looking in indexes: https://<personal_access_token_name>:****@@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement mypypipackage==0.1.3
ERROR: No matching distribution found for mypypipackage==0.1.3
I have managed to successfully download and manually install the wheels for both python 3.6 and 3.7 versions - no problems here. I have also did some experiments and uploaded e.g. only one wheel for 3.7 - and managed to install it via pip straight from the Package Registry without any problems.
So what am I missing? What am I doing wrong?
Is this a problem with Package Registry that it cannot handle/serve many versions of a given wheel for a given release?
Thanks,
Tomasz