Error with pip install from PyPI in self hosted Gitlab

General
,
1

Hello :slight_smile:,

so far I am quite happy with our self hosted Gitlab but I cannot pip install the release of a library I uploaded in our private PyPI (inside the library’s repo) with twine and a deploy key by following this tutorial.
The wheel definetely works (I have downloaded it and tested it in a fresh python environment) and so does the setup (it is not my first library, far from it).
I have looked many threads online with somewhat related issues (sorry I did not bother saving the links) but nothing that could resolve mine.

Error log

ERROR: Could not find a version that satisfies the requirement unidecode (from my-library) (from versions: none)
ERROR: No matching distribution found for unidecode (from my-library)
pip install with verbose (anonymyzed) 
Using pip 20.2.3 from /home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-bsz20c99
Created temporary directory: /tmp/pip-req-tracker-9ta9z3uz
Initialized build tracking at /tmp/pip-req-tracker-9ta9z3uz
Created build tracker: /tmp/pip-req-tracker-9ta9z3uz
Entered build tracker: /tmp/pip-req-tracker-9ta9z3uz
Created temporary directory: /tmp/pip-install-qk161tpl
Looking in indexes: https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple
1 location(s) to search for versions of my-library:
* https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/
Fetching project page and analyzing links: https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/
Getting page https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/
Found credentials in url for gitlab.self-hosted.dev
Starting new HTTPS connection (1): gitlab.self-hosted.dev:443
https://gitlab.self-hosted.dev:443 "GET /api/v4/projects/91/packages/pypi/simple/my-library/ HTTP/1.1" 200 None
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3.dev0-py3-none-any.whl#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2.3.dev0
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3.dev0.tar.gz#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2.3.dev0
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3-py3-none-any.whl#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2.3
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3.tar.gz#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2.3
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.2-py3-none-any.whl#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2.2
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2-py3-none-any.whl#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2
  Found link https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.tar.gz#sha256=123 (from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/my-library/), version: 2
Given no hashes to check 5 links for project 'my-library': discarding no candidates
Using version 2.3 (newest of versions: 2, 2.2, 2.3)
Collecting my_library
  Created temporary directory: /tmp/pip-unpack-_8nahw9n
  https://gitlab.self-hosted.dev:443 "GET /api/v4/projects/91/packages/pypi/files/123456/my_library-2.3-py3-none-any.whl HTTP/1.1" 200 22256
  Downloading https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3-py3-none-any.whl (22 kB)
  Added my_library from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3-py3-none-any.whl#sha256=123 to build tracker '/tmp/pip-req-tracker-9ta9z3uz'
  Removed my_library from https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/files/123456/my_library-2.3-py3-none-any.whl#sha256=123 from build tracker '/tmp/pip-req-tracker-9ta9z3uz'
1 location(s) to search for versions of unidecode:
* https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/unidecode/
Fetching project page and analyzing links: https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/unidecode/
Getting page https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/unidecode/
https://gitlab.self-hosted.dev:443 "GET /api/v4/projects/91/packages/pypi/simple/unidecode/ HTTP/1.1" 404 35
Could not fetch URL https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/unidecode/: 404 Client Error: Not Found for url: https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/unidecode/ - skipping
Given no hashes to check 0 links for project 'unidecode': discarding no candidates
ERROR: Could not find a version that satisfies the requirement unidecode (from my_library) (from versions: none)
ERROR: No matching distribution found for unidecode (from my_library)
Exception information:
Traceback (most recent call last):
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
    self._populate_link(req)
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
    req.link = self._find_requirement_link(req)
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
    best_candidate = self.finder.find_requirement(req, upgrade)
  File "/home/thibaultbtx/anaconda3/envs/test-env/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 926, in find_requirement
    raise DistributionNotFound(
pip._internal.exceptions.DistributionNotFound: No matching distribution found for unidecode (from my_library)
1 location(s) to search for versions of pip:
* https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/pip/
Fetching project page and analyzing links: https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/pip/
Getting page https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/pip/
Found credentials in url for gitlab.self-hosted.dev
Starting new HTTPS connection (1): gitlab.self-hosted.dev:443
https://gitlab.self-hosted.dev:443 "GET /api/v4/projects/91/packages/pypi/simple/pip/ HTTP/1.1" 404 35
Could not fetch URL https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/pip/: 404 Client Error: Not Found for url: https://gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple/pip/ - skipping
Given no hashes to check 0 links for project 'pip': discarding no candidates
Removed build tracker: '/tmp/pip-req-tracker-9ta9z3uz'

Note: For privacy reasons I have edited the name of the library to “my-library”.

Troubleshooting

pip looks at the wrong place

If you look into the verbose logs (see above) you will see that pip looks for libraries such as unidecode or ftfy inside of our self hosted repo instead of public PyPI.

the name of my library contains a dash but it should be an underscore

This might be due to my overzealous colleagues which always use a dash for the repository name although I tell them to use an underscore.
So instead of https://{MYGITLABACCOUNT}/my_library I have https://{MYGITLABACCOUNT}/my-library. But anywhere else (folder names, setup, code, …) it is an underscore (my_library).

Thanks in advance for your help :slight_smile:!

2

Have you tried upgrading unidecode ?

 pip install unidecode upgrade
3

Hello @alhemicar,

the problem is not unidecode, it is any dependency (see in my post above the section “pip looks at the wrong place”). I have tried doing pip install unidecode prior to installing my library and then I get this output:

Requirement already satisfied: unidecode in ./anaconda3/envs/test-env/lib/python3.8/site-packages (from my-library) (1.1.1)
ERROR: Could not find a version that satisfies the requirement pandas (from my-library) (from versions: none)
ERROR: No matching distribution found for pandas (from my-library)

If I install all requirements listed in requirements.txt before installing the library then I can actually install my library with the command:

pip install my-library --index-url https://__token__:<TOKEN>@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple

I suppose this is kind of a workaround…

4

UPDATE: I tried installing all dependencies first and then installing the library which does work. But yeah this kinds of defeats the point of the wheel which already lists all requirements.

And there is another problem: it asks for my username and password so I cannot use this in continuous integration :frowning: (unless there was a way to provide username and password to the pip install command :thinking:).

6

Hello @jamesmike! Sorry to hear about that :frowning:. I may have a workaround but I don’t have time to investigate it right now. Basically here is how it works:

  • Install all the dependencies from requirements.txt
  • Prepare credentials via keyring (keyring is a credentials manager and new versions of pip support it)
  • Pip install as you normally would

If you feel like trying it and it works it would be nice if you give me feedback on the exact procedure :slight_smile: .

7

@jamesmike GREAT NEWS! I had again a case where I had interdependencies and managed to make my idea (see previous post) work in a CI pipeline! Here is the setup in the CI file:

  1. Prepare the following environment variables in your repo (Setting->CI/CD->Variables): PIP_URL, PIP_USERNAME, PIP_PASSWORD.
    Where PIP_URL is the URL for pip with the token! E.g. https://__token__:123456789@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple (assuming 123456789 is my token).

  2. Install the private library like this in your CI config file (remember to change my-library with whatever the name of your library is):

before_script:
  # usual python setup
  - python -V # Print out python version for debugging
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate
  # install my private dependency, we'll use keyring and Gitlab env variables to circumvent pip asking username and password
  # we'll also have to install the dependencies of my private dependency due to a Gitlab bug(?)
  - pip install pandas ftfy unidecode loguru nltk requests # dependencies of my private dependency
  - pip install keyring keyrings.alt # we have to install a backend for keyring
  - python -c "import keyring, os;keyring.set_password(service_name=os.environ['PIP_URL'], username=os.environ['PIP_USERNAME'], password=os.environ['PIP_PASSWORD'])"
  - pip install my-library --index-url "$PIP_URL"
9

@thibault.btx Hi ! same here…

i’m installing my own library but getting

pip install --force-reinstall accounts --index-url https://__token__:***@gitlab.self-hosted.dev/api/v4/projects/63/packages/pypi/simple
Looking in indexes: https://__token__:****@gitlab.self-hosted.dev/api/v4/projects/63/packages/pypi/simple
ERROR: Could not find a version that satisfies the requirement accounts (from versions: none)
ERROR: No matching distribution found for accounts

When I just go to https://gitlab.self-hosted.dev/api/v4/projects/63/packages/pypi/simple/ via browser i have:
{“error”:“404 Not Found”}

Is that a bug of gitlab ?

10

@thibault.btx
access token needs to have read_api permission along with read_registry

11

Hello @PekopT! This was an example. Please replace https://__token__:<TOKEN>@gitlab.self-hosted.dev/api/v4/projects/91/packages/pypi/simple with the link that you can find in Gitlab under Packages and registries for your package and replace <TOKEN> with a personal access token with API (or read_api) scope.

By the way sorry I forgot to add --upgrade in my workaround. Sadly I cannot update my post. This is very important to fetch the latest version (otherwise Gitlab will look at the cache to check if any version of the library is present).

Sources:

13

Actually, you can install your private package with the dependencies by using --index-url and --extra-index-url, like:

pip install \
--index-url https://$PERSONAL_ACCESS_TOKEN_NAME:$PERSONAL_ACCESS_TOKEN@gitlab.com/api/v4/projects/44/packages/pypi/simple \
--extra-index-url https://pypi.org/simple \
mypackage

After a little bit of struggling, I wrote a blog post about this: Publishing your Python package in your Gitlab Package Registry | Ana Paula Gomes

1 Like
14

Thanks again @anapaulagomes for your solution! Sadly there is still a big problem.

I accidentaly named one of my libraries after an existing open source library and this open source library got fetched instead of my private library :crying_cat_face: !
Apart from using my own old solution I have no idea how to fix that…

UPDATE: There is a note on that in the Gitlab docs
UPDATE2: And there is an issue on pip as well…