Using private packages as dependencies

I have a python package released into Gitlab package registry, and it’s working perfectly fine. Now I want to use this as a dependency of another python package.

While I try to specify just the package name in setup.cfg, I am unable to install the new package as pip could not find the private package. I tried specifying the index URL in dependency_links, but in vain. Specifying the token directly in the same line as dependency works, but that means I can not commit setup.cfg which is not possible.

So, I want some guidance on how to use private package from Gitlab registries in other packages.

Note: More description can be found here.

We configure pip with an “extra index URL” wherever we want to install a package from out GitLab package repo. I recommend doing this via an environment variable, e.g. in your CI file or in a system config file, but you could also do it in a pip.conf file, e.g ./etc/pip.conf.

Our /etc/profile.d/pip.sh looks like this and allows all users of our Linux machines to install any of our packages with a simple pip install ....

export PIP_EXTRA_INDEX_URL=https://gitlab+deploy-token-123456:ABCDE-abcde12345txyz@gitlab.com/api/v4/groups/9876543/-/packages/pypi/simple/
1 Like

Hi @carlfischer1, thanks for the reply.

Can you please explain a bit more on how will you use it? Say dep is the package already released, I want to use it in main. How do I specify it in main’s setup.cfg somehow so that users of the main need not bother about token of dep, and that of main is sufficient?

Here, please assume that users of main do not have access to any of the two repositories at all, and I have to share a read-only token for them to install main.

Also, in case access token of dep needs to be provided, it’ll be very helpful if you give a bit more details on using that.

Sorry, I do not see anything here. Can you please share the format again?

Sorry about the missing code snippet. Bad formatting on my part. Corrected above.

Our dependencies are all given as simple package names in our requirements.txt (with version constraints where applicable). Most of the packages are available from pypi.org and pip will find them automatically. A few are on our internal GitLab package repository and pypi.org knows nothing about them. Trying to install these will fail unless you tell pip to use your private repository by configuring an extra index as described in my previous comment.

To avoid users having to provide their personal access token when installing packages, we’ve defined a deploy token with read-only access to our package registry (actually it’s at the group level so it works for all projects). This is the token we use in the index URL defined in the PIP_EXTRA_INDEX_URL environment variable that we define on all our machines. Users can then pip install any of our packages without needing to worry about tokens (because they’re defined globally on all machines in the company).

If you want to let people outside your company install your package, it’s a bit more difficult because you can’t configure their machines. But you can tell them to run a slightly different command. Instead of telling them to run pip install mypackage, you tell them to run PIP_EXTRA_INDEX_URL=https://...@gitlab.com/... pip install mypackage.

It looks like you can also provide the index URL (including the token) in your requirements.txt file by preceding the package name with the index: --extra-index-url https://...@gitlab.com/... mypackage. Credit to this article. I don’t know how this translates to a setup.cfg or setup.py or pyproject.toml file. Personally I wouldn’t include any tokens in the package itself. If you ever need to revoke the token you will need to issue a new version of the package. If you never need to revoke the token then why not just put your package on PyPI and make it public.

This StackOverflow question discusses how to install packages from a git repository which might be relevant to you, although it’s different from a private package repository.

1 Like

Thanks a lot for your detailed answer :slight_smile:

I do not want token to be part of package either, but I faced that I could not build the wheel (using python -m build) without checking all dependencies being present. I can set up the environment myself before, but the external users (which have no access to this registry whatsoever) can not do that, so I was looking for a way for them to do this by passing tokens “somehow” through environmental variables (not sure about feasibility) and have both dep and main installed in a hassle free way: pip install main --index-url <URL> or pip install main-<VERSION>.whl if I share the wheel directly. Let me try out PIP_EXTRA_INDEX_URL and see if it helps.

Making the package public will make life easy for everyone, but unfortunately it’s against company plans, even though users of the package will be able to see the package scripts anyway. I don’t have much say in that, but I think I can make it a bit easier for us by setting the group structure you suggested relying on one token only, as in case someone else want to use main as a dependency, it’ll be a pain.

An extra index allows you to use both your own packages and ones from pypi.org. If you just define the index then you will no longer be able to install most other packages unless your new package registry (index) also acts as a mirror or proxy for the public packages (I don’t think GitLab does this, you would need to host something like devpi).

If you define the same environment variable during package build, your build process should be able to access the dependencies too.

Essentially, if your package is private you have to provide credentials in order to install it, by definition. I don’t see a way around that. But using the extra index environment variable or config parameter or providing wheels for the private packages as you propose seem like pretty easy options. We’ve done both successfully and people inside and outside of the company have been able to copy-paste two or three commands to get the packages installed.