CI with Container & Package Registry

GitLab CI/CD
1

Dear Community,

I have some difficulties to finalize CI pipelines with dependencies loaded from the GitLab Package and Container Registry.

Context

I am using GitLab.com and I would like add CI/CD to a project including:

  1. Base Docker image to create baseline for services (eg. specific Python version with dependencies);
  2. Python package as wheel build against base image created in 1. (custom package);
  3. Python service packaged as a Docker image using image from 1. and wheel from 2.;

Reading the GitLab doc, I have found how to create basic CI/CD to build docker images or python packages and I also have found basic recipe to publish on respective registry.

But I am facing two different problem when using Image and Package from the GitLab Registry. I describe below both issues.

Working base image

Step 1. Base image builds perfectly, I have no problem at this step. The image looks like:

FROM balenalib/rpi-raspbian:buster-20210924

ENV TZ=Etc/UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN apt-get update \
    && apt-get install -y \
    build-essential \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

RUN python3 -m pip install --upgrade pip && \
    python3 -m pip install --upgrade setuptools virtualenv

ENTRYPOINT ["python3"]

Issue #1 with Python wheel

Step 2. has the following CI pipeline:

stages:
  - build

build:
  stage: build
  #image: registry.gitlab.com/my/project/base-image:1.0
  image: python:3.7-buster
  script:
    - python3 -m pip install twine
    - python3 setup.py sdist bdist_wheel
    - >
      TWINE_PASSWORD=${REGISTRY_TOKEN} TWINE_USERNAME=${REGISTRY_USER}
      python3 -m twine upload --verbose
      --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

Everything work fine with python base image. But when I use the base image I have built, I have a cryptic error just after GitLab runner cloned the repository:

`/opt/venv/bin/python3: can't open file 'sh': [Errno 2] No such file or directory`

It seems to claim for sh file, but I never use it in any command of this project and when I challenge my image to check if this shell exists, it does.

In order to explore the complete process, lets build the wheel with Python classic image.

Issue #2 with Python service image

Step 3. has the following CI pipeline:

variables:
  TAG: "1.0"

stages:
  - build

build:
  stage: build
  image: docker:stable
  services:
    - docker:dind
  script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_TOKEN $CI_REGISTRY
    - >
      docker build
      --platform linux/arm/v6
      --tag $CI_REGISTRY_IMAGE:$TAG .
    - docker push $CI_REGISTRY_IMAGE:$TAG

Where the docker file contains the following line:

RUN python3 -m pip install --index-url https://$USER:$TOKEN@gitlab.com/api/v4/projects/<id>/packages/pypi/simple --no-deps base

On this pipeline I have some problem to get authenticated to the Package Registry, I have tried many combination to pass my credentials, I always get this kind of error:

Step 6/11 : RUN python3 -m pip install --index-url https://$USER:$TOKEN@gitlab.com/api/v4/projects/<id>/packages/pypi/simple --no-deps base
 ---> Running in 9c6365c46f2e
Looking in indexes: https://gitlab.com/api/v4/projects/<id>/packages/pypi/simple, https://www.piwheels.org/simple
User for gitlab.com: ERROR: Exception:
Traceback (most recent call last):
[...]
  File "/opt/venv/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 202, in ask_input
    return input(message)
EOFError: EOF when reading a line

Which seems to indicate credentials were not found and it asks for through stdin which off course does not return anything. Anyway I have tested with plain text URL encoded credentials (I know I have a leak here) I get the same result. So, I am missing some key point here because I just cannot get my credentials to be sent.

Interesting observation is that if use the link provided by the registry (when you explore your packages), eg.: https://gitlab.com/my/project/base/-/package_files//download. Then I can access the wheel without any authentication and the pipeline succeed. Not sure what I should understand from that.

Anyway this approach will break the flow anytime I want to update the package as it’s is specific to the pipeline which generated the package.

Questions

My questions summarize as follow:

  • Where this call to sh command comes from? And: How can I make this image able to execute my python setup.py to build the wheel as it does when using official Python image?
  • How should I pass credentials to GitLab package registry in order to be able to download a wheel when building an image requiring it? Or: What is the correct way to install with pip from Package Registry within a Docker image created by a CI pipeline?

Thank you for reading.
Let me know if I need to adapt my question to reach community standard or provide more information.
I am also open to any other way to address this challenge.

Best regards,

Jean