Debian registry / repository not working?

How to Use GitLab Self-managed
, ,
1

Hello !

I’m trying to host Debian packages on my gitlab without success.
I tried with gitlab 15.4.1-ee & 15.4.2 with the same results : upload seems ok but package isn’t present. The same behaviour and issue described here : Debian repository - #5 by pmhahn

I’ve created a distribution (called “stable”) and uploaded the package :

Package stays in status incoming. Here’ what I did :

$ curl "http://localhost/api/v4/projects/2/debian_distributions?codename=stable" --request POST --header "PRIVATE-TOKEN: <TOKEN>" -I 

HTTP/1.1 201 Created

$ curl "http://localhost/api/v4/projects/2/debian_distributions/stable" --request GET --header "PRIVATE-TOKEN: <TOKEN>" | jq .
{
  "id": 3,
  "codename": "stable",
  "suite": null,
  "origin": null,
  "label": null,
  "version": null,
  "description": null,
  "valid_time_duration_seconds": null,
  "components": [
    "main"
  ],
  "architectures": [
    "all",
    "amd64"
  ]
}

$ file varnish.deb 
varnish.deb: Debian binary package (format 2.0), with control.tar.zs, data compression zst

$ curl --request PUT \
     --upload-file ./varnish.deb \
     --header "PRIVATE-TOKEN: <TOKEN>" \ 
     "http://<USER_TOKEN>:<TOKEN>@localhost/api/v4/projects/2/packages/debian/" -I

HTTP/1.1 100 Continue
HTTP/1.1 201 Created

No packages listed :

$ curl http://localhost/api/v4/projects/2/packages --request GET --header "PRIVATE-TOKEN: <TOKEN>"   
[]

But if I try with an id :

$ curl http://localhost/api/v4/projects/2/packages/1 --request GET --header "PRIVATE-TOKEN: <TOKEN>"  | jq .                        

{
  "id": 1,
  "name": "incoming",
  "version": null,
  "package_type": "debian",
  "status": "default",
  "_links": {
    "web_path": "/gitlab-instance-377ddb81/debian_repo/-/packages/1",
    "delete_api_path": "http://my.fqdn/api/v4/projects/2/packages/1"
  },
  "created_at": "2022-10-17T12:27:38.824Z",
  "last_downloaded_at": null,
  "tags": [],
  "versions": []
}

I tried with dput too, with no more success :

$ dput --config=dput.cf --unchecked --no-upload-log gitlab ./varnish.changes

Uploading varnish using http to gitlab (host: debian_repo:<TOKEN>@localhost; directory: /api/v4/projects/2/packages/debian)
running allowed-distribution: check whether a local profile permits uploads to the target distribution
running checksum: verify checksums before uploading
running suite-mismatch: check the target distribution for common errors
running gpg: check GnuPG signatures before the upload
Not checking GPG signature due to allow_unsigned_uploads being set.
Not writing upload log upon request
Uploading varnish_6.0.10.3.dsc
Uploading varnish_6.0.10.3.tar.gz
Uploading varnish.changes

Did some of you already meet this issue ?

2

Yes, our on-premise setup of GitLab 15.4.2-ee shows the same behavior. So far may research has led to #340307 Fix debian background jobs.

Strangely enough I once had it working in an older version of GitLab, but then it broke and since then remains broken. I think it broken when we moved our instance to use S3 backing for the Package registry: I darkly remember seeing some failures when GitLab talked to S3. Other types of Package registries (Generic, Python) work fine, only Debian packages are failing.

3

Sad.
Thanks for the answer. I’ll move to Nexus then :confused:

4

gitlab-org/gitlab#340307 is fixed for some time now and my quick test today worked for 4 projects. Jay!

But on one other projects it fails:

  • dput succeeds; no error reported
  • But Gitlab’s Debian package registry then just shows this error:

There was an error publishing a debian-temporary-package package
There was a timeout and the package was not published. Delete this package and try again.

  • It only shows a debian-temporary-package with a :warning: Error publishing
  • The tooltip mentions:

Invalid Package: failed metadata extraction

So far I have been unable to find any detailed log output in our on-premis installation; any help is appreciated.

5

We’re encountering the same error here when the pipeline that does the upload is created by someone other than me (who created the Debian package repository). It seems a bit unclear who’s supposed to have the right to publish packages, but if a certain user should not be permitted to publish a certain package, the upload should of course be rejected rather than an unexpected error occurring.

6

If I remember correctly it was related to some dpkg-deb-error, which GitLab uses to extract the metadata from the .deb package file. The host on which GitLab was running was an old Debian-10-Buster system with dpkg version 1.19.8 lacking support for compression zstd, but the package I was trying to upload was built on Ubuntu 22.04, which uses zstd by default. Thus dpkg-deb failed to unpack the package and failed to extract the data. See #422584. You can easily check for this by running ar t on your package (Debian binary packages are ar files containing the flag file debian-binary and two compressed tar files named control.tar.* and data.tar.*):

$ ar t deb-zstd_1.0_all.deb
debian-binary
control.tar.zst
            ^^^
data.tar.zst
         ^^^

There you have it: zstd is being used, albeit with a different suffix. So check the version of dpkg-deb on your server.
Quoting from Debian packages in the package registry:

Prerequisites:

The dpkg-deb binary must be installed on the GitLab instance. This binary is usually provided by the dpkg package, installed by default on Debian and derivatives.

Support for compression algorithm ZStandard requires version dpkg >= 1.21.18 from Debian 12 Bookworm or dpkg >= 1.19.0.5ubuntu2 from Ubuntu 18.04 Bionic Beaver.

You can point Gitlab.config.packages.dpkg_deb_path to an alternative implementation, see !129761 for more details.