Few issues with today's update (gitlab-ce_14.5.0)

Today I’ve decided to update my gitlab instance, unfortunately I ran into problems, check this out:

root@behemoth.home.lan:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 972 MB of archives.
After this operation, 23.4 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu focal/main amd64 gitlab-ce amd64 14.5.0-ce.0 [972 MB]
Fetched 972 MB in 1min 28s (11.1 MB/s)
dpkg-deb (subprocess): decompressing archive member: internal gzip read error: '<fd:4>: invalid block type'
dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg-deb (subprocess): cannot copy archive member from '/var/cache/apt/archives/gitlab-ce_14.5.0-ce.0_amd64.deb' to decompressor pipe: failed to write (Broken pipe)
Reading changelogs... Done
(Reading database ... 249936 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_14.5.0-ce.0_amd64.deb ...
gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2021-11-23 13:45:06 +0200 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2021-11-23 13:45:09 +0200 -- done
2021-11-23 13:45:09 +0200 -- Dumping repositories ...
2021-11-23 13:45:09 +0200 -- [SKIPPED]
2021-11-23 13:45:09 +0200 -- Dumping uploads ...
2021-11-23 13:45:09 +0200 -- [SKIPPED]
2021-11-23 13:45:09 +0200 -- Dumping builds ...
2021-11-23 13:45:09 +0200 -- [SKIPPED]
2021-11-23 13:45:09 +0200 -- Dumping artifacts ...
2021-11-23 13:45:09 +0200 -- [SKIPPED]
2021-11-23 13:45:09 +0200 -- Dumping pages ...
2021-11-23 13:45:09 +0200 -- [SKIPPED]
2021-11-23 13:45:09 +0200 -- Dumping lfs objects ...
2021-11-23 13:45:09 +0200 -- [SKIPPED]
2021-11-23 13:45:09 +0200 -- Dumping container registry images ...
2021-11-23 13:45:09 +0200 -- [DISABLED]
Creating backup archive: 1637667909_2021_11_23_14.4.2_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
Deleting old backups ... done. (0 removed)
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.
gitlab preinstall: Automatically backing up /etc/gitlab
Running configuration backup
Creating configuration backup archive: gitlab_config_1637667910_2021_11_23.tar
Configuration backup archive complete: /etc/gitlab/config_backup/gitlab_config_1637667910_2021_11_23.tar
Keeping all older configuration backups
Unpacking gitlab-ce (14.5.0-ce.0) over (14.4.2-ce.0) ...
dpkg-deb (subprocess): decompressing archive member: internal gzip read error: '<fd:4>: invalid block type'
dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_14.5.0-ce.0_amd64.deb (--unpack):
 cannot copy extracted data for './opt/gitlab/embedded/bin/gitlab-workhorse' to '/opt/gitlab/embedded/bin/gitlab-workhorse.dpkg-new': unexpected end of file or stream
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)

Fortunately my Gitlab instance is not broken. but still.
Can you give me some advice how to fix it up?

Already tried apt-get clean, but it didn't help.

I tried to compare the md5sum, from this site:

and the downloaded file show different md5sum.

So I downloaded the file again from my Windows box, then compared the sum again, and it matched the sum shown in the website. Then I have uploaded this file into the Linux box and tried to install - the same issue. So the issue is not in the transaction.

The odd is that I really don’t know why the Linux box shows different md5sum.

root@behemoth.home.lan:~# md5sum gitlab-ce_14.5.0-ce.0_amd64.deb
eeed32ff34a7755952fa4eef56cde3a0 gitlab-ce_14.5.0-ce.0_amd64.deb

Even the downloaded file which on the Windows box shows the correct sum, trasfered to Linux shows different sum:

root@behemoth.home.lan:~# md5sum gitlab-ce_14.5.0-ce.0_amd64_win_download.deb
77bd18a41970e8a50d218bbf72c44db3 gitlab-ce_14.5.0-ce.0_amd64_win_download.deb


I just tested downloading directly to my Linux machine with wget, and md5sum matches. Potentially you have a problem copying between machines. Maybe download directly to the server you are wanting to update Gitlab on using wget like I did?

ian@elise:~/Downloads$ wget --content-disposition https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/focal/gitlab-ce_14.5.0-ce.0_amd64.deb/download.deb

--2021-11-24 10:52:32--  https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/focal/gitlab-ce_14.5.0-ce.0_amd64.deb/download.deb
Resolving packages.gitlab.com (packages.gitlab.com)...,
Connecting to packages.gitlab.com (packages.gitlab.com)||:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://d20rj4el6vkp4c.cloudfront.net/7/8/ubuntu/package_files/116100.deb?t=1637747852_9bf80617b3f9b2574880228f649e9f4132924b19 [following]
--2021-11-24 10:52:32--  https://d20rj4el6vkp4c.cloudfront.net/7/8/ubuntu/package_files/116100.deb?t=1637747852_9bf80617b3f9b2574880228f649e9f4132924b19
Resolving d20rj4el6vkp4c.cloudfront.net (d20rj4el6vkp4c.cloudfront.net)...,,, ...
Connecting to d20rj4el6vkp4c.cloudfront.net (d20rj4el6vkp4c.cloudfront.net)||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 972020304 (927M) [application/x-debian-package]
Saving to: ‘gitlab-ce_14.5.0-ce.0_amd64.deb’

gitlab-ce_14.5.0-ce.0_amd64 100%[========================================>] 926.99M  4.90MB/s    in 3m 15s  

2021-11-24 10:55:47 (4.76 MB/s) - ‘gitlab-ce_14.5.0-ce.0_amd64.deb’ saved [972020304/972020304]

ian@elise:~/Downloads$ md5sum gitlab-ce_14.5.0-ce.0_amd64.deb 
b059dbc3aad0c19ab79a5b471f0b7497  gitlab-ce_14.5.0-ce.0_amd64.deb
1 Like

Even the md5sum matched on Windows side, and I have transferred the file to the server, I’m unable to perform the upgrade.

Have you tried to upgrade?

I upgrade using the apt repositories I dont download manually and it worked fine. I am running 14.5.0 for two days now.

1 Like

Yeah, thanks @iwalker Seems that, the issue is only local.

I tried to download the deb package using wget on another machine on the same network, then calculated the md5sum and it matched. Then transferred the file to the gitlab server, again calculated the sum and it doesn’t match anymore…

Most probably issue with the storage, not in the transfer, because as I said the box I downloaded the file is in the same network segment as the gitlab server.

Could be, if you can boot to recovery and then run filesystem checks with fsck just in case. Usually should show something though in dmesg or /var/log/syslog if it was having problems with blocks, etc. Either that or the network card is playing up.

1 Like

I’m going to close this one.

I saw some errors on the message log related to IO, it seems like Proxmox issue, because the physical servers are fine, and the other machines are fine as well.