Unable to upgrade past 13.0.14

How to Use GitLab Upgrade
#1

My Private GitLab Server is currently on 13.0.14 (ad4adc9d0e1)

When I try to upgrade to the latest version of GitLab (Community Edition):

  root@gitlab:~# sudo apt-get install gitlab-ce
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  The following packages will be upgraded:
    gitlab-ce
  1 upgraded, 0 newly installed, 0 to remove and 128 not upgraded.
  Need to get 826 MB of archives.
  After this operation, 306 MB of additional disk space will be used.
  Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic/main amd64 gitlab-ce amd64 13.6.1-ce.0 [826 MB]
  Fetched 826 MB in 55s (14.9 MB/s)
  (Reading database ... 144138 files and directories currently installed.)
  Preparing to unpack .../gitlab-ce_13.6.1-ce.0_amd64.deb ...
  Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab.json.
  This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
  This file is used to check if any of the unsupported configurations are enabled,
  and hence require a working reconfigure before upgrading.
  Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
  dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_13.6.1-ce.0_amd64.deb (--unpack):
   new gitlab-ce package pre-installation script subprocess returned error exit status 1
  Errors were encountered while processing:
  /var/cache/apt/archives/gitlab-ce_13.6.1-ce.0_amd64.deb
  E: Sub-process /usr/bin/dpkg returned an error code (1)

/opt/gitlab/embedded/nodes/gitlab.json only includes the following contents:

{
  "name": "gitlab"
}

When I try to Reconfigure GitLab:

root@gitlab:~# sudo gitlab-ctl reconfigure
Starting Chef Client, version 14.14.29
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - package (0.1.0)
  - postgresql (0.1.0)
  - redis (0.1.0)
  - monitoring (0.1.0)
  - registry (0.1.0)
  - mattermost (0.1.0)
  - consul (0.1.0)
  - gitaly (0.1.0)
  - praefect (0.1.0)
  - letsencrypt (0.1.0)
  - nginx (0.1.0)
  - runit (4.3.0)
  - acme (4.1.1)
  - crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...

================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================

RuntimeError
------------
Only one web server (Puma or Unicorn) can be enabled at the same time!

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/puma.rb:27:in `only_one_allowed!'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/puma.rb:21:in `parse_variables'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:185:in `block in generate_config'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:183:in `each'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:183:in `generate_config'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:24:in `from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'

Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/puma.rb:

 20:      def parse_variables
 21:        only_one_allowed!
 22:      end
 23:
 24:      def only_one_allowed!
 25:        return unless Services.enabled?('unicorn') && Services.enabled?('puma')
 26:
 27>>       raise 'Only one web server (Puma or Unicorn) can be enabled at the same time!'
 28:      end
 29:
 30:      def workers(total_memory = Gitlab['node']['memory']['total'].to_i)
 31:        [
 32:          2, # Two is the minimum or web editor will no longer work.
 33:          [
 34:            Gitlab['node']['cpu']['total'].to_i,
 35:            worker_memory(total_memory).to_i,
 36:          ].min # min because we want to exceed neither CPU nor RAM

System Info:
------------
chef_version=14.14.29
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client


Running handlers:
There was an error running gitlab-ctl reconfigure:

Only one web server (Puma or Unicorn) can be enabled at the same time!

Running handlers complete
Chef Client failed. 0 resources updated in 11 seconds

So I can’t seem to Reconfigure GitLab now …

This guide recommends I try to upgrade to 13.4.3. Here is what I get when I attempt to:

root@gitlab:~# apt-get install gitlab-ce=13.4.3-ce.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  gitlab-ce
1 upgraded, 0 newly installed, 0 to remove and 128 not upgraded.
Need to get 801 MB of archives.
After this operation, 193 MB of additional disk space will be used.
Get:1 https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu bionic/main amd64 gitlab-ce amd64 13.4.3-ce.0 [801 MB]
Fetched 801 MB in 50s (16.2 MB/s)
(Reading database ... 144138 files and directories currently installed.)
Preparing to unpack .../gitlab-ce_13.4.3-ce.0_amd64.deb ...
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/gitlab.json.
This usually happens when your last run of `gitlab-ctl reconfigure` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_13.4.3-ce.0_amd64.deb (--unpack):
 new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ce_13.4.3-ce.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

What should I do now to upgrade past 13.0.14? Thankfully the GItLab Web GUI is up and seems to be working, I just can’t seem to upgrade at all.

#2

So I figured out how to resolve the Only one web server (Puma or Unicorn) can be enabled at the same time! message when I run sudo gitlab-ctl reconfigure:

I opened /etc/gitlab/gitlab.rb and added the following:

puma['enable'] = false

(It was originally Commented out, so I removed the Comment)

Unicorn is one of the processes that come up when I run sudo gitlab-ctl status, so I figured Puma was less essential.

Ran sudo gitlab-ctl reconfigure and it Reconfigured just fine this time!

I should also mention that a couple hours ago, I was also receiving “500 - Whoops, something went wrong on our end” over and over on my Website. When I ran sudo gitlab-ctl status, it kept showing Prometheus was down. Sometimes a simple restart of GitLab will resolve this issue (sudo gitlab-ctl restart), but this time it wasn’t. So I followed the below guide to remove 0-byte JSON files:

Now Prometheus is running and does not go down after a couple of seconds.

I haven’t tried updating from 13.0.14, I’m just thankful GitLab is coming up online again!

#3

Puma is (and has been since 12.9) the default webserver for GitLab, and support for running under unicorn is planned to be removed in 14.0. So what you’re doing might still be supported, but is deprecated and will require some attention in the next 6 months.

There no suggestions or recommendation in that page listing 13.4.3, it simply reflects when that page was last updated. And as there is a known security problem in 13.4.3 (13.4.5 was labelled a security release), upgrading to that would be weird. Unless you have a good reason for choosing another version you should upgrade to the newest version which is currently 13.6.1. The documentation suggests you should be able to do that directly from 13.0.14, but I believe I have seen advice to go via a 13.1.x version, and I have no experience upgrading anything older than 13.5.5 to 13.6.1.

1 Like
#4

Thank you for the reply @grove!

Thank you for the heads up! So I assume it would be best now (or should I wait?) to update /etc/gitlab/gitlab.rb with the following:

unicorn['enable'] = false
puma['enable'] = true

I had never heard of Puma until I upgraded to GitLab 13.X, but I had heard of Unicorn because it’s used with Discourse (one of the other VMs on my Server, and the very Forum Tool we’re using here!).

I originally planned to update to 13.4.3, and then to the latest, as per the recommended Upgrade Path here. Thankfully, just this moment, performed a sudo apt-get install gitlab-ce and my GitLab updated from 13.0.14 (ad4adc9d0e1) to 13.6.1 (1b6a590b197) (the latest as of this post).

I assume the reason I couldn’t update past 13.0.14 before was due to both Unicorn and Puma being enabled before? Anyways I’ll wait on your response as to whether it is worth enabling Puma and disabling Unicorn before I do so.

#5

I have no idea.

I do know that unicorn was replaced with puma in our omnibus installation, and I believe the release notes said something about you having to do something to keep running unicorn.

f you still have unicorn running and doing something, I have no idea if it is as easy to switch as you suggest.

.Henrik

#6

My installation was using unicorn until 13.0 when I upgraded at this point it switched to puma. For me it wasn’t a problem because in gitlab.rb I never had any uncommented configuration for unicorn or anything else, so the switch was very easy. Maybe prior to 13.x both were available, and was possible to use puma before it became the default.

Therefore, if you don’t have any unicorn configuration uncommented in gitlab.rb then you don’t need to make any changes. Otherwise, best would be to comment any unicorn config so that it’s no longer being used, and then there would be no need to set puma to true and unicorn to false.

the link above shows how to enable unicorn, when upgrading to 13.x did the switch to puma. But since it will disappear in the future, better just to stop using it to be honest. I believe puma will be faster than unicorn from what I read about it.

#7

I’d love to be able to upgrade, but so far I just can’t figure it out. I’m also stuck on 13.0.14 but am running the Omnibus docker. So far, I have figured out I need to prepare with the below, however it still doesn’t fully work.

  1. Go into docker (docker exec -it gitlab-ce bash)
  2. Delete the data.10 folder in the postgresql folder
  3. (Postgres deets in gitlab.rb file)
  4. In mine it had both users of Postgres and gitlab-psql with the latter being the one I think I used
  5. Go into postresql
  6. psql
  7. ALTER USER “gitlab” WITH SUPERUSER;

I try it from time to time, to see how much further I can get. It should be simple and I don’t understand why the upgrade scripts don’t work automatically, nor why I have to enable superuser which is not cool.

#8

So since I’ve updated to 13.6.1 … my Project Developers have been experiencing a number of what are called Git Smudge Errors. I’m beginning to wonder if the upgrade was the cause or not, or something else.

For details, see this StackOverflow question I posted. The one user to answer me back believes the Server is the issue, that it isn’t “check[ing] the size of the object that’s uploaded”. Maybe it’s completely unrelated to Puma/Unicorn and/or upgrading to 13.6.1?

As I mentioned to you before, I disabled Puma (and had Unicorn enabled) in order to get the gitlab-ctl reconfigure command to work (which I needed to before I could upgrade from 13.0.14 (ad4adc9d0e1) to 13.6.1 (1b6a590b197)). Once I did, I never changed my /etc/gitlab/gitlab.rb settings.

So recently I made these changes to gitlab.rb:

unicorn['enable'] = false
puma['enable'] = true

@grove do you think Puma being enabled (and Unicorn disabled) would be affecting the way Git Pulls are handled? Or do you think I need to revert to a previous GitLab version? Or do you think something else is going on? Keep in mind my Project’s master branch works just fine, it’s just the other branches that are experiencing Smudge Errors.

#9

I’ve never come across smudge errors, and I don’t know much about either unicorn or puma.

If you access your git repositories over SSH, puma/unicorn is not involved, but if you use HTTP(S) I would guess data goes through. And if data goes through something there is a, at least theoretical, risk it might be changed in unhealthy ways.

#10

So I resolved the issue: we ended up rolling back our GitLab VM to a time prior to a Git Push that had caused the GitLab VM to lock up (it took several attempts before the Git Branch was pushed at that time). After rolling back the GitLab VM, I added more RAM and CPU to the VM: the Git Branch was pushed again and this time without the GitLab VM locking up. Once successfully pushed, the branch could be pulled without error! So we no longer receive Smudge Errors when pulling the branch (or any other branch in the Git Project for that matter, as I’ll explain further).

We believe that ever since that event of the GitLab VM locking up during a Git Push, the Project Git Repository was corrupted. Indeed, anyone pulling from any of our Project Branches (minus master for some reason) were receiving Smudge Errors, including from previous branches that were pullable without error before! I am very thankful that regular snapshots and backups are performed on our Git Server.

Word of warning to anyone hosting a GitLab Server: if a big push is about to be attempted, snapshot AND backup the GitLab Server first. Trust me, you REALLY don’t want a corrupted Git repository!

1 Like