Upgrading from 13.12 to 14.0

I am not a linux (Ubuntu) expert. I have been using GitLab since version 12.5. Every 22nd of the month I do my upgrades using the command:

sudo apt-get update && sudo apt-get install gitlab-ee

I had a problem moving from 12.12 to 13.0 and after that everything run smoothly including the upgrade from 13.11 to 13.12. Now I am experiencing some problems again with upgrade from 13.12 to 14.0. I receive the following message:

"* unicorn['worker_processes'] has been deprecated since 13.10 and was removed in 14.0. Starting with GitLab 14.0, Unicorn is no longer supported and users must switch to Puma, following https://docs.gitlab.com/ee/administration/operations/puma.html."

Going to the documentation link, I did not see anything that would guide me to move from Unicorn to Puma, and by that I mean which commands should I use, or which files I have to change, what are the changes and where are the files located by default.
Is there any other guideline for dummies, providing a step by step procedure?
The four steps provided in the document are not clear to me:

  1. Determine suitable Puma worker and thread [settings]. → In the link provided, it is said that “The GitLab Linux package defaults to the recommended Puma settings.”, so I do not think I should worry about this step or change anything here.
  2. If you’re switching from Unicorn, [convert any custom settings to Puma]. → This is the one I think I should work on, but besides a table showing configuration keys to Unicorn and Puma, nothing else is said. I have no idea where to find configuration keys. Which is the file I should be looking for? Where is it located? Is this just a matter of changing “unicorn” name to “puma” and adding the missing keys? Just that change would make my GitLab to start using Puma instead of Unicorn?
  3. For multi-node deployments, configure the load balancer to use the [readiness check]. → I do not think this is a concern to me, as I do not have a multi-node deployment.
  4. Reconfigure GitLab so the above changes take effect. → Finally a step showing the command to achieve what we want…

Can anyone throw some light on my questions? I would really appreciate that.
Here you have the complete result of issuing the command I mentioned above:


or if the image is not good, here is the copy and paste:

/$ sudo apt-get update && sudo apt-get install gitlab-ee
[sudo] password for rmattos:
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Hit:5 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic InRelease
Fetched 252 kB in 1s (365 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.15.0-51 linux-headers-4.15.0-51-generic linux-image-4.15.0-51-generic linux-modules-4.15.0-51-generic
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  gitlab-ee
1 upgraded, 0 newly installed, 0 to remove and 38 not upgraded.
Need to get 0 B/991 MB of archives.
After this operation, 6,762 kB of additional disk space will be used.
(Reading database ... 258500 files and directories currently installed.)
Preparing to unpack .../gitlab-ee_14.0.0-ee.0_amd64.deb ...
* unicorn['worker_processes'] has been deprecated since 13.10 and was removed in 14.0. Starting with GitLab 14.0, Unicorn is no longer supported and users must switch to Puma, following https://docs.gitlab.com/ee/administration/operations/puma.html.
Deprecations found. Please correct them and try again.
dpkg: error processing archive /var/cache/apt/archives/gitlab-ee_14.0.0-ee.0_amd64.deb (--unpack):
 new gitlab-ee package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/gitlab-ee_14.0.0-ee.0_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

The easiest thing for you to do to switch from unicorn to puma is go into /etc/gitlab/gitlab.rb and edit this file. Find the lines with unicorn on them and place a # in front of the lines to comment them out.

Here is how my gitlab.rb looks, and this is one from at least Gitlab 12 or earlier:

root@repo:/etc/gitlab# cat gitlab.rb | grep -i unicorn
###! Time between sampling of unicorn socket metrics, in seconds
# gitlab_rails['monitoring_unicorn_sampler_interval'] = 10
## GitLab Unicorn
##! Tweak unicorn settings.
##! Docs: https://docs.gitlab.com/omnibus/settings/unicorn.html
# unicorn['worker_timeout'] = 60
# unicorn['worker_processes'] = 2
# unicorn['listen'] = '127.0.0.1'
# unicorn['port'] = 8080
# unicorn['socket'] = '/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
# unicorn['pidfile'] = '/opt/gitlab/var/unicorn/unicorn.pid'
# unicorn['tcp_nopush'] = true
# unicorn['backlog_socket'] = 1024
# unicorn['somaxconn'] = 1024
# unicorn['log_directory'] = "/var/log/gitlab/unicorn"
###! Docs: https://about.gitlab.com/2015/06/05/how-gitlab-uses-unicorn-and-unicorn-worker-killer/
###!       https://github.com/kzk/unicorn-worker-killer
# unicorn['worker_memory_limit_min'] = "400 * 1 << 20"
# unicorn['worker_memory_limit_max'] = "650 * 1 << 20"

that’s the only lines relating to unicorn at least in mine. Once commented out, then do:

gitlab-ctl reconfigure

now it should automatically start to use puma, since any lines relating to unicorn are now disabled. Then try your upgrade again.

As I said, your file might have different lines with unicorn on them, so just find them all and comment them out.

I don’t have any puma lines in my file so it’s using the defaults. Once you have it running, you can look at tuning puma later if you really need to.

3 Likes

As for puma config:

###! This value needs to be lower than the worker timeout set in unicorn/puma.
###! and lower than the worker timeout set in unicorn/puma. (default: 30)
## GitLab Puma
##! Tweak puma settings. You should only use Unicorn or Puma, not both.
##! Docs: https://docs.gitlab.com/omnibus/settings/puma.html
# puma['enable'] = true
# puma['ha'] = false
# puma['worker_timeout'] = 60
# puma['worker_processes'] = 2
# puma['min_threads'] = 4
# puma['max_threads'] = 4
# puma['listen'] = '127.0.0.1'
# puma['port'] = 8080
# puma['socket'] = '/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket'
# puma['somaxconn'] = 1024
# puma['pidfile'] = '/opt/gitlab/var/puma/puma.pid'
# puma['state_path'] = '/opt/gitlab/var/puma/puma.state'
# puma['log_directory'] = "/var/log/gitlab/puma"
###! Docs: https://github.com/schneems/puma_worker_killer
# puma['per_worker_max_memory_mb'] = 850
# puma['exporter_enabled'] = false
# puma['exporter_address'] = "127.0.0.1"
# puma['exporter_port'] = 8083
### This value should be the address at which Prometheus is available to GitLab Rails(Puma/Unicorn, Sidekiq) node.

pulled that from my clean installed 13.x test server, so you have some options to play with and add to your file if needed.

3 Likes

Dear iwalker,

You made things look easy. Your guidelines worked as a charm. I was completely lost and without your step by step procedure I would not have succeeded. Thank you so much for replying to my question.

Best Regards and stay safe!

rmattos

1 Like

Hi , We are having the same issue post upgrade 14.0 started getting this exception and realized Unicorn is still being used. then changed as per the above. then started getting these certficate exceptions. Any pointers…
There was an error running gitlab-ctl reconfigure:letsencrypt_certificate (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: RuntimeError: ruby_block[create certificate for hub.cimm2.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [hub.cimm2.com] Validation failed, unable to request certificate

Make sure port 80 is unblocked and accessible for your machine, since letsencrypt requires port 80 to be open for it to renew certificates. If it is accessible, check your external_url option in gitlab.rb, and see if set to http or https.

We aren’t using Unicorn any more. We migrated to puma a while ago.
But it still complains during upgrade to GitLab 14

* unicorn['svlogd_prefix'] has been deprecated since 13.10 and was removed in 14.0. Starting with GitLab 14.0, Unicorn is no longer supported and users must switch to Puma, following https://docs.gitlab.com/ee/administration/operations/puma.html.
Deprecations found. Please correct them and try again.

All entried of unicorn has been removed from the /etc/gitlab/gitlab.rb

@sverre are you redirecting gitlab logs anywhere? I saw a post whereby if you were redirecting logs from the gitlab.rb config, that it would then irrespective of unicorn entries or not, automatically run svlogd.

Workaround two worked for us.

1 Like