GitLab won't start after upgrade

New user of GL. Ubuntu did an software upgrade and GitLab was one of them.
Now GitLab won’t start!

Error text:

There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20181003-6008-1jwgrlb” ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load’
/opt/gitlab/embedded/bin/bundle:23:in `’
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of “bash” “/tmp/chef-script20181003-6008-1jwgrlb” ----
Ran “bash” “/tmp/chef-script20181003-6008-1jwgrlb” returned 1

Now I really need yor help. I’m hosting 10+ projects and I’ll hate if they are lost. I do have backups thou :slight_smile:

Edit: updated error printout

Be gentle now, I’m really new on this

Hi and welcome to the GitLab forum! :slight_smile:

If you give us some more info about your installation, that will make it easier for someone to be able to help you. From what you are saying, it seems you are using either the packaged version of GitLab in Ubuntu, or a more recent package from https://packages.gitlab.com.

Could you tell us:

  • Which version of Ubuntu you are running? You can find out by running the lsb_release -a command.
  • Which package version of GitLab do you have installed? You can find out by running apt-cache policy gitlab; or alternatively apt-cache policy gitlab-ce if that does not work.

I would also suggest to use the code formatting function of the editor, as it makes it easier to read the output of commands, particularly log output. It’s the </> icon at the top of the editor window when you’re writing a post, or you can alternatively use markdown directly.

Thank you for your patience. I’ll try to better myself as I go.

Ubuntu:
No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic

GitLab:
gitlab-ee: Installed: 11.3.1-ee.0 Candidate: 11.3.1-ee.0 Version table: *** 11.3.1-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status 11.3.0-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.2.4-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.2.3-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.2.2-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.2.1-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.2.0-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.1.7-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.1.6-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.1.4-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.1.2-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.1.1-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.1.0-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.6-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.5-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.4-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.3-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.2-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.1-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 11.0.0-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.7-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.6-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.5-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.4-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.3-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.2-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.1-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.8.0-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.7-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.6-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.5-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.4-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.3-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.2-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.1-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages 10.7.0-ee.0 500 500 https://packages.gitlab.com/gitlab/gitlab-ee/ubuntu bionic/main amd64 Packages

Thank you for helping me!

Ok, thanks for the follow up, it seems you’re now on:

My first thought would be to address that message in your log:

PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket “/var/opt/gitlab/postgresql/.s.PGSQL.5432”?
  • Is PostgreSQL still running? It could be that the service was stopped after the package upgrades on your system. What’s the output of sudo service postgresql status?

Also, it seems you installed the gitlab-ee (Enterprise Edition) vs. the gitlab-ce (Community Edition) package. I’m not sure it makes much of a difference, as I’m not familiar with the GitLab .deb packages. But it might be that you installed gitlab-ee because you are a paying customer, in which case you might want to contact GitLab support directly.

Hi. I’m not a paying customer. As I read somwhere the ee and ce is the same if you don’t pay.

Anyway. Here’s the output:
Unit postgresql.service could not be found.

Did this: sudo gitlab-psql --version
and got: psql (PostgreSQL) 9.6.8

Anyway I don’t really know how to proceed.

Edited out alot of useless “me being noobish” on gitlab

Thanx

My mistake, please scratch the question about the output of sudo service postgresql status. PostgreSQL in the gitlab-ee (or -ce) packages is not installed that way in the system. As I said, I don’t have much experience with the .deb packages, but I’ve just done an install to see how they work.

To see if the database is running, you can run sudo gitlab-ctl postgresql status. In fact, you can just run sudo gitlab-ctl status to output a list with the status of all services on which GitLab depends. That at least should give you some idea if the database server is running at all.

Never fear asking questions. People here are welcoming and willing to help. Even experienced users have got a lot to learn from beginners.

sudo gitlab-ctl status
produced:

fail: alertmanager: runsv not running fail: gitaly: runsv not running fail: gitlab-monitor: runsv not running fail: gitlab-workhorse: runsv not running fail: logrotate: runsv not running fail: nginx: runsv not running fail: node-exporter: runsv not running fail: postgres-exporter: runsv not running fail: postgresql: runsv not running fail: prometheus: runsv not running fail: redis: runsv not running fail: redis-exporter: runsv not running fail: sidekiq: runsv not running fail: unicorn: runsv not running

My google skills got me this:
gitlab-runsvdir not starting after upgrading to 11.2

I’m looking into it right now. It looks like it may be the problem.

Okay… GitLab IS NOW RUNNING!!! Wohoo!

I had to disable Plymouth. It was blocking multi-user.target and gitlab-runsvdir.service among other things.

After I disabled Plymouth I did a reboot and an reconfigure. At that point my db was migrated.

My deepest thanks to you, @dplanella for listening to me and trying to help me!!

1 Like

Excellent, glad you got back up and running! :slight_smile:

I also learnt something new. For others facing this issue, this is worth a read too: https://docs.gitlab.com/omnibus/common_installation_problems/#gitlab-runsvdir-not-starting

1 Like

Heyy… I need help with the upgrade from 10.7 to 10.8. I read it from here: https://docs.gitlab.com/ee/update/10.7-to-10.8.html

In step 6 it just says i should checkout and fetcch new version of code. But i am not connected to remote repository not there is any way i can get the code. I am a begginner with this and it would be great if you could just help me. I know im making a mistake somewhere but cant figure it out.

Thanks in advance

I’m pretty fresh myself but I can try to help you anyway.
Which installation do you have? If it is an omnibus installation and you just want the latest version you can do this pretty easy: Update GitLab

And here is the source: Source code

Hope this helps

Thank you for your fast reply.

Anyway, this does not work since im using 10.7 version. So, first I need to upgrade to 10.8, and then to 11.0 and higher. I tried this before and it did not work.

The most confusing thing for me is the step 6 which says just to checkout the gitlab project branch and fetch the code. That seems weird and of course it does not work.

Thank you once again.

I’m sorry but I don’t understand.
No disrespect but do you know how to use Git? In my previous post there was a link to GitLab source. You can fetch it from there.

:)) of course I know, no problem. I guess i was expecting them to write that i need to create a repository, clone remote repository there, checkout to a specific version and then do that for all those git repositories (there are few of them). I never done it that way. Previously i just did it the way you already wrote. I just never did it that way.

You know, when i go to /home/git/gitlab and run fetch, nothing happens. I need to first init the repository and so on… I guess i expected them to write that as well

Thank you very much for your response. Its gitlab instalation that we use while developing, so i was not very confident about doing all those things manually for the first time. Ill just do it on some test environment first.

Thanks again.

1 Like