Gitlab container keeps restarting after upgrade attempt

How to Use GitLab Upgrade
1

Hello,

Sorry if the question is very basic, but I think I went beyond my skills and now I’m a bit stuck.

We have a Debian 9 server running 3 docker containers via docker-compose:

  • Nginx reverse proxy
  • Gitlab runner
  • Gitlab

That was installed two years ago by our previous IT. Now due to staff shortages I (junior developper, no devops skills) have been asked to upgrade Gitlab as we need the Release feature.

I followed the instructions here by doing

  • docker exec -t gitlab gitlab-backup create
  • docker-compose pull
  • docker-compose up -d

Now when I do docker container ls it looks like the gitlab container just keeps restarting. And when I check the logs with sudo docker logs gitlab, I get this:

Thank you for using GitLab Docker Image!
Current version: gitlab-ce=13.2.4-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab vim /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Cleaning stale PIDs & sockets
It seems you are upgrading from major version 11 to major version 13.
It is required to upgrade to the latest 13.0.x version first before proceeding.
Please follow the upgrade documentation at https://docs.gitlab.com/ee/policy/maintenance.html#upgrading-major-versions

But I cannot edit gitlab.rb as it keeps restarting. And anyway, I’m not sure to understand what I would have to do.

And cherry on the cake, I just realized that the backup command didn’t work.(I’m very stupid, I was doing something else at the same time):

gitlab docker OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"gitlab-backup\": executable file not found in $PATH": unknown

Does anyone have a clue on what I could do ?

Edit: Or if there is a way to rollback to the Gitlab version from before the upgrade attempt ?

I know I should have been a lot more careful, lesson learnt.

2

Update: I managed to rollback to 11.11.8 and it’s back online.

I’m going to figure out how to update first but then, for the upgrade, should I do it like said here: 12.0.12 -> 12.10.14 -> 13.0.12 -> 13.2.3 ? Is it definitely the recommended way ?

Edit: so I’m still getting
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"gitlab-backup\": executable file not found in $PATH": unknown
when I try to update using docker exec -t gitlab gitlab-backup create

3

Hi @AnotherMunshine!

GitLab didn’t release the gitlab-backup wrapper script until GitLab 12.1 - since you are on 11.11.8 - you’ll just want to use the rake task directly:

docker exec -t <container name> gitlab-rake gitlab:backup:create

It’s definitely a little involved, but you’ll want to go through the recommended Major version upgrade progression for sure. There are migrations and component updates that only properly occur in the “dot-0” release upgrades.

Also make sure to see the section on background migrations in the Upgrade docs.

1 Like
4

Ho I see ! Thanks a lot for the clear reply :slight_smile:

1 Like