Struggle on old v8.11.0 with migration "problems" and non-working 10.2 instance from scratch

I tried to upgrade my gitlab-ce 8.7.0 instance (installed from gitlab/sources) by minor version upgrades, but I got stuck because of strange bundle/gem issues and errors.

My v8.11.0 instance seems to work, but throws 500 entering any project. Other frontend-pages seem to work.

Gitlab check shows this:

System information
System: Debian 8.9
Current User: git
Using RVM: no
Ruby Version: 2.2.8p477
Gem Version: 2.7.2
Bundler Version:1.16.0
Rake Version: 10.5.0
Sidekiq Version:4.1.4
rake aborted!
NoMethodError: undefined method issues_enabled=' for #<Project:0x9ce8a70> /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activemodel-4.2.7.1/lib/active_model/attribute_methods.rb:433:in method_missing’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/default_value_for-3.0.2/lib/default_value_for.rb:178:in block in set_default_values' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/default_value_for-3.0.2/lib/default_value_for.rb:154:in each’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/default_value_for-3.0.2/lib/default_value_for.rb:154:in set_default_values' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:432:in block in make_lambda’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:228:in call' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:228:in block in halting_and_conditional’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in call' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in block in call’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in each' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:506:in call’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:92:in __run_callbacks__' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activesupport-4.2.7.1/lib/active_support/callbacks.rb:778:in _run_initialize_callbacks’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:284:in initialize' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/default_value_for-3.0.2/lib/default_value_for.rb:142:in initialize’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/state_machines-activerecord-0.4.0/lib/state_machines/integrations/active_record.rb:463:in initialize' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in new’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in new' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/reflection.rb:141:in build_association’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:250:in build_record' /home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_association.rb:146:in build’
/home/git/gitlab/vendor/bundle/ruby/2.2.0/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_proxy.rb:259:in build' /home/git/gitlab/lib/tasks/gitlab/info.rake:33:in block (3 levels) in <top (required)>’
Tasks: TOP => gitlab:env:info
(See full trace by running task with --trace)

Because the “migration” is going to make me crazy the 5.th day “trying” now I also tried on another fresh ubuntu system to setup gitlab-ce 10.2 from source (all from scratch including db) to import only my repositories afterwards (loosing rest of my v8 database). The installation looks fine at first glance, but web-frontend does not even show login throwing error 500.

self check for 10.2 “try” looks like this:

Checking GitLab Shell …

GitLab Shell version >= 5.9.3 ? … OK (5.9.3)
Repo base directory exists?
default… yes
Repo storage directories are symlinks?
default… no
Repo paths owned by git:root, or git:git?
default… yes
Repo paths access is drwxrws—?
default… yes
hooks directories in repos are links: … can’t check, you have no projects
Running /home/git/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /home/git/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell … Finished

Checking Sidekiq …

Running? … yes
Number of Sidekiq processes … 1

Checking Sidekiq … Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP …

LDAP is disabled in config/gitlab.yml

Checking LDAP … Finished

Checking GitLab …

Git configured correctly? … yes
Database config exists? … yes
All migrations up? … yes
Database contains orphaned GroupMembers? … no
GitLab config exists? … yes
GitLab config up to date? … yes
Log directory writable? … yes
Tmp directory writable? … yes
Uploads directory exists? … yes
Uploads directory has correct permissions? … yes
Uploads directory tmp has correct permissions? … skipped (no tmp uploads folder yet)
Init script exists? … yes
Init script up-to-date? … yes
Projects have namespace: … can’t check, you have no projects
Redis version >= 2.8.0? … yes
Ruby version >= 2.3.3 ? … yes (2.3.5)
Git version >= 2.7.3 ? … yes (2.7.4)
Git user has default SSH configuration? … yes
Active users: … 0

Checking GitLab … Finished

I just need a working instance. Anyone knows how to fix one of the instances?

@typoworx-de Hi. So are you trying to upgrade your old instance or move to a new one right now? Which would be the best path for you to take?

Well in fact at first I tried to upgrade my old one in smooth steps. But it did not work as well as assumed. I got many problems due to the updates and some versions seemed to behave buggy in upgrade progress (gem errors and exceptions on db migration).

So my second idea was to get rid of these “old” problems and set-up a new instance from scratch on a second VM. But even this does not work well and I run into similar problems. I’m testing and fiddling around now (both path to get gitlab running on one or the other) now for at least 5 days … tested different versions for fresh install and also tested again and again (on my first vm with the old instance) where the upgrade-process may stuck and how to solve it.

The “problems” on both are very different. For me it would be fine to have a fresh install only importing the repositories and don’t import any database stuff.

One to mention… I’m using gitlab-ce source install with mysql so far. I’m not very familiar with postgresql backup/maintaince and already have a working mysql-server on my vm-setup. So if possible anyway I would be glad to keep using mysql (I even thought about getting EE-Licence, but don’t think this will get me rid of the main problems I’m having here…)

@typoworx-de just continuing the discussion from here: https://gitlab.com/gitlab-org/gitlab-ce/issues/38280#note_47523833.

Does the EE-Licenced versions work better than CE? I just need to get up a working instance of gitlab with my old repositories even fine if I have to create users/groups/projects myself without old database.

I know omnibus install - but in fact I don’t feel very well with using that as it looks like omnibus installs more or less a “second linux system” inside my existing one instead of using the distribution packages (like nginx, etc.).

Another thing keeping me away from omnibus is that I’m running on mysql currently (having a VM for mysql) and I’m not very familiar with postgresql either in maintainance nor backup. But I’m familiar with mysql and also having other software using it (so I hope I don’t need a postgres only for gitlab now).

There is one thing that an EE licensed version could get you, and it’s MySQL support.

We bundle all dependencies in the omnibus package, which is not 100% distro friendly but for good reasons, it’s really hard to support so many distros and their specific way of packaging and it’s tight integration with other dependencies.

In short, even if we take the effort, this would mean we would be breaking dependencies from time to time and having to support “why my package Y in distro Z version W broke after I upgraded one of your packages”, or we would be stuck into a minimum common denominator (the oldest version supported by the oldest distro).

While this may sound bad at first, it’s really good and powerful to have everything bundled together. You can always decide not to use / disable one of the dependencies (like the database for instance), or redis, etc and point to your own, all you have to do is change few configuration in /etc/gitlab/gitlab.rb. You can still have control of the parts that matter, without the difficulty of maintaining a source install.

There is one thing that an EE licensed version could get you, and it’s MySQL support.

Can you please tell me until which version of CE mysql is supported? I think I’ve seen that EE costs about 40$ per year… and thats not how much time (or lost working hours) I’ve invested the last days to get up a recent working version of gitlab!

We bundle all dependencies in the omnibus package, which is not 100% distro friendly but for good reason[…]

I really didn’t want to blame the the omnibus package. I’m developer myself and understand the reasons for it. May be it would be a nice idea to make a “Gitlab OS” Distro, too wrapping the whole thing in a ready-to-work distro with kind of setup-guide for initial run and later upgrades? (may be understand as a joke or irony, but it wasn’t).

I think another good start for source-installer like me would some kind of version-matrice being able to see which gitlab-version recommends which components-versions.

Well… what would you advise me to do? Which is the latest version of gitlab-ce supporting mysql? Does EE really solve my problems or I’m still having to install postgres or omnibus? I’m at a point that this get me stuck on my work I’m earning my money with so I even won’t say no to someone make it work for a given amount of time and money to someone saying he/she can make it.

Agree, you can open an issue request for that.

There are efforts from the community itself like (like GitLab debian package), but they follow the same limitations most distro packages have, which is they lag several releases before you can get an update.

My advice is to use the Omnibus version. Go with the community version first to try to import your repositories and if see if it works for you. If you still feel you need MySQL, you can try the EE package (ask for a trial), if it works for you than go ahead.

The main benefit of Omnibus is that we do our best to make the update as painless as possible.

It may be possible to “convert” your original old installation into Omnibus one: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update/README.md#upgrading-from-non-omnibus-postgresql-to-an-omnibus-installation-in-place
(if you want to keep MySQL you will need a license, and you can try to follow the guide and then update the package to latest version).

Thanks for your help @brodock.

I did some “extra work” last night and luckily I was able to recover my old gitlab instance and found out that something in the issues table of db was causing the strange errors on migraton (truncated issues table). Everything works fine now at least with the old version (I was able to upgrade it carefully to v8.14.0 until now).

I will think about upgrading to postgres or using EE/mysql later… just happy it’s working again for the moment to continue usual work now.