Unable to upgrade from 13.8.7 to 13.9.x or 13.10.x

I’m using the Docker Omnibus for a couple of years, and for the first time, I can’t upgrade my instance and I don’t understand why :tired_face:
If someone can help me, that will be so appreciated.

You can find all details from my original issue: Unable to upgrade from 13.8.7 to 13.9.x or 13.10.x (#327063) · Issues · GitLab.org / GitLab · GitLab

Thank you in advance for your help :pray:t2:

Hi @Laul0
please see this issue which is the same error.
Also make sure you are using supported PostgreSQL database.

Hi @balonik
Thank you for your answer and this information :pray:t2:

With Omnibus Docker, I had to upgrade the PostgreSQL database to 12.6, in accordance with the GitLab releases information, it seems this is the latest version embed with the Docker image.
I will try to alter the database manually and I will let you know :crossed_fingers:t2:

Hello !!!

I finally found a solution :partying_face:
Before sharing the solution, I have to thanks the little group of people who share their cases and who try to explain the expected behavior and the database configuration, really helpful! :pray:t2:
Thank you again @balonik to shared this issue that allowed me to deepen my investigation.

And finally, NOT thank you to the GitLab team for their no help :unamused: (I know there are a lot of open issues and a lot of people are requested some help… but, this kind of issue can touch everyone who migrated their MySQL database to PostgreSQL and the GitLab migration scripts do not handle this kind of use cases. Some issues were opened about that…)

Well, what I did fix the issue? Effectively, the main issue was related to:

  • An very old migration from MySQL to PostgreSQL
  • The timestamps without time zone

The steps:

  1. The base issue was not totally the same. It was not linked to the ci_job_artifacts but with the onboarding_progresses

  2. I altered the database manually to change the timestamp with time zone for this table:

    gitlab-rails dbconsole
    
    ALTER TABLE onboarding_progresses ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE onboarding_progresses ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE onboarding_progresses ALTER COLUMN git_pull_at type timestamp with time zone;
    ALTER TABLE onboarding_progresses ALTER COLUMN git_write_at type timestamp with time zone;
    
  3. After that, I had the same error as the open issue ci_group_variables

  4. To try to win time, I checked manually the 460+ tables and identify all columns with timestamp without time zone

  5. Compare the found tables with the original database structure to change only the good ones (db/structure.sql · master · GitLab.org / GitLab FOSS · GitLab)

  6. For each column, I prepared an ALTER TABLE SQL command:

    ALTER TABLE onboarding_progresses ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE onboarding_progresses ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE onboarding_progresses ALTER COLUMN git_pull_at type timestamp with time zone;
    ALTER TABLE onboarding_progresses ALTER COLUMN git_write_at type timestamp with time zone;
    ALTER TABLE ci_group_variables ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE ci_group_variables ALTER COLUMN updated_at type timestamp with time zone;
    DROP INDEX "expired_artifacts_temp_index";
    ALTER TABLE ci_job_artifacts ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE ci_job_artifacts ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE ci_job_artifacts ALTER COLUMN expire_at type timestamp with time zone;
    ALTER TABLE events ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE events ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE gpg_keys ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE gpg_keys ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE gpg_signatures ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE gpg_signatures ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE group_custom_attributes ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE group_custom_attributes ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE project_auto_devops ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE project_auto_devops ALTER COLUMN updated_at type timestamp with time zone;
    ALTER TABLE project_custom_attributes ALTER COLUMN created_at type timestamp with time zone;
    ALTER TABLE project_custom_attributes ALTER COLUMN updated_at type timestamp with time zone;
    
  7. As you can notice, I have a DROP INDEX. This index used the created_at columns and generate the IMMUTABLE error with the ci_job_artifacts. Then, I dropped it :joy:

  8. Execute these SQL commands and here we go! The upgrading passed successfully

I hope these information will help someone else.

Cheers :clinking_glasses:

1 Like

If you want help from the Gitlab Team, then purchase a subscription - there was no need for such a comment @Laul0. If you do pay, then you could have opened a support ticket, but the fact that you posted here suggests you are a free user, so you cannot expect for help from Gitlab when you don’t pay. The help comes via the forum, and the Gitlab team do come by, so they do help.

This is the Gitlab forum for the community, or if you like free users who don’t pay for support and the community users are the ones that help out with your problems like @balonik so kindly did. You should be glad that there are community members actually willing to help out and resolve your problem or point you in the right direction.

That aside, thank you for sharing the steps you provided to help give something back to the community, as together we are stronger when everyone helps out. Gitlab employees do visit the forum and do help out when they can, but their priorities are their paying customers, which is completely understandable.

As far as I know, the GitLab team won’t do any support ticket for the CE version. I also opened one before, never get any response, but as long as you can put your ticket as bugs or something, they will assign someone and have a look at it.

1 Like

Yes, since CE is community edition. EE being Enterprise means you can purchase a subscription and with that subscription get support. EE can also be utilised just like CE, with the same level of functionality for free, but also wouldn’t be supported by Gitlab Team if no subscription has been purchased - since help is via the forums and the Gitlab team do visit. Obviously this is when the community would help out.

I wouldn’t start opening issues though as bugs just for support queries, first because it’s wasting time for the Gitlab Team then having to close tickets that have nothing to do with bugs. I would only suggest doing this if it actually is a bug, so that the Gitlab Team can concentrate on what is important.

A lot of upgrade problems can be mitigated by upgrading regularly - especially when postgresql versions change between releases - and also to ensure that the upgrade path is followed so that certain DB upgrades or even migrations aren’t missed. I feel far less problems are encountered actually using the packaged omnibus versions than docker, but that’s just my personal view point.