Upgrade 13.1.11 > 13.10.0 "ERROR: column ci_builds.commands does not exist"

PeteSTZzz · March 23, 2021, 1:00pm

Upgraded using the docker images, upgraded in succession
FROM gitlab/gitlab-ce:12.10.14-ce.0 {being use at the version for a few moths}
Then in succession upgraded in these steps to get the latest.
FROM gitlab/gitlab-ce:13.0.14-ce.0
FROM gitlab/gitlab-ce:13.1.11-ce.0
FROM gitlab/gitlab-ce:13.10.0-ce.0
The logs periodically posted an error."ci_builds.commands do not exist"
when I looked at the logs at step 13.1.11 I don’t recall seeing this error.
but now at 13.10.0, it’s logging every 10 sec or so.

Any ideas, I’ve tried turning it off and on again
and gitlab-rake db:migrate:status reports all up
Did I miss a step?

Here is a snippet form the logs…

==> /var/log/gitlab/postgresql/current <==
2021-03-23_12:46:09.50862 ERROR: column ci_builds.commands does not exist at character 825
2021-03-23_12:46:09.50868 STATEMENT: SELECT
2021-03-23_12:46:09.50869 subquery.namespace_id,
2021-03-23_12:46:09.50869 subquery.shared_runners_enabled,
2021-03-23_12:46:09.50870 subquery.project_id,
2021-03-23_12:46:09.50871 subquery.status,
2021-03-23_12:46:09.50872 MAX(subquery.count) as count
2021-03-23_12:46:09.50873 FROM (
2021-03-23_12:46:09.50873 SELECT
2021-03-23_12:46:09.50874 projects.namespace_id,
2021-03-23_12:46:09.50874 projects.shared_runners_enabled,
2021-03-23_12:46:09.50875 ci_builds.project_id,
2021-03-23_12:46:09.50875 ci_builds.commit_id,
2021-03-23_12:46:09.50876 ci_builds.status,
2021-03-23_12:46:09.50877 COUNT() AS count*
2021-03-23_12:46:09.50877 FROM ci_builds
2021-03-23_12:46:09.50878 JOIN projects
2021-03-23_12:46:09.50878 ON projects.id = ci_builds.project_id
2021-03-23_12:46:09.50879 JOIN namespaces
2021-03-23_12:46:09.50879 ON namespaces.id = projects.namespace_id
2021-03-23_12:46:09.50880 WHERE ci_builds.type = ‘Ci::Build’
2021-03-23_12:46:09.50880 AND ci_builds.status IN (‘running’, ‘pending’)
2021-03-23_12:46:09.50881 – The created_at filter has been introduced for performance reasons only
2021-03-23_12:46:09.50882 AND ci_builds.created_at > NOW() - INTERVAL ‘7 days’
2021-03-23_12:46:09.50882 GROUP BY
2021-03-23_12:46:09.50883 projects.namespace_id,
2021-03-23_12:46:09.50883 projects.shared_runners_enabled,
2021-03-23_12:46:09.50884 ci_builds.project_id,
2021-03-23_12:46:09.50884 ci_builds.commit_id,
2021-03-23_12:46:09.50885 ci_builds.status,
2021-03-23_12:46:09.50889 ci_builds.commands
2021-03-23_12:46:09.50889 HAVING COUNT() > 2*
2021-03-23_12:46:09.50889 ) AS subquery
2021-03-23_12:46:09.50890 GROUP BY
2021-03-23_12:46:09.50890 subquery.namespace_id,
2021-03-23_12:46:09.50892 subquery.shared_runners_enabled,
2021-03-23_12:46:09.50893 subquery.project_id,
2021-03-23_12:46:09.50893 subquery.commit_id,
2021-03-23_12:46:09.50894 subquery.status
*2021-03-23_12:46:09.50894 *

==> /var/log/gitlab/gitlab-exporter/current <==
2021-03-23_12:46:09.50973 E, [2021-03-23T12:46:09.509635 #5552] ERROR – : Error connecting to the database: ERROR: column ci_builds.commands does not exist
2021-03-23_12:46:09.50976 LINE 30: ci_builds.commands
2021-03-23_12:46:09.50977 ^
*2021-03-23_12:46:09.50977 *
2021-03-23_12:46:09.82227 127.0.0.1 - - [23/Mar/2021:12:46:09 UTC] “GET /database HTTP/1.1” 200 2655
2021-03-23_12:46:09.82238 - → /database
2021-03-23_12:46:10.21984 127.0.0.1 - - [23/Mar/2021:12:46:10 UTC] “GET /ruby HTTP/1.1” 200 998
2021-03-23_12:46:10.21994 - → /ruby

Can post more logging if anyone would find it helpful.

Thanks for looking.

gitlab-greg · March 23, 2021, 7:12pm

Hi @PeteSTZzz , welcome to the GitLab Community forum!

This error indicates your postgresql database lacks a ci_builds.commands column. I suspect that not all background migrations successfully completed on either version 13.0 or 13.1 to cause this.

This shouldn’t affect performance as this column was set to be removed, but because it doesn’t exist we’ll continue seeing errors in the logs.

gitlab.com

gitlab-org/gitlab/blob/master/db/post_migrate/20210218105431_remove_deprecated_ci_builds_columns.rb#L18



disable_ddl_transaction!

def up
  with_lock_retries do
    remove_column :ci_builds, :artifacts_file
    remove_column :ci_builds, :artifacts_file_store
    remove_column :ci_builds, :artifacts_metadata
    remove_column :ci_builds, :artifacts_metadata_store
    remove_column :ci_builds, :artifacts_size
    remove_column :ci_builds, :commands
  end
end

def down
  # rubocop:disable Migration/AddColumnsToWideTables
  with_lock_retries do
    add_column :ci_builds, :artifacts_file, :text
    add_column :ci_builds, :artifacts_file_store, :integer
    add_column :ci_builds, :artifacts_metadata, :text
    add_column :ci_builds, :artifacts_metadata_store, :integer

Can you run the following commands and let us know if the problem persists?

sudo gitlab-ctl reconfigure
sudo gitlab-rake db:migrate

Thanks!

PeteSTZzz · March 24, 2021, 9:42am

hi @gitlab-greg

Thanks for the welcome and for looking at this with me.

At the end off gitlab-ctl reconfigure
we get
Running handlers complete
Chef Infra Client finished, 1/752 resources updated in 16 seconds
gitlab Reconfigured!
ran gitlab-rake db:migrate but no terminal output.
also did
gitlab-ctl restart
gitlab-rake db:migrate:status reports all up
but still, the error pops up every 10sec or so.

I guess that table has been changed but there is a query in the above post snippet (probably a monitoring query) that’s not been adapted. Looks like the ‘GROUP BY’.

Although I have only recently picked this up, I don’t notice anything exotic about our setup,

I’m working on a VM so have rolled back to square one(12.10.14) and tried the process 3 more times, with the same results.

If step up as far as FROM gitlab/gitlab-ce:13.9.3-ce.0 in the docker file, I don’t see the issues.

Thanks again,

gitlab-greg · March 24, 2021, 5:22pm

Thanks for the update and path to reproduce this @PeteSTZzz !

I was able to reproduce the error on my end. While it does not appear to cause problems with functionality, it certainly creates noise in the logs.

I’ve created a bug report to bring this to the attention of our developers and product team.

I encourage you to subscribe to notifications on the bug report and contribute any additional details in the issue comments. Thanks again!

PeteSTZzz · March 24, 2021, 5:42pm

Nice, I’ll watch the bug report with interest.

gitlab-greg · March 24, 2021, 8:58pm

Hi @PeteSTZzz

I was able to track down what introduced this error and found a workaround:

While we work on an official fix, can you try this workaround and let us know if the error persists?

PeteSTZzz · March 25, 2021, 8:24am

Sweet, for me, the workaround has suppressed the error being raised.
sudo gitlab-rake db:migrate:down VERSION=20210218105431 RAILS_ENV=production

Looks like my developers will be getting the latest and greatest GitLab,
they will be pleased

bbaassssiiee · March 29, 2021, 4:04pm

I see the same error message and I am locked out of my test instance.

The change you requested was rejected.

ben1 · March 30, 2021, 8:59am

FYI, I’m getting this error message also in an new gitlab-ce:13.10.0-container.

sr105 · April 1, 2021, 6:17pm

I got this error as well, and it prevented all logins with a message from GitLab that it was taking too long. I rolled back a version and will wait for a fix.

gitlab-greg · April 1, 2021, 7:18pm

Hi everyone! Thanks again for bringing this to our attention.

@bbaassssiiee @ben1 @sr105 I’ve noted what you’re communicating in the issue comments ERROR: column ci_builds.commands does not exist following after upgrade to 13.10.0 (#325872) · Issues · GitLab.org / GitLab · GitLab Please feel free to add any additional details or feedback there!

A fix has been merged and will be available in GitLab version 13.10.2.