Upgrade 13.1.11 > 13.10.0 "ERROR: column ci_builds.commands does not exist"

Upgraded using the docker images, upgraded in succession
FROM gitlab/gitlab-ce:12.10.14-ce.0 {being use at the version for a few moths}
Then in succession upgraded in these steps to get the latest.
FROM gitlab/gitlab-ce:13.0.14-ce.0
FROM gitlab/gitlab-ce:13.1.11-ce.0
FROM gitlab/gitlab-ce:13.10.0-ce.0
The logs periodically posted an error."ci_builds.commands do not exist"
when I looked at the logs at step 13.1.11 I don’t recall seeing this error.
but now at 13.10.0, it’s logging every 10 sec or so.

Any ideas, I’ve tried turning it off and on again :wink:
and gitlab-rake db:migrate:status reports all up
Did I miss a step?

Here is a snippet form the logs…

==> /var/log/gitlab/postgresql/current <==
2021-03-23_12:46:09.50862 ERROR: column ci_builds.commands does not exist at character 825
2021-03-23_12:46:09.50868 STATEMENT: SELECT
2021-03-23_12:46:09.50869 subquery.namespace_id,
2021-03-23_12:46:09.50869 subquery.shared_runners_enabled,
2021-03-23_12:46:09.50870 subquery.project_id,
2021-03-23_12:46:09.50871 subquery.status,
2021-03-23_12:46:09.50872 MAX(subquery.count) as count
2021-03-23_12:46:09.50873 FROM (
2021-03-23_12:46:09.50873 SELECT
2021-03-23_12:46:09.50874 projects.namespace_id,
2021-03-23_12:46:09.50874 projects.shared_runners_enabled,
2021-03-23_12:46:09.50875 ci_builds.project_id,
2021-03-23_12:46:09.50875 ci_builds.commit_id,
2021-03-23_12:46:09.50876 ci_builds.status,
2021-03-23_12:46:09.50877 COUNT() AS count*
2021-03-23_12:46:09.50877 FROM ci_builds
2021-03-23_12:46:09.50878 JOIN projects
2021-03-23_12:46:09.50878 ON projects.id = ci_builds.project_id
2021-03-23_12:46:09.50879 JOIN namespaces
2021-03-23_12:46:09.50879 ON namespaces.id = projects.namespace_id
2021-03-23_12:46:09.50880 WHERE ci_builds.type = ‘Ci::Build’
2021-03-23_12:46:09.50880 AND ci_builds.status IN (‘running’, ‘pending’)
2021-03-23_12:46:09.50881 – The created_at filter has been introduced for performance reasons only
2021-03-23_12:46:09.50882 AND ci_builds.created_at > NOW() - INTERVAL ‘7 days’
2021-03-23_12:46:09.50882 GROUP BY
2021-03-23_12:46:09.50883 projects.namespace_id,
2021-03-23_12:46:09.50883 projects.shared_runners_enabled,
2021-03-23_12:46:09.50884 ci_builds.project_id,
2021-03-23_12:46:09.50884 ci_builds.commit_id,
2021-03-23_12:46:09.50885 ci_builds.status,
2021-03-23_12:46:09.50889 ci_builds.commands
2021-03-23_12:46:09.50889 HAVING COUNT() > 2*
2021-03-23_12:46:09.50889 ) AS subquery
2021-03-23_12:46:09.50890 GROUP BY
2021-03-23_12:46:09.50890 subquery.namespace_id,
2021-03-23_12:46:09.50892 subquery.shared_runners_enabled,
2021-03-23_12:46:09.50893 subquery.project_id,
2021-03-23_12:46:09.50893 subquery.commit_id,
2021-03-23_12:46:09.50894 subquery.status
*2021-03-23_12:46:09.50894 *

==> /var/log/gitlab/gitlab-exporter/current <==
2021-03-23_12:46:09.50973 E, [2021-03-23T12:46:09.509635 #5552] ERROR – : Error connecting to the database: ERROR: column ci_builds.commands does not exist
2021-03-23_12:46:09.50976 LINE 30: ci_builds.commands
2021-03-23_12:46:09.50977 ^
*2021-03-23_12:46:09.50977 *
2021-03-23_12:46:09.82227 - - [23/Mar/2021:12:46:09 UTC] “GET /database HTTP/1.1” 200 2655
2021-03-23_12:46:09.82238 - → /database
2021-03-23_12:46:10.21984 - - [23/Mar/2021:12:46:10 UTC] “GET /ruby HTTP/1.1” 200 998
2021-03-23_12:46:10.21994 - → /ruby

Can post more logging if anyone would find it helpful.

Thanks for looking.

1 Like

Hi @PeteSTZzz , welcome to the GitLab Community forum! :tada:

This error indicates your postgresql database lacks a ci_builds.commands column. I suspect that not all background migrations successfully completed on either version 13.0 or 13.1 to cause this.

This shouldn’t affect performance as this column was set to be removed, but because it doesn’t exist we’ll continue seeing errors in the logs.

Can you run the following commands and let us know if the problem persists?

sudo gitlab-ctl reconfigure
sudo gitlab-rake db:migrate


hi @gitlab-greg

Thanks for the welcome and for looking at this with me.

At the end off gitlab-ctl reconfigure
we get
Running handlers complete
Chef Infra Client finished, 1/752 resources updated in 16 seconds
gitlab Reconfigured!
ran gitlab-rake db:migrate but no terminal output.
also did
gitlab-ctl restart
gitlab-rake db:migrate:status reports all up
but still, the error pops up every 10sec or so.

I guess that table has been changed but there is a query in the above post snippet (probably a monitoring query) that’s not been adapted. Looks like the ‘GROUP BY’.

Although I have only recently picked this up, I don’t notice anything exotic about our setup,

I’m working on a VM so have rolled back to square one(12.10.14) and tried the process 3 more times, with the same results.

If step up as far as FROM gitlab/gitlab-ce:13.9.3-ce.0 in the docker file, I don’t see the issues.

Thanks again,

Thanks for the update and path to reproduce this @PeteSTZzz !

I was able to reproduce the error on my end. While it does not appear to cause problems with functionality, it certainly creates noise in the logs.

I’ve created a bug report to bring this to the attention of our developers and product team.

I encourage you to subscribe to notifications on the bug report and contribute any additional details in the issue comments. Thanks again!


Nice, I’ll watch the bug report with interest.


Hi @PeteSTZzz

I was able to track down what introduced this error and found a workaround:

While we work on an official fix, can you try this workaround and let us know if the error persists?


Sweet, for me, the workaround has suppressed the error being raised.
sudo gitlab-rake db:migrate:down VERSION=20210218105431 RAILS_ENV=production

Looks like my developers will be getting the latest and greatest GitLab,
they will be pleased :slight_smile:


I see the same error message and I am locked out of my test instance.

The change you requested was rejected.

FYI, I’m getting this error message also in an new gitlab-ce:13.10.0-container.

I got this error as well, and it prevented all logins with a message from GitLab that it was taking too long. I rolled back a version and will wait for a fix.

Hi everyone! Thanks again for bringing this to our attention.

@bbaassssiiee @ben1 @sr105 I’ve noted what you’re communicating in the issue comments ERROR: column ci_builds.commands does not exist following after upgrade to 13.10.0 (#325872) · Issues · GitLab.org / GitLab · GitLab Please feel free to add any additional details or feedback there!

A fix has been merged and will be available in GitLab version 13.10.2.