Update gitlab 17 failed: GitLab KAS: RuntimeError: Execution of the command /opt/gitlab/embedded/bin/gitlab-kas --version failed

There was an error running gitlab-ctl reconfigure:

version_file[Create version file for Gitlab KAS] (gitlab-kas::enable line 67) had an error: RuntimeError: Execution of the command /opt/gitlab/embedded/bin/gitlab-kas --version failed with a non-zero exit code (2)
stdout:
stderr: panic: failed to parse “2024-05-15T09:09:49+0000” into RFC3339 compliant time object, because: parsing time “2024-05-15T09:09:49+0000” as “2006-01-02T15:04:05Z07:00”: cannot parse “+0000” as “Z07:00”. Fix the build process.

goroutine 1 [running]:
gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
/var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3

same issue here.
Upgrade from 16.11.2 → 17.0.0

I have the same issue too.

Can I downgrade back to 16.11.2, to fix this?

I disabled the gitlab-kas, and then upgrade success

1 Like

It seems to happen only on CentOS7, I also have RedHat9 with gitlab-kas enabled and it doesn’t broke.

Thanks this seems to have worked for me as well.

edit /etc/gitlab/gitlab.rb and add: gitlab_kas[‘enable’] = false
Then reconfigure: gitlab-ctl reconfigure

Also running Gitlab on CentOS7

1 Like

I can confirm that it affected my upgrade to version 17.0.0 on CentOS 7.

repo    [2024-05-16T16:24:52+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
repo    [2024-05-16T16:24:52+02:00] FATAL: ---------------------------------------------------------------------------------------
repo    [2024-05-16T16:24:52+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
repo    [2024-05-16T16:24:52+02:00] FATAL: ---------------------------------------------------------------------------------------
repo    [2024-05-16T16:24:52+02:00] FATAL: RuntimeError: version_file[Create version file for Gitlab KAS] (gitlab-kas::enable line 67) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/gitlab-kas --version` failed with a non-zero exit code (2)
repo    stdout: 
repo    stderr: panic: failed to parse "2024-05-15T09:24:36+0000" into RFC3339 compliant time object, because: parsing time "2024-05-15T09:24:36+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00". Fix the build process.
repo
repo    goroutine 1 [running]:
repo    gitlab.com/gitlab-org/cluster-integration/gitlab-agent/v17/cmd.init.0()
repo            /var/cache/omnibus/src/gitlab-kas/cmd/build_info.go:18 +0xf3
repo
repo
repo    ===
repo    There was an error running gitlab-ctl reconfigure. Please check the output above for more
repo    details.
repo    ===
repo    warning: %posttrans(gitlab-ce-17.0.0-ce.0.el7.x86_64) scriptlet failed, exit status 1
repo      Verifying  : gitlab-ce-17.0.0-ce.0.el7.x86_64                             1/2 
repo      Verifying  : gitlab-ce-16.11.2-ce.0.el7.x86_64                            2/2 
repo
repo    Updated:
repo      gitlab-ce.x86_64 0:17.0.0-ce.0.el7                                            
repo
repo    Complete!
repo    Non-fatal POSTTRANS scriptlet failure in rpm package gitlab-ce-17.0.0-ce.0.el7.x86_64
Exit 6

Editing and disabling gitlab-kas in /etc/gitlab/gitlab.rb and re-running gitlab-ctl reconfigure and restarting it all solved the issue. Should this not be filed as a bug report?

It seems only Centos7 is affected - no problems on Ubuntu 22.

Confirmed solution for now (centos7):
gitlab_kas[‘enable’] = false in /etc/gitlab/gitlab.rb + gitlab-ctl reconfigure

I don’t think they consider this as a bug because Centos7 is EOL… so it should be expected - sooner or later we must migrate to the newer OS.

It is nearly EOL (end of june 2024) but as its alter ego Oracle Linux is supported until the end of this year. They are identical. And Oracle is in the list of supported operating systems for GitLab.

Suggest raising an issue here for the Gitlab devs to look at it: Issues · GitLab.org / GitLab · GitLab

Probably best to search first, just in case an issue has already be opened. But yes, in one month EL7 will effectively be EOL, so I expect Gitlab will drop it as this is what they normally do when a version becomes EOL. It would be prudent to start thinking about migrating by using backup/restore procedures to go from EL7 to EL8 or EL9.

Assuming Gitlab do support Oracle until Dec 2024, then it would be worth opening the issue to resolve the problem.

Thanks for reporting, and sorry for the troubles. The problem is tracked in

Suggest adding your environment details, findings, workarounds. Thanks!

I will add a comment linking to this forum topic, too.

2 Likes

Me too. 16.11.2->17.0.0. Similarly centos 7. Yes centos 7 is near EOL but not yet beyond. I was updating in anticipation of it probably being easier to switch to a new Amazon Linux host later this week. I guess I will have to downgrade (ages since I’ve had a failed gitlab upgrade and can’t remember how, but there you are)

BTW I can confirm that the workaround of Install the GitLab agent server for Kubernetes (KAS) | GitLab seems to work - not so important for us as we don’t use it. Even then had to explicitly restart gitlab and still not 100% sure things are OK.

hi,

I have the same problem:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"

I had issues with Gitlab KAS. I disabled it in the config file, and it worked smoothly.

I use Oracle enterprise linux 7. This also has same problem.

This is fixed in 17.0.1 GitLab Patch Release: 17.0.1, 16.11.3, 16.10.6 | GitLab

1 Like

But for those who have already upgraded to 17.0.0, this still shows up error.

I tried to disable gitlab-kas. But ended up with the below error when it is disabled.

Running handlers:
[2024-05-24T18:37:35+05:30] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  ----
STDOUT: rake aborted!
ActiveRecord::DatabaseConnectionError: There is an issue connecting to your database with your username/password, username: gitlab.

Please check your database configuration to ensure the username/password are valid.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:124:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:128:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'

Caused by:
PG::ConnectionBad: connection to server on socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:124:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:128:in `configure_database'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:107:in `configure_pg_databases'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:94:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash"  ----
Ran "bash"  returned 1

Maybe your 17.0.0 update is missing some migrations about db? 17.0.0 was the first version with no support for PostgreSQL 13. What about disabling gitlab-kas (force it to false in gitlab.rb) and launch a reconfigure?

1 Like

just go to /etc/gitlab/gitlab.rb
set the gitlab_kas[‘enable’] = false
and run again
sudo gitlab-ctl reconfigure.
if you use nfs to save the gitlab-data,you can
su git && chmod 777 $your-nfs-path-of-gitlab
then run again sudo gitlab-ctl reconfigure.
it will be OK. and you can start your gitlab or upgrade gitlab


image