Canceling pipeline results in an error

A couple weeks ago I migrated my GitLab-EE (EDU license, so no support) instance from an Ubuntu 14.04 vm to a CentOS 7 vm.

Shortly after, one of my projects had a pipeline get stuck. Whenever I try to cancel the pipeline, I get an error message.

This pipeline has stayed stuck, and throwing an error when I try to cancel, through an upgrade, and multiple vm reboots.

To further complicate things, this is specific to this single project and this single pipeline. I was able to cancel newer pipelines in the project just fine. :\

I think these lines from /var/log/gitlab/gitlab-rails/production.log are the most relavent. If you want me to watch for logs in another file, let me know which one.

Started GET "/groupname/projectname/pipelines.json?scope=all&page=1" for laptopipaddr at 2019-05-01 16:15:02 -0700
Started POST "/groupname/projectname/pipelines/14947/cancel.json" for laptopipaddr at 2019-05-01 16:15:04 -0700
Processing by Projects::PipelinesController#cancel as JSON
  Parameters: {"namespace_id"=>"ansible", "project_id"=>"utilstack", "id"=>"14947"}
Completed 500 Internal Server Error in 43ms (ActiveRecord: 7.0ms | Elasticsearch: 0.0ms)
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:32:in `ensure_token'
app/models/concerns/token_authenticatable.rb:40:in `block in add_authentication_token_field'
app/models/ci/pipeline.rb:443:in `block (2 levels) in cancel_running'
app/models/ci/pipeline.rb:441:in `block in cancel_running'
lib/gitlab/optimistic_locking.rb:9:in `block in retry_lock'
lib/gitlab/optimistic_locking.rb:8:in `retry_lock'
app/models/ci/pipeline.rb:440:in `cancel_running'
app/controllers/projects/pipelines_controller.rb:126:in `cancel'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Marked stuck import jobs as failed. JIDs: eeb3090228ddbfa418a595af
Started GET "/groupname/projectname/pipelines.json?scope=all&page=1" for laptopipaddr at 2019-05-01 16:15:12 -0700
Started GET "/-/metrics" for 127.0.0.1 at 2019-05-01 16:15:14 -0700
Processing by MetricsController#index as HTML
Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)

Iā€™m using the Omnibus installation from GitLabs yum repo.

OS: CentOS Linux release 7.6.1810 (Core).

environment info:

[root@internalhost system]# gitlab-rake gitlab:env:info

System information
System:     
Proxy:      no
Current User:   git
Using RVM:  no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.5
Go Version: unknown

GitLab information
Version:    11.10.4-ee
Revision:   88a3c791734
Directory:  /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.6.11
URL:        https://internalhost
HTTP Clone URL: https://internalhost/some-group/some-project.git
SSH Clone URL:  git@internalhost:some-group/some-project.git
Elasticsearch:  no
Geo:        no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: github

GitLab Shell
Version:    9.0.0
Repository storage paths:
- default:  /var/opt/gitlab/git-data/repositories
GitLab Shell path:      /opt/gitlab/embedded/service/gitlab-shell
Git:        /opt/gitlab/embedded/bin/git

It sounds like https://gitlab.com/gitlab-org/gitlab-ce/issues/59413 case. Please try to remove the problematic pipeline Ci::Pipeline.find(:pipeline_id).destroy!

Um, how do I run that?

# gitlab-rake Ci::Pipeline.find(14947).destroy!
-bash: syntax error near unexpected token `('
> sudo gitlab-rails console # Login to the rails console
$ Ci::Pipeline.find(14947).destroy!
1 Like

Thanks! That fixed it.