500 Error on Saving Settings

Hi community,

I’ve run into a snag with my locally hosted version of GitLab 10.0.0-rc2. In one of my projects, I am unable to change the CI/CD timeout. In fact, I can’t save any of the CI/CD settings. Other settings save fine, and other projects save this one setting fine as well.

Steps to reproduce

  1. Go to project
  2. Go to Settings -> CI/CD -> General pipelines settings
  3. Hit Save

Output

I see traces of problem in the log, but am not familiar enough with GitLab nor Ruby to pin it down.

...
2017-11-21T11:35:42.366732304Z 172.17.0.1 - - [21/Nov/2017:11:35:42 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.5.0 (9-5-stable; go1.8.3; linux/amd64)"
2017-11-21T11:35:42.921771137Z
2017-11-21T11:35:42.921791729Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-11-21T11:35:42.921796112Z Started PATCH "/mycompany/myproject/pipelines/settings" for 172.17.0.1 at 2017-11-21 11:35:42 +0000
2017-11-21T11:35:42.929118463Z Processing by Projects::PipelinesSettingsController#update as HTML
2017-11-21T11:35:42.929138642Z   Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "project"=>{"auto_devops_attributes"=>{"enabled"=>"true", "domain"=>""}, "runners_token"=>"[FILTERED]", "build_allow_git_fetch"=>"true", "build_timeout_in_minutes"=>"120", "ci_config_path"=>"", "public_builds"=>"1", "auto_cancel_pending_pipelines"=>"enabled", "build_coverage_regex"=>""}, "namespace_id"=>"mycompany", "project_id"=>"myproject"}
2017-11-21T11:35:42.939530274Z
2017-11-21T11:35:42.939544551Z ==> /var/log/gitlab/postgresql/current <==
2017-11-21T11:35:42.939548283Z 2017-11-21_11:35:42.93938 ERROR:  null value in column "project_id" violates not-null constraint
2017-11-21T11:35:42.939551048Z 2017-11-21_11:35:42.93940 DETAIL:  Failing row contains (1, null, 2017-11-20 10:46:22.253987+00, 2017-11-21 11:35:42.938635+00, t, ).
2017-11-21T11:35:42.939553633Z 2017-11-21_11:35:42.93940 STATEMENT:  UPDATE "project_auto_devops" SET "project_id" = $1, "updated_at" = $2 WHERE "project_auto_devops"."id" = $3
2017-11-21T11:35:42.940531477Z
2017-11-21T11:35:42.940544757Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-11-21T11:35:42.940548750Z Completed 500 Internal Server Error in 11ms (ActiveRecord: 1.4ms)
2017-11-21T11:35:42.941252591Z
2017-11-21T11:35:42.941264272Z ==> /var/log/gitlab/gitlab-rails/production_json.log <==
2017-11-21T11:35:42.941268816Z {"method":"PATCH","path":"/mycompany/myproject/pipelines/settings","format":"html","controller":"Projects::PipelinesSettingsController","action":"update","status":500,"error":"ActiveRecord::StatementInvalid: PG::NotNullViolation: ERROR:  null value in column \"project_id\" violates not-null constraint\nDETAIL:  Failing row contains (1, null, 2017-11-20 10:46:22.253987+00, 2017-11-21 11:35:42.938635+00, t, ).\n: UPDATE \"project_auto_devops\" SET \"project_id\" = $1, \"updated_at\" = $2 WHERE \"project_auto_devops\".\"id\" = $3","duration":11.87,"view":0.0,"db":1.42,"time":"2017-11-21T11:35:42.929Z","params":{"utf8":"✓","_method":"patch","authenticity_token":"[FILTERED]","project":{"auto_devops_attributes":{"enabled":"true","domain":""},"runners_token":"[FILTERED]","build_allow_git_fetch":"true","build_timeout_in_minutes":"120","ci_config_path":"","public_builds":"1","auto_cancel_pending_pipelines":"enabled","build_coverage_regex":""},"namespace_id":"mycompany","project_id":"myproject"},"remote_ip":"172.17.0.1","user_id":1,"username":"root"}
2017-11-21T11:35:42.942672085Z
2017-11-21T11:35:42.942686548Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-11-21T11:35:42.942691209Z
2017-11-21T11:35:42.942693987Z ActiveRecord::StatementInvalid (PG::NotNullViolation: ERROR:  null value in column "project_id" violates not-null constraint
2017-11-21T11:35:42.942696855Z DETAIL:  Failing row contains (1, null, 2017-11-20 10:46:22.253987+00, 2017-11-21 11:35:42.938635+00, t, ).
2017-11-21T11:35:42.942699417Z : UPDATE "project_auto_devops" SET "project_id" = $1, "updated_at" = $2 WHERE "project_auto_devops"."id" = $3):
2017-11-21T11:35:42.942702440Z   config/initializers/active_record_locking.rb:16:in `_update_record'
2017-11-21T11:35:42.942716264Z   app/controllers/projects/pipelines_settings_controller.rb:9:in `update'
2017-11-21T11:35:42.942719248Z   lib/gitlab/i18n.rb:46:in `with_locale'
2017-11-21T11:35:42.942721405Z   lib/gitlab/i18n.rb:52:in `with_user_locale'
2017-11-21T11:35:42.942723527Z   app/controllers/application_controller.rb:326:in `set_locale'
2017-11-21T11:35:42.942725944Z   lib/gitlab/middleware/multipart.rb:93:in `call'
2017-11-21T11:35:42.942728384Z   lib/gitlab/request_profiler/middleware.rb:14:in `call'
2017-11-21T11:35:42.942730671Z   lib/gitlab/middleware/go.rb:17:in `call'
2017-11-21T11:35:42.942733038Z   lib/gitlab/etag_caching/middleware.rb:11:in `call'
2017-11-21T11:35:42.942735636Z   lib/gitlab/request_context.rb:18:in `call'
2017-11-21T11:35:42.942738162Z   lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
2017-11-21T11:35:42.942740956Z
2017-11-21T11:35:42.942743674Z
2017-11-21T11:35:42.943122531Z
2017-11-21T11:35:42.943134695Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2017-11-21T11:35:42.943139188Z 2017-11-21_11:35:42.94301 gitlab.mycompany.io @ - - [2017-11-21 11:35:42.920785624 +0000 UTC] "POST /mycompany/myproject/pipelines/settings HTTP/1.1" 500 2902 "https://gitlab.mycompany.io/mycompany/myproject/settings/ci_cd" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393" 0.022169
2017-11-21T11:35:42.943289300Z
2017-11-21T11:35:42.943299696Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2017-11-21T11:35:42.943303346Z 172.17.0.1 - - [21/Nov/2017:11:35:42 +0000] "POST /mycompany/myproject/pipelines/settings HTTP/1.1" 500 2902 "https://gitlab.mycompany.io/mycompany/myproject/settings/ci_cd" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393"
2017-11-21T11:35:43.362561586Z
2017-11-21T11:35:43.362583330Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-11-21T11:35:43.362588243Z Started POST "/api/v4/jobs/request" for 172.17.0.1 at 2017-11-21 11:35:43 +0000
2017-11-21T11:35:43.365088270Z
2017-11-21T11:35:43.365106045Z ==> /var/log/gitlab/gitlab-rails/api_json.log <==
2017-11-21T11:35:43.365110611Z {"time":"2017-11-21T11:35:43.364Z","severity":"INFO","duration":0.81,"db":0.19,"view":0.6200000000000001,"status":204,"method":"POST","path":"/api/v4/jobs/request","params":{"info":{"name":"gitlab-ci-multi-runner","version":"9.5.0","revision":"413da38","platform":"linux","architecture":"amd64","executor":"virtualbox","features":{"variables":"[FILTERED]","image":false,"services":false,"features":false,"cache":false}},"token":"[FILTERED]","last_update":"f881ed08d5dcf8f986888c14f49ed98e"},"host":"gitlab.mycompany.io","response":{"message":"204 No Content"},"ip":"172.17.0.3, 172.17.0.1","ua":"gitlab-ci-multi-runner 9.5.0 (9-5-stable; go1.8.3; linux/amd64)"}
2017-11-21T11:35:43.365495930Z
2017-11-21T11:35:43.365505404Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2017-11-21T11:35:43.365522342Z 2017-11-21_11:35:43.36533 gitlab.mycompany.io @ - - [2017-11-21 11:35:43.36189243 +0000 UTC] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.5.0 (9-5-stable; go1.8.3; linux/amd64)" 0.003403
2017-11-21T11:35:43.365526718Z
2017-11-21T11:35:43.365529571Z ==> /var/log/gitlab/nginx/gitlab_access.log <==
2017-11-21T11:35:43.365532410Z 172.17.0.1 - - [21/Nov/2017:11:35:43 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "" "gitlab-ci-multi-runner 9.5.0 (9-5-stable; go1.8.3; linux/amd64)"
...

What could I do to try and narrow this down further? I’m tempted to start a new project and potentially import this one into it to see if that fixes it…

1 Like
  1. sudo gitlab-rails c

Wait info about env…

  1. settings = ApplicationSetting.last

  2. settings.update_column(:runners_registration_token_encrypted, nil)

  3. settings.update_column(:encrypted_ci_jwt_signing_key, nil)

  4. exit

  5. gitlab-ctl restart