I have the same problem when i upgrade Gitlab, i upgrade to the latest version every 2 month aprox.
Last time, 16.2.4 → 16.3.6 → 16.5.1 , Following this page for the path upgrade, Upgrade Path
Before any upgrade i do gitlab-clt reconfigure to confirm all the configuration and data are OK to upgrade.
Then this error shows up:
Chef::Exceptions::JSON::ParseError: parse error: trailing garbage
mail_auth_token": null } } (right here) ------^
Here all the cinc-stacktrace.out:
Generated at 2023-11-08 16:06:46 +0100
Chef::Exceptions::JSON::ParseError: parse error: trailing garbage
mail_auth_token": null } } (right here) ------^
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/json_compat.rb:33:in `rescue in parse'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/json_compat.rb:30:in `parse'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/json_compat.rb:37:in `from_json'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:46:in `load_gitlab_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:53:in `read_gitlab_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:167:in `generate_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:194:in `generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:24:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:233:in `load_ruby_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:203:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:417:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:373:in `block in include_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:372:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:372:in `include_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/include_recipe.rb:26:in `include_recipe'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:27:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:233:in `load_ruby_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:203:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:417:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:228:in `block in compile_recipes'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:225:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:225:in `compile_recipes'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:109:in `compile'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:247:in `load'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/policy_builder/expand_node_object.rb:103:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:501:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:281:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
>>>> Caused by FFI_Yajl::ParseError: parse error: trailing garbage
mail_auth_token": null } } (right here) ------^
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/parser.rb:87:in `do_yajl_parse'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/parser.rb:87:in `parse'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/ffi-yajl-2.6.0/lib/ffi_yajl/parser.rb:46:in `parse'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/json_compat.rb:31:in `parse'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/json_compat.rb:37:in `from_json'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:46:in `load_gitlab_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:53:in `read_gitlab_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:167:in `generate_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:194:in `generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:24:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:233:in `load_ruby_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:203:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:417:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:373:in `block in include_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:372:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:372:in `include_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/dsl/include_recipe.rb:26:in `include_recipe'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:27:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/mixin/from_file.rb:34:in `from_file'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:233:in `load_ruby_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/cookbook_version.rb:203:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:417:in `load_recipe'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:228:in `block in compile_recipes'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:225:in `each'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:225:in `compile_recipes'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context/cookbook_compiler.rb:109:in `compile'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/run_context.rb:247:in `load'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/policy_builder/expand_node_object.rb:103:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/3.0.0/forwardable.rb:238:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:501:in `setup_run_context'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/client.rb:281:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:264:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application/base.rb:352:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-17.10.0/lib/chef/application.rb:67:in `run'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/chef-bin-17.10.0/bin/cinc-client:25:in `<top (required)>'
/opt/gitlab/embedded/bin/cinc-client:25:in `load'
/opt/gitlab/embedded/bin/cinc-client:25:in `<main>'
The troubleshoot is to open /etc/gitlab/gitlab-secret.json with any editor and clean the garbage at the end of the file:
.
.
.
"mailroom": {
"incoming_email_auth_token": null,
"service_desk_email_auth_token": null
}
}
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^......
And i ask, why this happen?
The permissions of the file are:
-rw------- 1 root root 19K Nov 9 13:24 gitlab-secrets.json
Thanks in advance.