Invalid byte sequence in UTF-8 on Gitlab installation

Hi, I’m trying to install Gitlab via Helm chart but my gitlab-unicorn, sidekiq and runner pods are always crashing.

Setup
Kubernetes version: 1.15.6
Helm version: 3.0.1
Storage: rook-ceph with rook-ceph-block StorageClass

Helm values:

certmanager:
  install: false
gitlab-runner:
  runners:
    privileged: true
global:
  email:
    display_name: ****
    from: ****
    reply_to: ****
  hosts:
    domain: ****
    https: false
  ingress:
    annotations:
      kubernetes:
        io/tls-acme: true
    tls:
      enabled: false
  smtp:
    address: smtp.gmail.com
    authentication: login
    domain: ********
    enabled: true
    force_ssl: false
    openssl_verify_mode: peer
    password:
      secret: smtp-gmail
    port: 587
    starttls_auto: true
    tls: false
    user_name: ******

Relevant logs
Unicorn

bundler: failed to load command: unicorn (/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/unicorn)
ArgumentError: invalid byte sequence in UTF-8
/srv/gitlab/config/initializers/smtp_settings.rb:12:in `strip'
/srv/gitlab/config/initializers/smtp_settings.rb:12:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:657:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:656:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:614:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/usr/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/usr/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `tsort_each_child'
/usr/lib/ruby/2.6.0/tsort.rb:415:in `call'
/usr/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/lib/ruby/2.6.0/tsort.rb:347:in `each'
/usr/lib/ruby/2.6.0/tsort.rb:347:in `call'
/usr/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/usr/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
/srv/gitlab/config/environment.rb:6:in `<top (required)>'
/srv/gitlab/config.ru:19:in `require'
/srv/gitlab/config.ru:19:in `block in <main>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/builder.rb:55:in `instance_eval'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/builder.rb:55:in `initialize'
/srv/gitlab/config.ru:1:in `new'
/srv/gitlab/config.ru:1:in `<main>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/unicorn-5.4.1/lib/unicorn.rb:56:in `eval'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/unicorn-5.4.1/lib/unicorn.rb:56:in `block in builder'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:795:in `build_app!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:139:in `start'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/unicorn:23:in `<top (required)>'

Sidekiq pod

invalid byte sequence in UTF-8
/srv/gitlab/config/initializers/smtp_settings.rb:12:in `strip'
/srv/gitlab/config/initializers/smtp_settings.rb:12:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:285:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:657:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:170:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:656:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:614:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:613:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
/usr/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/lib/ruby/2.6.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/usr/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.6.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:50:in `tsort_each_child'
/usr/lib/ruby/2.6.0/tsort.rb:415:in `call'
/usr/lib/ruby/2.6.0/tsort.rb:415:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/lib/ruby/2.6.0/tsort.rb:347:in `each'
/usr/lib/ruby/2.6.0/tsort.rb:347:in `call'
/usr/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
/usr/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
/srv/gitlab/config/environment.rb:6:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `require'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `load'
/srv/gitlab/vendor/bundle/ruby/2.6.0/bin/sidekiq:23:in `<top (required)>'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/usr/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'

Runner

ERROR: Registering runner... failed runner=LM28mwta status=404 Not Found
PANIC: Failed to register this runner. Perhaps you are having network problems

Any help is welcome.

In case someone comes across this issue, I’ve fixed it by reinstalling gitlab without smtp settings.
EDIT: this was due to my smtp password using non standard characters.

Actually, the message means that the special characters in your password were not encoded in UTF-8, but in another encoding, like Latin-1 or WIN1251 (now outdated 8-bit special character sets). Ruby (the interpreter behind a great deal of gitlab’s moving parts) expects them to be UTF-8.

It is highly recommended to switch your entire working environment to UTF-8. If you’re a Linux user, this Stackexchange question has a ton of information about making that switch: https://unix.stackexchange.com/questions/23389/how-can-i-set-vims-default-encoding-to-utf-8

Before making the change though, one bit of advice: temporarily change your password to not use special characters, and set up a test account to make sure that you can login with a utf-8 password.