Backup error: OpenSSL::SSL::SSLError: SSL_CTX_load_verify_file: system lib

It appears upgrading to Ubuntu 22.04 was a bad idea, which, based on several other ruby threads elsewhere, points this error at an incompatibility between Ruby 3.0 and OpenSSL 3.0. Hope I am wrong!

The error when running backups with --trace:

Caused by:
OpenSSL::SSL::SSLError: SSL_CTX_load_verify_file: system lib
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `socket'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:121:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/mock.rb:57:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:286:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:677:in `_request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:672:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/requests/storage/initiate_multipart_upload.rb:29:in `initiate_multipart_upload'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:324:in `multipart_save'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:279:in `save'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/home/git/gitlab/lib/backup/manager.rb:345:in `upload'
/home/git/gitlab/lib/backup/manager.rb:234:in `run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:47:in `create'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:13:in `block in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:62:in `lock_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:101:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/bin/rake:25:in `load'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/bin/rake:25:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `load'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:23:in `run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli.rb:451:in `exec'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli.rb:34:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli.rb:28:in `start'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/exe/bundle:28:in `block in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:backup:create

Anyone have any pointers on how we might mitigate this issue? Everything else seems to be running correctly. Just uploading the backups to AWS is where we are stuck.

I guess if all else fails we can backup locally and us s3cmd to sync the backups.

What version of Gitlab? Perhaps it’s too old for the libraries on Ubuntu 22.04 hence the SSL errors? Or was it upgraded to the latest Gitlab before you moved to Ubuntu 22.04?

From the path being under /home, I’m guessing it’s a source installation, and not the omnibus package install.

Sorry 'bout that, I thought I had included all that info:

System information
System:		Ubuntu 22.04
Current User:	git
Using RVM:	no
Ruby Version:	3.1.4p223
Gem Version:	3.3.26
Bundler Version:2.5.4
Rake Version:	13.0.6
Redis Version:	6.2.9
Sidekiq Version:7.1.6
Go Version:	go1.20.8 linux/amd64

GitLab information
Version:	16.8.5
Revision:	90576f95c15
Directory:	/home/git/gitlab
DB Adapter:	PostgreSQL
DB Version:	14.11

Issue was evident in v16.1 which is when we upgraded to Ubuntu 22.04, and is still evident in v16.8.

In case it helps others. The solution was fairly simple.

ln -s /etc/ssl/certs/ca-certificates.crt /usr/lib/ssl/cert.pem

Ruby is looking for /usr/lib/ssl/cert.pem and it wasn’t there. Creating a symlink to ca-certificates.crt was the solution.

After creating the link, backups work as expected.