Backup error: OpenSSL::SSL::SSLError: SSL_CTX_load_verify_file: system lib

outofcontrol · April 10, 2024, 2:06pm

It appears upgrading to Ubuntu 22.04 was a bad idea, which, based on several other ruby threads elsewhere, points this error at an incompatibility between Ruby 3.0 and OpenSSL 3.0. Hope I am wrong!

The error when running backups with --trace:

Caused by:
OpenSSL::SSL::SSLError: SSL_CTX_load_verify_file: system lib
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/ssl_socket.rb:139:in `initialize'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `new'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:474:in `socket'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:121:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/mock.rb:57:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/instrumentor.rb:34:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/idempotent.rb:19:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/middlewares/base.rb:22:in `request_call'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/excon-0.99.0/lib/excon/connection.rb:286:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:35:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:677:in `_request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/storage.rb:672:in `request'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/requests/storage/initiate_multipart_upload.rb:29:in `initiate_multipart_upload'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:324:in `multipart_save'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-aws-3.18.0/lib/fog/aws/models/storage/file.rb:279:in `save'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/home/git/gitlab/lib/backup/manager.rb:345:in `upload'
/home/git/gitlab/lib/backup/manager.rb:234:in `run_all_create_tasks'
/home/git/gitlab/lib/backup/manager.rb:47:in `create'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:13:in `block in create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:62:in `lock_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/home/git/gitlab/lib/tasks/gitlab/backup.rake:101:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/bin/rake:25:in `load'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/bin/rake:25:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `load'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:58:in `kernel_load'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli/exec.rb:23:in `run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli.rb:451:in `exec'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli.rb:34:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/cli.rb:28:in `start'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/exe/bundle:28:in `block in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/git/gitlab/vendor/bundle/ruby/3.1.0/gems/bundler-2.5.4/exe/bundle:20:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:backup:create

Anyone have any pointers on how we might mitigate this issue? Everything else seems to be running correctly. Just uploading the backups to AWS is where we are stuck.

I guess if all else fails we can backup locally and us s3cmd to sync the backups.

iwalker · April 10, 2024, 3:10pm

What version of Gitlab? Perhaps it’s too old for the libraries on Ubuntu 22.04 hence the SSL errors? Or was it upgraded to the latest Gitlab before you moved to Ubuntu 22.04?

From the path being under /home, I’m guessing it’s a source installation, and not the omnibus package install.

outofcontrol · April 10, 2024, 3:26pm

Sorry 'bout that, I thought I had included all that info:

System information
System:		Ubuntu 22.04
Current User:	git
Using RVM:	no
Ruby Version:	3.1.4p223
Gem Version:	3.3.26
Bundler Version:2.5.4
Rake Version:	13.0.6
Redis Version:	6.2.9
Sidekiq Version:7.1.6
Go Version:	go1.20.8 linux/amd64

GitLab information
Version:	16.8.5
Revision:	90576f95c15
Directory:	/home/git/gitlab
DB Adapter:	PostgreSQL
DB Version:	14.11

Issue was evident in v16.1 which is when we upgraded to Ubuntu 22.04, and is still evident in v16.8.