Could not create GitLab API client: getting certificate - Another upgrade issue related to certs after upgrade

{"error":"[Sentry] DsnParseError: invalid url: parse \"https://\u003ckey\u003e:\u003csecret\u003e@sentry.io/\u003cproject\u003e\": net/url: invalid userinfo","level":"warning","msg":"unable to initialize sentry client","pid":362554,"time":"2025-10-27T23:29:04.094Z"}
unclean Gitaly shutdown: could not create GitLab API client: getting certificate: loading x509 keypair: open /var/opt/gitlab/gitaly/certificate.pem: no such file or directory

This is my gitaly log output.

I found this solution on the GitLab forum, where @stanhu offered a solution, but I could not see how to implement this solution using the gitlab.rb file, since the gitaly config file should not be manually configured.

I’m pretty close to resolving this GitLab upgrade if I could just get access to my repositories again. However, this gitaly error is really blocking my progress.

Any help here would be greatly appreciated.

-Marc

You haven’t listed what’s in your configuration in /etc/gitlab/gitlab.rb.

The error suggests that you’ve enabled TLS for Gitaly in gitaly[‘configuration’], and Gitaly can’t find the certificate for it.

Stan, thanks for the reply.

I have pasted below only the UNCOMMENTED sections of my gitlab.rb file.

I haven’t needed anything fancy for my GitLab instance, as I use this particular instance to store my “local lab” repositories. We have a larger corporate Git server that I use with “git bundle” to move repos from this GitLab instance.

The gitaly configuration is at the bottom. I saw that the certificate path was an issue, because that certificate is not there in that directory.

However, I couldn’t find in the docs how to add this certificate in properly. The GitLab configuration is extensive, so I have only made adjustments that were outlined in the installation procedures and left the rest of the options alone.

external_url '``https://jme-gitlab.xxx.xxxxx.com``'

gitlab_rails[‘gitlab_repository_downloads_path’] = ‘tmp/repositories’
gitlab_rails[‘gitlab_shell_ssh_port’] = 22
gitlab_rails[‘gitlab_shell_git_timeout’] = 800

gitlab_rails['rack_attack_git_basic_auth'] = {
  'enabled' => false,
  'ip_whitelist' => ["127.0.0.1"],
  'maxretry' => 10,
  'findtime' => 60,
  'bantime' => 3600
}

gitlab_rails['dir'] = "/var/opt/gitlab/gitlab-rails"
gitlab_rails['log_directory'] = "/var/log/gitlab/gitlab-rails"
gitlab_rails['log_group'] = nil


gitlab_rails['initial_root_password'] = "xxxxxxxxxxxxxxxxxx"
gitlab_rails['initial_shared_runners_registration_token'] = "token"

#### Toggle if root password should be printed to STDOUT during initialization
gitlab_rails['display_initial_root_password'] = true

#### Toggle if initial root password should be written to /etc/gitlab/initial_root_password
gitlab_rails['store_initial_root_password'] = true
gitlab_rails['auto_migrate'] = true

user['username'] = "git"
user['group'] = "git"
# user['uid'] = nil
# user['gid'] = nil

##! The shell for the git user
user['shell'] = "/bin/sh"

##! The home directory for the git user
user['home'] = "/home/git"

user['git_user_name'] = "GitLab"
user['git_user_email'] = "gitlab@#{node['fqdn']}"

prometheus['listen_address'] = '0.0.0.0:19090'

The gitaly configuration below…

gitlab_rails[‘internal_api_url’] = ‘https://jme-gitlab.xxx.xxxxx.com/’
gitlab_rails[‘trusted_cert_dir’] = ‘/etc/gitlab/trusted-certs’

gitaly[‘open_files_ulimit’] = 15000 # Maximum number of open files allowed for the gitaly process

##! Service name used to register Gitaly as a Consul service

gitaly[‘consul_service_name’] = ‘gitaly’

##! Semantic metadata used when registering Gitaly as a Consul service

gitaly[‘consul_service_meta’] = {}

gitaly[‘configuration’] = {
storage: [
{
name: ‘default’,
path: ‘/home/jmarcedwards/data-disk-1/git-data/repositories’
},
],
socket_path: ‘/var/opt/gitlab/gitaly/gitaly.socket’,
runtime_dir: ‘/var/opt/gitlab/gitaly/run’,
listen_addr: ‘localhost:8075’,
prometheus_listen_addr: ‘localhost:9236’,
tls_listen_addr: ‘localhost:9075’,
tls: {
certificate_path: ‘/var/opt/gitlab/gitaly/certificate.pem’,
key_path: ‘/var/opt/gitlab/gitaly/key.pem’,
},
graceful_restart_timeout: ‘1m’, # Grace time for a gitaly process to finish ongoing requests
logging: {
dir: “/var/log/gitlab/gitaly”,
level: ‘warn’,
format: ‘json’,
sentry_dsn: ‘https://<key>:<secret>@sentry.io/’,
sentry_environment: ‘production’,
},
prometheus: {
grpc_latency_buckets: [0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0],
},
auth: {
token: ‘’,
transitioning: false, # When true, auth is logged to Prometheus but NOT enforced
},
git: {
catfile_cache_size: 100, # Number of ‘git cat-file’ processes kept around for re-use
bin_path: ‘/opt/gitlab/embedded/bin/git’, # A custom path for the ‘git’ executable
use_bundled_binaries: true, # Whether to use bundled Git.
signing_key: ‘/var/opt/gitlab/gitaly/signing_key.gpg’,
## Gitaly knows to set up the required default configuration for spawned Git
## commands automatically. It should thus not be required to configure anything
## here, except in very special situations where you must e.g. tweak specific
## performance-related settings or enable debugging facilities. It is not safe in
## general to set Git configuration that may change Git output in ways that are
## unexpected by Gitaly.
config: [
{ key: ‘pack.threads’, value: ‘4’ },
{ key: ‘http.http://example.com.proxy’, value: ‘http://example.proxy.com’ },
],
},
hooks: {
custom_hooks_dir: ‘/var/opt/gitlab/gitaly/custom_hooks’,
},
daily_maintenance: {
disabled: false,
start_hour: 22,
start_minute: 30,
duration: ‘30m’,
storages: [‘default’],
},
cgroups: {
mountpoint: ‘/sys/fs/cgroup’,
hierarchy_root: ‘gitaly’,
## memory_bytes: 1048576,
memory_bytes: 40000000000,
cpu_shares: 512,
cpu_quota_us: 400000,
repositories: {
count: 100,
memory_bytes: 12884901888,
## memory_bytes: 0,
cpu_shares: 128,
cpu_quota_us: 200000
},
},
concurrency: [
{
rpc: ‘/gitaly.SmartHTTPService/PostReceivePack’,
max_per_repo: 20,
},
{
rpc: ‘/gitaly.SSHService/SSHUploadPack’,
max_per_repo: 5,
},
],
rate_limiting: [
{
rpc: ‘/gitaly.SmartHTTPService/PostReceivePack’,
interval: ‘1m’,
burst: 10,
},
{
rpc: ‘/gitaly.SSHService/SSHUploadPack’,
interval: ‘1m’,
burst: 5,
},
],
pack_objects_cache: {
enabled: true,
dir: ‘/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache’,
max_age: ‘5m’,
},
}
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 0
letsencrypt['auto_renew_minute'] = nil # Should be a number or cron expression, if specified.
letsencrypt['auto_renew_day_of_month'] = "*/4"
letsencrypt['auto_renew_log_directory'] = '/var/log/gitlab/lets-encrypt'

But, WHY would this have STOPPED WORKING from the original configuration AFTER I performed the upgrade from v17.x (I forgot which version I was at) to v18.5?

Was there just a change in the certificate management for gitaly in the major version upgrade, i.e., v17 to v18?

I could transition this discussion to the forum if you prefer. :grinning_face:

-M

I did find some steps for creating certificates for the Gitaly server. The instructions were instructing to add these certificates to the /etc/gitlab/ssl directory. In that directory were the GitLab certificates, so I was thinking that I should just use those certificates.

I updated my gitaly certificate and key paths. I then ran “gitlab-ctl reconfigure” and then “gitlab-ctl restart”.

When I checked the /var/log/gitlab/gitaly logs, I didn’t see the certificate issue, but am now seeing this “no space left on device”.

This is weird, and now I guess I have to sort this out.

I’m going to restart my logs and check this again.

I’m still unable to access my project repositories, though.

I’m still not sure if I have set up my TLS correctly for gitaly. What do you think?

{"error":"[Sentry] DsnParseError: invalid url: parse \"https://\u003ckey\u003e:\u003csecret\u003e@sentry.io/\u003cproject\u003e\": net/url: invalid userinfo","level":"warning","msg":"unable to initialize sentry client","pid":2678691,"time":"2025-10-28T14:17:27.787Z"}
unclean Gitaly shutdown: failed setting up cgroups: failed creating parent cgroup: mkdir /sys/fs/cgroup/memory/gitaly/gitaly-2678691: no space left on device

I would start here: Deprecations and removals by version | GitLab Docs

Some to mention that may or may not affect you based on a quick look at your configuration:

You can search the deprecations link for info relating to gitaly or TLS. There is something about TLS 1.0 and 1.1 deprecating in 17.x releases, so whether your TLS configuration also needs to be amended, or SSL certificates being upgraded it would need to be checked to see if this is also a potential problem.

Ian:

Thanks for the reference to the deprecations. I looked through it, but could still not see anything relevant. I looked again at my gitaly logs. I didn’t see the cgroups storage issue anymore, but a URL issue with the Sentry DNS. I corrected that, reconfigured, and then had a username issue in the Sentry. I guessed on that and reconfigured again and I got to the error below on the “empty project id”.

I’m wondering here if I can just DISABLE this Sentry stuff from my configuration.

Again, the issue I am having is I can’t access my repositories as shown in the screenclip below.

Yet, in the Admin page, I can see links to ALL of the repositories.

I’m thinking of getting a single license for self-hosted so that I can get someone to look at this with me. Will an online GitLab license provide me also with a license for a self-hosted configuration?

Any other ideas or GitLab contacts that could offer any advice on this problem of not being able to get to my repositories after this GitLab upgrade from v17.x to v18.5.1?

Kind regards, Marc

Yeah you probably could do since Sentry is an app/platform for analysing problems. It’s not necessary or required for Gitlab to operate. I expect it’s something you or your company use with a cloud-based installation of Sentry. I use Sentry but self-hosted on a VM, but I don’t have that linked in with my Gitlab installation. So the real question is, if you need it or not - but probably will require you to login to your cloud sentry.io account to check/verify. If you didn’t configure it, then you may wish to check with whoever else manages your Gitlab and/or your Sentry installation.

As for the rest, my Gitaly defaults to the /var/opt/gitlab/gitaly directory as I don’t have any configured Gitaly options. Yours is most likely configured that way, to use a different storage path to keep it away from /var or /var/opt/gitlab due to lack of disk space on that particular partition, or due to wanting an entirely different location for your Gitlab repository data. So you could also potentially look at reducing the Gitaly configuration to a bare minimum. The other question for TLS with Gitaly is, that is it really necessary? If you were using a Gitaly server external to your main Gitlab installation, then I could understand the need for TLS certificate to ensure data in transit is encrypted. However, based on your configuration you are using Gitaly on the same server as your Gitlab installation because the connection is localhost. At this point, nobody external to the Gitlab server is going to be able to intercept it anyway. So the use of TLS at this point is somewhat moot.

Ian:

I got it working!!! After I removed all of the gitaly settings EXCEPT for the storage location, I got this hmac” error. When I looked up the “hmac” error, it took me here.

I then set the “gitaly” token on the gitaly server and gitaly client (gitlab_rails).

And it worked. Thanks again for communicating with me. Our exchange kept me thinking and probing until I got to this point.

THANK YOU AGAIN!!!

Regards, Marc

1 Like