GitLab shows 502 error

Infrastructure as Code & Cloud Native
,
1

Hi,

After reconfiguring GitLab, it returns the following messages:


==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2023-08-16T13:47:59+08:00","uri":""}

==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2023-08-16T05:48:01.927Z","message":"Starting cluster with 1 processes"}
/opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:457:in `parse': (<unknown>): did not find expected key while parsing a block mapping at line 13 column 5 (Psych::SyntaxError)
        from /opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:457:in `parse_stream'
        from /opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:391:in `parse'
        from /opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:280:in `unsafe_load'
        from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:103:in `initialize'
        from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:60:in `new'
        from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:60:in `instance'
        from /opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:66:in `method_missing'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:177:in `sidekiq_exporter_enabled?'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:185:in `metrics_server_enabled?'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:163:in `start_metrics_server'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:137:in `start_and_supervise_workers'
        from /opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:111:in `run'
        from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:11:in `<main>'
---
==> /var/log/gitlab/sidekiq/current <==
2023-08-16T05:48:05.752Z pid=110086 tid=2nnu WARN: Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 13 column 5
2023-08-16T05:48:05.752Z pid=110086 tid=2nnu WARN: /opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:457:in `parse'
/opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:457:in `parse_stream'
/opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:391:in `parse'
/opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:280:in `unsafe_load'
/opt/gitlab/embedded/lib/ruby/gems/3.0.0/gems/settingslogic-2.0.9/lib/settingslogic.rb:103:in `initialize'

And the port are used as following, the external_url uses 8888 port:

[16:03:47] root@swpmsnew:~# netstat -tupln |grep 80
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 10630/nginx: master
tcp 0 0 0.0.0.0:8060 0.0.0.0:* LISTEN 10630/nginx: master
[16:03:52] root@swpmsnew:~# netstat -tupln |grep 8888
tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN 10630/nginx: master
[16:03:56] root@swpmsnew:~# netstat -tupln |grep 8080

And the GitLab website shows 502 error.

I have no idea about this, could you please help this?
Thanks.

2

And then I updated nginx[‘enable’] = false and it shows as follows:

Running handlers:
[2023-08-16T17:17:31+08:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 496) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 13 column 5
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:8:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1


Deprecations:
* gitaly['storage'] has been deprecated since 15.10 and will be removed in 16.0. In GitLab 15.10, Gitaly's configuration in Omnibus GitLab was changed to structurally match Gitaly's own configuration. Please see the migration instructions at https://docs.gitlab.com/ee/update/#1510

Running handlers complete
[2023-08-16T17:17:31+08:00] ERROR: Exception handlers complete
Infra Phase failed. 12 resources updated in 01 minutes 43 seconds

Deprecations:
* gitaly['storage'] has been deprecated since 15.10 and will be removed in 16.0. In GitLab 15.10, Gitaly's configuration in Omnibus GitLab was changed to structurally match Gitaly's own configuration. Please see the migration instructions at https://docs.gitlab.com/ee/update/#1510

[2023-08-16T17:17:31+08:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
[2023-08-16T17:17:31+08:00] FATAL: ---------------------------------------------------------------------------------------
[2023-08-16T17:17:31+08:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2023-08-16T17:17:31+08:00] FATAL: ---------------------------------------------------------------------------------------
[2023-08-16T17:17:31+08:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 496) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 13 column 5
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:8:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
3

And then I executed gitlab-rake cache:clear --trace, it returns the following error message:

[17:22:18] root@swpmsnew:~# /opt/gitlab/bin/gitlab-rake cache:clear
rake aborted!
Psych::SyntaxError: (<unknown>): did not find expected key while parsing a block mapping at line 13 column 5
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:8:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
4

Now after executing “gitlab-ctl tail nginx”, it shows the following warning message and the GitLab website still cannot be accessed:

2023-09-23_21:46:52.12139 nginx: [emerg] cannot load certificate "/etc/gitlab/ssl/test.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/gitlab/ssl/test.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)

I’ve checked the path of the certificate name and it is correct.
But it shows "No such file or directory" error.
Does anyone know about this?
Thanks.

5

Hi,

Check file permissions on that certificate. Mine that I have in the /etc/gitlab/ssl directory have permissions of 644.

chmod 644 /etc/gitlab/ssl/test.crt

you may also want to check the permissions of the /etc/gitlab/ssl directory as well, and ensure it’s set to 755.

6

Hi,

The permission is as follows:

root@vm:/etc/gitlab# ls -l
drwx------ 2 root root  4096  九  24 00:00 config_backup
-rw-r--r-- 1 root root 75060  九  24 22:19 gitlab.rb
-rw------- 1 root root 19615  九  24 22:19 gitlab-secrets.json
-rw-r--r-- 1 root root     0  九  24 09:10 skip-unmigrated-data-check
drwxr-xr-x 2 root root  4096  九  24 09:36 ssl
drwxr-xr-x 2 root root  4096  九  21 00:28 trusted-certs

root@vm:/etc/gitlab/ssl# ls -l
total 8
-rw-r--r-- 1 root root 2204  九  24 05:25 test.crt
-rw-r--r-- 1 root root 1675  九  24 05:25 test.key

Even I changed the name of the cert to the other name as well as the path in the gitlab.rb, it still shows the following message:

2023-09-23_21:46:52.12139 nginx: [emerg] cannot load certificate "/etc/gitlab/ssl/test.crt": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/gitlab/ssl/test.crt','r') error:2006D080:BIO routines:BIO_new_file:no such file)
7

How are the certificates configured in gitlab.rb? The name must match exactly. If no SSL cert lines are unhashed, then usually the certificate will be the name of your server. So for example, if my server has external_url of gitlab.example.com, then my certificate would be gitlab.example.com.crt and gitlab.example.com.key

The permissions on the files and directory are OK, so either something is wrong in the gitlab.rb configuration file or the contents of the certificate file are invalid.

8

Hi,

The certificate is named as “wildcard_xxx_com_until_2024_xx_xx.crt”, you can replace the “test.crt” I mentioned with the “wildcard_xxx_com_until_2024_xx_xx.crt”.

The external_url is aaa.xxx.com:7777.

I executed “gitlab-ctl tail nginx” on September 25th but the date in the error message is “2023-09-23_21:4X”. How can I find out the related logs or configurations for that time?

Besides, there is the keyword “wildcard_xxx_com_until_2024_xx_xx.crt” in the “/opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf”
and “/opt/gitlab/embedded/nodes/swpms.json” files.

May I delete the files since executing “gitlab-rake cache:clear” is failed?

Thanks.

9

Hi,

No you never edit files under /opt/gitlab or /var/opt/gitlab. All changes are to be made in /etc/gitlab/gitlab.rb and only here. These changes are then propagated to config files under /opt/gitlab or /var/opt/gitlab - when you run:

gitlab-ctl reconfigure

most likely is you have config issues either because of duplicate entries in gitlab.rb or by editing files manually. But without knowing exactly what you have configured and where it is difficult for us to tell.

Another alternative is to copy /opt/gitlab/etc/gitlab.rb.template to /etc/gitlab/gitlab.rb and then apply your configuration changes again from zero. Obviously you need to set external_url, plus any other custom options that are not the default from your original file.

The gitlab-rake command most likely failed because you don’t have your instance running properly because of misconfiguration in gitlab.rb

1 Like
10

Hi @iwalker ,
Thanks for your explanation.

This problem is resolved by copying the gitlab.rb.template file plus other custom options that are not the default from our original file.

The error about the following message is caused by the characters “[” and “]” in the “[GitLab] Server Notification” value of the gitlab_rails[‘gitlab_email_display_name’] attribute.
The characters “[” and “]” were asked to be added before I posted this question.

After removing the “[” and “]” characters, no error message appears.

/opt/gitlab/embedded/lib/ruby/3.0.0/psych.rb:457:in `parse': (<unknown>): did not find expected key while parsing a block mapping at line 13 column 5 (Psych::SyntaxError)
1 Like