13.4 > 13.5 now 502 error on update of latest CE - permissions error but I havent changed those :(

How to Use GitLab Upgrade
1

all has been well and then I updated to latest CE update yesterday 13.5.3 (This may have been a jump from 13.4.6) I cant recall previous version I try to never miss an update but…

CentOS 8 / using my own nginx configure as stated was all previously working

And now I have a 502 error on loading my page which is so frustrating

Tail is fine, check all fine. everything is up so when I look in the error log and its advises;

1409#0: *183 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (2: No such file or directory) while connecting to upstream, client: 185.191.171.8, server: gitlab.adamprocter.co.uk, request: "GET /adamprocter/nodenoggin/tree/9791da965acc2de1ebe2fdd5d27770b0f909c376/node_modules/@babel/code-frame HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/adamprocter/nodenoggin/tree/9791da965acc2de1ebe2fdd5d27770b0f909c376/node_modules/@babel/code-frame", host: "gitlab.adamprocter.co.uk"

Restarted nginx, server and reconfigured a number of times

Which from all my digging seemed to be permission related… not that I have changed any permissions but no matter what I try in terms of permission I cant seem to get past this error and everything seems to suggest the whole thing is up and running fine

Any help much appreciated as I really need to push some code changes to my server, kinda regret updating.

System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.5.3
Revision:	eaa194f15e6
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.9
URL:		https://gitlab.adamprocter.co.uk
HTTP Clone URL:	https://gitlab.adamprocter.co.uk/some-group/some-project.git
SSH Clone URL:	git@gitlab.adamprocter.co.uk:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.11.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git
2

Hi @adamprocter,

It sounds like there’s an issue with nginx, and possibly an issue with TLS/SSL certificates.

To troubleshoot this, I suggest running a few diagnostic commands both locally and on the server running GitLab and looking for relevant errors in either the gitlab and/or nginx logs.

On the GitLab server:

# tail the gitlab logs
gitlab-ctl tail | tee -a /tmp/gitlab.log
# tail the nginx logs
tail -f /path/to/nginx/logs | tee -a /tmp/nginx.log

After that, try to connect to localhost

# attempt to connect to localhost
curl -L -vvv localhost
curl -L -k -vvv localhost
echo | /opt/gitlab/embedded/bin/openssl s_client -connect localhost:443

You can also open the homepage in your browser or run the curl commands above locally replacing localhost with gitlab.yourname.co.uk.

When you’re getting the 502 errors in the frontend, there will be additional details about whats going to cause the 502 in both gitlab logs and the nginx log.

Specifically, the NGINX access/error logs and these GitLab logs will contain info on the problem that can be used to find a solution.

==> /var/log/gitlab/gitlab-rails/production.log <==
==> /var/log/gitlab/gitlab-rails/production_json.log <==
==> /var/log/gitlab/gitlab-workhorse/current <==

Would you be able or willing to try the NGINX service that ships with GitLab Omnibus instead of using external NGINX?

If you’re not using NGINX on this server for any other purposes, the easy boring solution would be to use the NGINX bundled with GitLab Omnibus.
The quickest solution to get you back up and running would be to stop/disable the external NGINX daemon, enable the GitLab NGINX service and letsencrypt integration, and gitlab-ctl reconfigure. If this is not an option, I think analyzing your nginx logs and configuration will be a good next step.

3

Thanks I cannot run gitlab nginx as I am running other things on the server… I want and should docker the gitlab really but its a low priority and I dont know how to do it yet with my SSL certs (although this has bumped its priority!)

There seems to e something up with SSL although like I said I have not changed ?

the echo gets

Verify return code: 21 (unable to verify the first certificate)

off to google

4

my niginx config starts with the right location

Secure does all the real work

server {
  listen 217.147.85.104:443 ssl;
  listen [::]:443 ssl  ipv6only=on;
  server_name gitlab.adamprocter.co.uk;
  server_tokens off;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
  #ssl on;
  ssl_certificate     /etc/nginx/ssl/gitlab.adamprocter.co.uk.crt;
  ssl_certificate_key /etc/nginx/ssl/gitlab.adamprocter.co.uk.key;
  
## GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
  ssl_ciphers  'THIS HAS CIPHER IN';
  ssl_protocols  TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_session_cache  shared:SSL:10m;
  ssl_session_timeout  5m;

  ## Individual nginx logs for this GitLab vhost
  access_log  /var/log/nginx/gitlab_access.log;
  error_log   /var/log/nginx/gitlab_error.log;

  location / {
    
    client_max_body_size 0;
    gzip off;

    ## https://github.com/gitlabhq/gitlabhq/issues/694
    ## Some requests take more than 30 seconds.
    proxy_read_timeout      300;
    proxy_connect_timeout   300;
    proxy_redirect          off;

    proxy_http_version 1.1;

    proxy_set_header    Host                gitlab.adamprocter.co.uk;
    proxy_set_header    X-Forwarded-Host    "";
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-Ssl     on;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_pass http://gitlab-workhorse;
  }

  error_page 404 /404.html;
  error_page 422 /422.html;
  error_page 500 /500.html;
  error_page 502 /502.html;
  location ~ ^/(404|422|500|502)\.html$ {
    root /opt/gitlab/embedded/service/gitlab-rails/public;
    internal;
  }

  # Also include LetsEncrypt on secure
  location ~ /\.well-known {
    root /usr/share/nginx/html;
  }
  
  location ~ /\.well-known\/acme-challenge {
	 root /usr/share/nginx/html;
    
}
  
}
5

no issues is the other logs… but I am not convinced the 502 is the SSL but its the only thing to go on but I cannot understand what has changed in that regard

6

Your SSL looks fine to me from visiting the url. I expect it’s more likely to do with the proxy stuff trying to redirect to something that isn’t running.

Can you check output from: gitlab-ctl status to make sure all services are running?

Also, are you running selinux? Do you have all required selinux packages installed?

7

I have added an intermediate certificate as well so I now get another error in the running

curl -L -vvv localhost

curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate.

Here are the services running and selinux was already disabled

run: gitaly: (pid 12447) 190360s; run: log: (pid 2646) 250001s
run: gitlab-workhorse: (pid 13474) 190209s; run: log: (pid 2655) 250001s
run: logrotate: (pid 11083) 1377s; run: log: (pid 2645) 250001s
run: postgresql: (pid 5863) 192179s; run: log: (pid 2644) 250001s
run: puma: (pid 13681) 190183s; run: log: (pid 2648) 250001s
run: redis: (pid 11924) 190437s; run: log: (pid 2647) 250001s
run: sidekiq: (pid 12861) 190245s; run: log: (pid 2658) 250001s

SELinux status: disabled

8

You will get an error using curl on localhost, because the cert is purchased for your domain name. That is normal output. Your best bet would be to use your gitlab.yourdomain.co.uk with curl.

9

thanks I wasnt sure but kinda assumed that… I just get 502 just like the browser

10

You have this in nginx but not this:

  upstream gitlab-workhorse {
    server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket;
  }

I checked this with my nginx under gitlab-ce. This is why your proxy for gitlab-workhorse isn’t working I think unless you have that config elsewhere…

11

sorry I have that earlier

upstream gitlab-workhorse {
  server unix:/var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0;
}

# Non-secure redirects to secure (except for LetsEncrypt)
server {
  listen 217.147.85.104:80;
  listen [::]:80 ipv6only=on;
  server_name gitlab.adamprocter.co.uk;
  server_tokens off; 
  
  ## Don't show the nginx version number, a security best practice
  

  # Handle LetsEncrypt validation
  location ~ /\.well-known {
  	root /usr/share/nginx/html;
  	allow all;
  
  
  }

  location ~ /\.well-known\/acme-challenge {
  	root /usr/share/nginx/html;
allow all;

}


  # Redirect all other requests to secure
  location / {
  return 301 https://$http_host$request_uri;
 	 }
 	 
 	 
}
12

From your first post with initial error, you have gitlab-workhorse in the logs, but gitlab-workhorse/socket instead of gitlab-workhorse/sockets/socket like my post above.

13

Change that line to:

server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket fail_timeout=0;

1 Like
14

yes yes yes - amazing ! thank you maybe the location changed from 13.4 > 13.5

1 Like
15

It’s possible, to be honest I’m not sure, would have to create a new instance 13.4 or earlier to find out for 100%. However, glad it’s all working now.