502 error after upgrade from 7.6.2 to 7.14.3

Hi,

I need to start upgrading gitlab from version 7.6.2 to latest version and my plan is to upgrade to latest 7.X, them 8.0 then latest 8.X and so on, but I got stuck on the first one

In browser I get 502 error, gitlab taking too much time.

When running “sudo gitlab-ctl reconfigure” I get “FATAL: SocketError: getaddrinfo: Name or service not known”

When running “sudo gitlab-ctl tail” I get

==> /var/log/gitlab/nginx/gitlab_error.log <==
2020/03/22 09:58:48 [error] 18844#0: *28271 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 192.176.50.152, server: gitlab.digitalera.se, request: “GET / HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/”, host: “gitlab.digitalera.se”
2020/03/22 09:58:52 [error] 18844#0: *28271 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 192.176.50.152, server: gitlab.digitalera.se, request: “GET / HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/”, host: “gitlab.digitalera.se”
2020/03/22 10:22:04 [error] 18844#0: *28283 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 192.176.50.152, server: gitlab.digitalera.se, request: “GET / HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/”, host: “gitlab.digitalera.se”
2020/03/22 10:42:20 [error] 1201#0: *6 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 192.176.50.152, server: gitlab.digitalera.se, request: “GET / HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/”, host: “gitlab.digitalera.se”

When running “sudo gitlab-ctl status” I get

run: logrotate: (pid 16784) 2492s; run: log: (pid 977) 6094s
run: postgresql: (pid 975) 6094s; run: log: (pid 964) 6094s
run: redis: (pid 968) 6094s; run: log: (pid 963) 6094s
run: sidekiq: (pid 16963) 5s; run: log: (pid 962) 6094s
run: unicorn: (pid 17016) 1s; run: log: (pid 976) 6094s

My /etc/gitlab/gitlab.rb looks like

"external_url ‘https://gitlab.digitalera.se’

gitlab_rails[‘smtp_enable’] = true
gitlab_rails[‘smtp_address’] = “smtp.digitalera.se”
gitlab_rails[‘smtp_port’] = 587
gitlab_rails[‘smtp_user_name’] = “admin@digitalera.se”
gitlab_rails[‘smtp_password’] = “XXXXXXXXXX”
gitlab_rails[‘smtp_domain’] = “smtp.digitalera.se”
gitlab_rails[‘smtp_authentication’] = “login”
gitlab_rails[‘smtp_enable_starttls_auto’] = false
gitlab_rails[‘gitlab_email_from’] = “admin@digitalera.se”

postgresql[‘shared_buffers’] = “100MB”

unicorn[‘worker_processes’] = 1

nginx[‘enable’] = false
nginx[‘redirect_http_to_https’] = true
nginx[‘ssl_certificate’] = “/etc/gitlab/ssl/ssl.cer”
nginx[‘ssl_certificate_key’] = “/etc/gitlab/ssl/ssl.key”

web_server[‘external_users’] = [“www-data”]"

Any ideas? Please advise!

Thanks,

/Per

7.6.2 is more than 5 years old, but it seems like there are still later 7. packages available, so it might actually not be too late.

I have no experience with GitLab versions that old, but the error messages suggest to me that your GitLab server is not actually reachable via HTTP on gitlab.digitalera.se.

Where can I find those older packages?

gitlab.digitalera.se is the same server as I get this error messages. Thats where I have this gitlab installation that have been working for more than 5 years until this upgrade attempt. Is it some internal process (client 192.176.50.152? Dont recognize that ip, but I guess it´s internal because starting with 192?) that tries to connect to http://gitlab.digitalera.se?

Only 192.168.0.0/16 is reserved for local use by RFC1918, so if anybody is using that ip internally they are not RFC compliant, and things will break. A reverse lookup of that IP gives c152.broadband8.quicknet.se, as that is on .se like the domain you want your gitlab server to be on, I guess you might be able to figure what is going on.

As for the availability of old packages, I looked at

that page shows each version several times tagged by various distributions, I guess that shows which distributions these are available for (I don’t have access to any server running any of those old distributions, so I can’t check).

Ok, I found that 192.176.50.152 is the ip of my computer that I use for ssh to gitlab.digitalera.se server. I just don´t get why connection is refused? Both http and https works, I can ssh into the server, I have my pub key for authorization. Any ideas?

I have also looked at gitlab.com for old packages on that link you suggested, but the oldest version I find is gitlab-ee 7.11 and gitlab-ce 7.10. I can´t find 7.6.2 anywhere.

I couldn’t see 7.6.2 either, but as I understood the question, that was the version you already have, so you don’t need that to upgrade.

As you plan to go to the latest 7.x first sounds good (in line with the current recommendations, I don’t know if it was different 5 years ago), I would guess that the versions actually available are enough.

That 192-IP is just shown as the client for some failed requests, so it doesn’t really have any thing to do with the server. What looks weird in those log lines is http://unix/, that looks like you’ve mixed up some protocols, and maybe is doing something in a (very) wrong way.

I don’t really understand what you’re doing, so please provide some more details, perhaps also on how the server was installed.

.Henrik

Hi @perforsberg, thanks for posting on our forum!

First of all I’d suggest restarting GitLab’s components with sudo gitlab-ctl restart to see if this resolves or changes the issue you’re having.

There are a few settings in your gitlab.rb that could be causing this problem:

postgresql[‘shared_buffers’] = “100MB”
unicorn[‘worker_processes’] = 1
nginx[‘enable’] = false

For unicorn['worker_processes'], GitLab expects a minimum of two workers. Can you please change this by settingunicorn['worker_processes'] = 2

For nginx['enable'] = false, having nginx turned off in combination with your other nginx settings (http to https, certificate paths) could be also be problematic. I suspect this might be fixed by settingnginx['enable'] = true.

Last but not least, postgresql[‘shared_buffers’] = “100MB” is quite low. 256MB is default minimum, but we suggest using up to 1/4 of available RAM. If you have 8Gb ram, giving the shared buffers more membory (100MB > up to 2GB) will result in better performance.

How many CPU cores and how much RAM do you have on the GitLab instance?

Can you update your gitlab.rb with these configuration values:

postgresql[‘shared_buffers’] = “512MB”
unicorn[‘worker_processes’] = 2
nginx[‘enable’] = true

Then run gitlab-ctl reconfigure, and let us know if the error persists or changes?

Thanks!

Hi,

I made the changes in gitlab.rb (the settings I had before worked until I made the upgrade to 7.14.3 attempt)

Same error as before:

root@u3288988-01:~# sudo gitlab-ctl restart
ok: run: logrotate: (pid 6600) 0s
ok: run: postgresql: (pid 6619) 0s
ok: run: redis: (pid 6622) 0s
ok: run: sidekiq: (pid 6628) 0s
ok: run: unicorn: (pid 6639) 0s
root@u3288988-01:~# sudo gitlab-ctl reconfigure
[2020-03-23T19:47:25+01:00] FATAL: SocketError: getaddrinfo: Name or service not known

As said I have no experience with GitLab versions that old so I don’t know if gitlab-ctl status was only supposed to output 5 lines, but in the versions I know it has checked services in alphabetical order, and the services shown here are in alphabetical order, and unicorn fairly late in the alphabet, so I see no reason to assume it’s not as it should be. And it says all 5 services are running.

I have no idea what name or service it’s trying to contact when it gives that error. If it was my server I would probably try strace'ing `gitlab-ctl reconfigure, but that can produce a lot of output, and it might take some time to find anything of relevance in it.