On going 502 error no idea why

I have set up gitlab on cent OS but I am got stuck in a 502 error and
not sure where to go, I have upgraded my server as I feared it was
Memory but the error is the same after upgrade.

gitlab.rb

external_url 'http://gitlab.adamprocter.co.uk:8081'
web_server['external_users'] = ['www-data']

My server is running its own nginx as well for other service on port
80 etc but I could not get gitlab to even get a 502 with -
nginx[‘enable’] = false and my own settings in my custom.conf so I think
this should work configure seem ok as does restart

ok: run: gitlab-workhorse: (pid 32665) 1s
ok: run: logrotate: (pid 32673) 0s
ok: run: nginx: (pid 32680) 1s
ok: run: postgresql: (pid 32697) 0s
ok: run: redis: (pid 32705) 0s
ok: run: sidekiq: (pid 32733) 0s
ok: run: unicorn: (pid 32737) 0s

but tail has this error

 2016-04-15_20:31:17.45447 2016/04/15 16:31:17 error: proxyRoundTripper: GET "/" failed with: "dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: permission denied"

also further in the log

2016-04-15_20:45:27.58466   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load'
2016-04-15_20:45:27.58469   from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>'
2016-04-15_20:45:27.60371 master failed to start, check stderr log for details
2016-04-15_20:45:28.61194 failed to start a new unicorn master
2016-04-15_20:45:28.63357 starting new unicorn master
2016-04-15_20:45:30.23368 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.9.0/lib/unicorn/http_server.rb:778:in `initialize': Permission denied @ rb_sysopen - /var/log/gitlab/unicorn/unicorn_stderr.log (Errno::EACCES)

I had tried to edit permissions but I dont know if I did it right… and made no difference

sudo chmod 0660 /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
sudo chown nginx:gitlab-www /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket

What is your output of running this command: id www-data

id: www-data: no such user

I made some changes to make sure I am not using gitlab nginx but my own server and nginx and get 502 bad gateway error still.

I edited rb file as follows

external_url 'http://gitlab.adamprocter.co.uk:8081'
web_server['external_users'] = ['nginx']
nginx['enable'] = false
unicorn['enable'] = true

And my custom nginx as this

upstream gitlab {
    server unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket;
}

server {
    listen 217.147.85.86:8081 default_server;
    listen [::]:8081 default_server ipv6only=on;

    root /usr/share/nginx/html;
    index index.html index.htm;
    server_name gitlab.adamprocter.co.uk;

    location / {
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header Host $http_host;
       proxy_redirect off;
       proxy_pass http://gitlab;
    }
}

But I get error

2016/04/22 06:07:31 [error] 1826#0: *34384 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: 86.137.111.86, server: gitlab.adamprocter.co.uk, request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/", host: “gitlab.adamprocter.co.uk:8081"

tail also loops this now! :frowning:

2016-04-22_10:19:18.14455 master failed to start, check stderr log for details
2016-04-22_10:19:19.16765 failed to start a new unicorn master
2016-04-22_10:19:19.22892 starting new unicorn master

You should be able to find steps on how to use an unbundled web server in this case, nginx here: http://doc.gitlab.com/omnibus/settings/nginx.html#using-a-non-bundled-web-server.

Sorry for the earlier confusion, for centos, the web server user is going to be nginx rather than www-data.

You should also be able to find a link to available sample configs under https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/nginx

Then finally run a check on your system and paste the output here:
sudo gitlab-rake gitlab:check SANITIZE=1

Also include the output of sudo gitlab-ctl tail on the next message even though this seems to be just a unicorn issue

1 Like

Also nginx should be in the gitlab-www group.

the rake output -

 Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
 Check GitLab API access: FAILED: Failed to connect to internal API
 gitlab-shell self-check failed

and tail

 2016-04-26_12:36:20.29704 master failed to start, check stderr log for details
 2016-04-26_12:36:21.30506 failed to start a new unicorn master
 2016-04-26_12:36:21.32684 starting new unicorn master

We’re having the same issue here. Site comes up for us, runs for a while, then goes down.

Output from gitlab-ctl tail indicates API is timing out, and gitlab-rake gitlab:check also reports this.

We’re on 8.7.0 of gitlab, on Ubuntu 14.04, adamproctor did not report his operating system yet. Wonder if they are the same.

kelvin: What do you mean “this is a unicorn issue”. Is unicorn the Gitlab Web API Service at /api/v3/… Would having directory /opt/gitlab/var/unicorn not be writable by the unicorn process, which runs as git, if that’s what’s in the .yml file as the unicorn username, cause a similar issue?

adam: Does restarting with gitlab-ctl restart get you back online for a bit?

I cant get online at all yet :frowning:

You checked that if gitlab services run as user git, that all your gitlab files are owned by, and readable by git, and that var (writable) dirs. are writeable by git user?