I tried to upgrade my Omnibus GitLab instance from 14.9.2 to 14.10.1 using Docker image and almost everything went fine - the full migration procedure finished successfully. But Mattermost is not running/available, so I’ve started checking logs and found the following errors:
Running handlers:
Running handlers complete
Chef Infra Client finished, 436/1700 resources updated in 03 minutes 24 seconds
gitlab Reconfigured!
Checking for unmigrated data on legacy storage
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: OK
The latest version 12.7 is already running, nothing to do
==> /var/log/gitlab/mattermost/mattermost.log <==
{"timestamp":"2022-05-03 07:11:11.434 Z","level":"error","msg":"License key from https://mattermost.com required to unlock enterprise features.","caller":"app/license.go:90","error":"resource: License id: "}
{"timestamp":"2022-05-03 07:11:12.103 Z","level":"error","msg":"Failed to update assets subpath from config","caller":"web/static.go:26","error":"failed to update root.html with subpath /: open /opt/gitlab/embedded/service/mattermost/client/root.html: permission denied"}
==> /var/log/gitlab/mattermost/state <==
==> /var/log/gitlab/mattermost/current <==
2022-05-03_07:12:15.85260 {"timestamp":"2022-05-03 07:12:15.852 Z","level":"info","msg":"Server.Shutdown","caller":"app/plugin_api.go:928","plugin_id":"focalboard"}
2022-05-03_07:12:15.85296 {"timestamp":"2022-05-03 07:12:15.852 Z","level":"info","msg":"2022/05/03 07:12:15 [ERR] plugin: plugin server: accept unix /tmp/plugin539124286: use of closed network connection\n","caller":"io/io.go:425","plugin_id":"focalboard","source":"plugin_stderr"}
2022-05-03_07:12:16.25154 {"timestamp":"2022-05-03 07:12:16.251 Z","level":"info","msg":"Server is initializing...","caller":"app/server.go:255","go_version":"go1.16.7"}
2022-05-03_07:12:16.25156 {"timestamp":"2022-05-03 07:12:16.251 Z","level":"info","msg":"Starting websocket hubs","caller":"app/web_hub.go:93","number_of_hubs":16}
2022-05-03_07:12:16.25308 {"timestamp":"2022-05-03 07:12:16.252 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:215","database":"master"}
2022-05-03_07:12:16.26300 {"timestamp":"2022-05-03 07:12:16.262 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:215","database":"replica-0"}
2022-05-03_07:12:17.70836 {"timestamp":"2022-05-03 07:12:17.707 Z","level":"info","msg":"Server is initializing...","caller":"app/server.go:255","go_version":"go1.16.7"}
2022-05-03_07:12:17.71131 {"timestamp":"2022-05-03 07:12:17.708 Z","level":"info","msg":"Starting websocket hubs","caller":"app/web_hub.go:93","number_of_hubs":16}
2022-05-03_07:12:17.71134 {"timestamp":"2022-05-03 07:12:17.710 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:215","database":"master"}
2022-05-03_07:12:17.71601 {"timestamp":"2022-05-03 07:12:17.715 Z","level":"info","msg":"Pinging SQL","caller":"sqlstore/store.go:215","database":"replica-0"}
==> /var/log/gitlab/nginx/gitlab_mattermost_error.log <==
2022/05/03 07:12:39 [error] 641#0: *30 connect() failed (111: Connection refused) while connecting to upstream, client: 10.100.0.106, server: 10.100.0.1, request: "POST /api/v4/users/status/ids HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "10.100.0.1:7181"
2022/05/03 07:12:45 [error] 641#0: *38 connect() failed (111: Connection refused) while connecting to upstream, client: 10.100.0.6, server: 10.100.0.1, request: "POST /api/v4/users/status/ids HTTP/1.1", upstream: "http://127.0.0.1:8065/api/v4/users/status/ids", host: "10.100.0.1:7181"
I have managed to upgrade my Omnibus GitLab instance many times in previous few years without any problem and this is first time when something went wrong. I don’t have anything specific in my setup - it’s almost pure Omnibus GitLab with Mattermost enabled.
Could anyone advice how to fix the problem?