WebUI assets failing to load after upgrade (Omnibus 16.4.0-ee.0)

mpounsett · September 26, 2023, 2:21pm

After the last two updates (16.3.3→16.3.4 and 16.3.4→16.4.0) the Gitlab web UI seems to have lost all of its generated assets. We’re seeing 404s for style sheets and icons. This is running Omnibus on Debian Bullseye. We’re using our own Apache, but otherwise it’s a very generic configuration.

From my browser console:

Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.
gitlab.redacted.com/:1 Refused to apply style from 'https://gitlab.redacted.com/assets/application_utilities-e77145525f157f955ee659a045c8699f51814ca7ef5cdd32c45a9bab7764f35b.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
gitlab.redacted.com/:1 Refused to apply style from 'https://gitlab.redacted.com/assets/application-6687cea99d08705f41c22ba6c0f625668940d80a47e99488bf0d3d4bafa9d398.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
gitlab.redacted.com/:1 Refused to apply style from 'https://gitlab.redacted.com/assets/highlight/themes/white-798c2d2c1560fb1734a7653f984135b2ce22a62aa9b46f914905648669930db1.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
gitlab.redacted.com/:1 Refused to apply style from 'https://gitlab.redacted.com/assets/themes/theme_light_green-c6036c3d644a02e1030ad874a9be43756d7790d56ce2651e5e06e73a88488a62.css' because its MIME type ('text/plain') is not a supported stylesheet MIME type, and strict MIME checking is enabled.
gitlab.redacted.com/:115     GET https://gitlab.redacted.com/assets/icons-b25b55b72e1a86a9ca8055a5c421aae9b89fc86363fa02e2109034d756e56d28.svg net::ERR_ABORTED 404 (Not Found)
vue.runtime.esm.js:7372     GET https://gitlab.redacted.com/assets/icons-b25b55b72e1a86a9ca8055a5c421aae9b89fc86363fa02e2109034d756e56d28.svg net::ERR_ABORTED 404 (Not Found)

Checking the “refused to apply style” MIME errors reveal those are 404s.

198.84.225.178 - - [26/Sep/2023:14:10:22 +0000] "GET /assets/application_utilities-e77145525f157f955ee659a045c8699f51814ca7ef5cdd32c45a9bab7764f35b.css HTTP/1.1" 404 371 "https://gitlab.redacted.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"

gitlab-ctl reconfigure runs without any errors that I can see. There are some warnings about a log_group not being set for various components, but that seems harmless. e.g.:

[2023-09-26T14:13:05+00:00] WARN: gitlab-shell does not have a log_group or default logdir mode defined. Setting to 0700.

Likewise, check-config and restart do their thing without errors.

Last time I fixed this with an apt reinstall, but that’s a very heavy hammer. Any suggestions for how to troubleshoot this?

Some possibly-useful detail below. Happy to share configs if it’s useful.

% sudo apt list apache2 gitlab-ee
Listing... Done
apache2/oldstable,now 2.4.56-1~deb11u2 amd64 [installed]
gitlab-ee/bullseye,now 16.4.0-ee.0 amd64 [installed]

% sudo gitlab-ctl check-config && echo "ok"
ok

% sudo gitlab-ctl status
run: alertmanager: (pid 2004428) 95s; run: log: (pid 4066359) 86649s
run: gitaly: (pid 2004453) 95s; run: log: (pid 4066364) 86649s
run: gitlab-exporter: (pid 2004473) 94s; run: log: (pid 4066377) 86649s
run: gitlab-kas: (pid 2004524) 84s; run: log: (pid 4066369) 86649s
run: gitlab-workhorse: (pid 2004538) 84s; run: log: (pid 4066358) 86649s
run: logrotate: (pid 2004553) 84s; run: log: (pid 4066375) 86649s
run: node-exporter: (pid 2004560) 83s; run: log: (pid 4066421) 86649s
run: postgres-exporter: (pid 2004568) 83s; run: log: (pid 4066361) 86649s
run: postgresql: (pid 2004578) 82s; run: log: (pid 4066374) 86649s
run: prometheus: (pid 2004589) 82s; run: log: (pid 4066376) 86649s
run: puma: (pid 2004616) 81s; run: log: (pid 4066365) 86649s
run: redis: (pid 2004623) 81s; run: log: (pid 4066372) 86649s
run: redis-exporter: (pid 2004630) 80s; run: log: (pid 4066426) 86649s
run: registry: (pid 2004637) 80s; run: log: (pid 4066363) 86649s
run: sidekiq: (pid 2005190) 74s; run: log: (pid 4066381) 86649s

iwalker · September 26, 2023, 2:26pm

Try clearing the cache:

gitlab-rake cache:clear

sometimes after updates it will output for services that need restarting. Lately one of this has been redis:

gitlab-ctl restart redis

although I believe you already did a restart, so perhaps clearing the cache will help.

mpounsett · September 26, 2023, 2:29pm

Thanks for the suggestion, but unfortunately this doesn’t seem to be it.

Cache clear on its own didn’t do it. I followed that up with a gitlab-ctl restart to get all components, just in case, and still no joy. We’re still seeing the same errors.

iwalker · September 26, 2023, 2:44pm

Can you also try a:

systemctl restart gitlab-runsvdir

sometimes gitlab-ctl restart doesn’t restart everything and so some processes may have been left behind.

Something else that might potentially be an issue is if Content Security Policy is enabled in /etc/gitlab/gitlab.rb you can disable that by setting it to false, and then see if that might have stopped the assets from loading. Look for this section:

gitlab_rails['content_security_policy'] = {
 'enabled' => true,

if enabled, set to false and run reconfigure/restart after this. All that I can think of right now.

mpounsett · September 26, 2023, 2:52pm

I have tried restarting gitlab-runsvdir as well (and just did it again) with no effect.

That rails setting isn’t even present as a comment in our gitlab.rb. I added it to disable the feature, in case the default is true, but again no joy.

gitlab_rails['content_security_policy'] = {
 'enabled' => false,
}

iwalker · September 26, 2023, 2:59pm

Can you curl your server to view the headers, I’m specifically looking for the output if it’s similar as mine below:

permissions-policy: interest-cohort=()

you can do that with:

curl -I https://gitlab.redacted.com

the above results is from my server, so if you have something different, that means a permission policy is configured somewhere and messing it up.

mpounsett · September 26, 2023, 3:02pm

FYI, I put the security policy setting back to default after the last test failed.

Looks the same to me though… here’s the full headers:

HTTP/1.1 302 Found
Date: Tue, 26 Sep 2023 15:00:59 GMT
Server: Apache/2.4.56 (Debian)
Cache-Control: no-cache
Content-Security-Policy:
Content-Type: text/html; charset=utf-8
Location: https://gitlab.redacted.com/users/sign_in
Permissions-Policy: interest-cohort=()
Referrer-Policy: strict-origin-when-cross-origin
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Gitlab-Meta: {"correlation_id":"01HB903Y5Y3G4CB6YRX5VCTWXC","version":"1"}
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 01HB903Y5Y3G4CB6YRX5VCTWXC
X-Runtime: 0.034064
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block
Strict-Transport-Security: max-age=15768000;includeSubdomains

iwalker · September 26, 2023, 3:19pm

Yeah there’s no problem there with CSP or even Permissions and Referrer policy. Not entirely sure what is causing that error - could be a red-herring though and nothing related to the problem you have as from your original post:

Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest-cohort'.

however we’ve just verified that it exists in the headers, so weird that is showing up. However, just found this relating to webUI assets: Upgrade GitLab by using the GitLab package | GitLab so not sure if anything maybe there helps out?

mpounsett · September 26, 2023, 3:29pm

Aha! Thanks, I had not found that page when I was googling. Looks like it was a dangling puma socket file that was the problem.

QiaXi · February 17, 2024, 9:32am

Hello, could you elaborate "Looks like it was a dangling puma socket file that was the problem." a bit further?

I might be having the same issue (not sure if it’s during 6.3.3→16.3.4 or 16.3.4→16.4.0 or elsewhere. I don’t check the web UI very often. The only thing I can be sure is it was working fine about half a year ago …)

I am currently running 16.8.2-ee.0, "content_security_policy" is configured to "false". "permissions-policy: interest-cohort=() " is there, e.g.

HTTP/1.1 302 Found
Date: Sat, 17 Feb 2024 09:24:03 GMT
Server: nginx
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Security-Policy:
Permissions-Policy: interest-cohort=()
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Gitlab-Meta: {"correlation_id":"01HPV62GCAT8A8FADJRE25S5RY","version":"1"}
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 01HPV62GCAT8A8FADJRE25S5RY
X-Runtime: 0.277803
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block
Strict-Transport-Security: max-age=63072000
Referrer-Policy: strict-origin-when-cross-origin

And all the steps suggested by @iwalker have been tried with no luck…

What I can see is the WebUI is trying to use assets with wrong IDs. For example, when it complains the resource "assets/icons-5af6a635d810e<some_long_tail>.svg" is missing, there is actually "assets/icons-1563760c6022424c<some_long_tail>.svg" under /opt/gitlab/embedded/service/gitlab-rails/public

It happens to all resources WebUI complains about: there are resources under /opt/gitlab/embedded/service/gitlab-rails/public/assets, but with different IDs …

QiaXi · February 22, 2024, 2:48am

For anyone who runs into the same issue. Gitlab documentation for troubleshooting MISSING assets might be a good starting point.

For me, it’s the Duplicate sprockets files.