Work item errors

Problem to solve

When viewing a work item, I often get these errors (see screenshot):

image1361×766 64.7 KB

Reloading the page a few times seems to make these errors go away

Steps to reproduce

Click on a work igem. Google yielded no results.

Configuration

I am using GitLab CE, behnd a Nginx reverse proxy

Versions

Please add an x whether options apply, and add the version information.

• Self-managed
• GitLab.com SaaS
• Dedicated

Versions

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	3.3.11
Gem Version:	3.7.1
Bundler Version:4.0.10
Rake Version:	13.4.2
Redis Version:	7.2.13
Sidekiq Version:7.3.9
Go Version:	unknown

GitLab information
Version:	19.1.2
Revision:	d1f345880dd
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	17.8
URL:		https://git.eccentric.dk
HTTP Clone URL:	https://git.domain.dk/some-group/some-project.git
SSH Clone URL:	git@git.domain.dk:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.54.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	19.1.2
- default Git Version: 	2.54.0-rc1.gb15384c

Sounds like the Nginx reverse proxy is blocking or rate-limiting some routes. The errors in the screenshot indicate that the frontend cannot fetch data from the backend server (usually GraphQL). What does the browser dev tools say, any specific errors logged?

Recommend also checking the production.log and correlate the timestamp of the frontend error with log lines.

The reverse proxy configuration + gitlab.rb settings would be helpful too.

Is SAML or otherwise SSO (AD, LDAP, Kerberos, etc) configured on the reverse proxy?

I have diabled OmniAuth as I do not use it.

Below are the relevant entries of production.log:

GraphQL-Ruby encountered mismatched types in this query: `WorkItemID!` (at 8:11) vs. `ID!` (at 19:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `WorkItemID!` (at 8:11) vs. `ID!` (at 25:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `WorkItemID!` (at 8:11) vs. `WikiPageMetaID!` (at 31:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `ID!` (at 19:11) vs. `WikiPageMetaID!` (at 31:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `ID!` (at 25:11) vs. `WikiPageMetaID!` (at 31:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `String` (at 10:11) vs. `String!` (at 21:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `String!` (at 21:11) vs. `String` (at 27:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

GraphQL-Ruby encountered mismatched types in this query: `String!` (at 21:11) vs. `String` (at 33:11).
This will return an error in future GraphQL-Ruby versions, as per the GraphQL specification
Learn about migrating here: https://graphql-ruby.org/api-doc/2.6.3/GraphQL/Schema.html#allow_legacy_invalid_return_type_conflicts-class_method

and the errors I get from the bowser console:

Error sending the query 'currentUser' TypeError: NetworkError when attempting to fetch resource. vue-apollo.esm.js:726:17
    value vue-apollo.esm.js:726
    value vue-apollo.esm.js:1007
    _ module.js:137
    k module.js:176
    error module.js:229
    r iteration.js:4
    r iteration.js:4
    reportError ObservableQuery.js:406
    error ObservableQuery.js:378
    r iteration.js:4
    r iteration.js:4
    error Concast.js:36
    _ module.js:137
    k module.js:176
    error module.js:229
    c asyncMap.js:27

Error sending the query 'workItemTypes' TypeError: NetworkError when attempting to fetch resource. vue-apollo.esm.js:726:17
    value vue-apollo.esm.js:726
    value vue-apollo.esm.js:1007
    _ module.js:137
    k module.js:176
    error module.js:229
    r iteration.js:4
    r iteration.js:4
    reportError ObservableQuery.js:406
    error ObservableQuery.js:378
    r iteration.js:4
    r iteration.js:4
    error Concast.js:36
    _ module.js:137
    k module.js:176
    error module.js:229
    c asyncMap.js:27

This is the Nginx configuration for my Gitlab:

## GitLab
##
## Lines starting with two hashes (##) are comments with information.
## Lines starting with one hash (#) are configuration parameters that can be uncommented.
##
##################################
##        CONTRIBUTING          ##
##################################
##
## If you change this file in a merge request, please also create
## a merge request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
##
###################################
##         configuration         ##
###################################
##
## See installation.md#using-https for additional HTTPS configuration details.

upstream gitlab-workhorse {
  # GitLab socket file,
  # for Omnibus this would be: unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket
  server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket fail_timeout=0;
}

map $http_upgrade $connection_upgrade_gitlab {
    default upgrade;
    ''      close;
}

## NGINX 'combined' log format with filtered query strings
log_format gitlab_access '$remote_addr - $remote_user [$time_local] "$request_method $gitlab_filtered_request_uri $server_protocol" $status $body_bytes_sent "$gitlab_filtered_http_referer" "$http_user_agent"';

## Remove private_token from the request URI
# In:  /foo?private_token=unfiltered&authenticity_token=unfiltered&feed_token=unfiltered&token=unfiltered&...
# Out: /foo?private_token=[FILTERED]&authenticity_token=unfiltered&feed_token=unfiltered&token=unfiltered&...
map $request_uri $gitlab_temp_request_uri_1 {
  default $request_uri;
  ~(?i)^(?<start>.*)(?<temp>[\?&]private[\-_]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
}

## Remove authenticity_token from the request URI
# In:  /foo?private_token=[FILTERED]&authenticity_token=unfiltered&feed_token=unfiltered&token=unfiltered&...
# Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&feed_token=unfiltered&token=unfiltered&...
map $gitlab_temp_request_uri_1 $gitlab_temp_request_uri_2 {
  default $gitlab_temp_request_uri_1;
  ~(?i)^(?<start>.*)(?<temp>[\?&]authenticity[\-_]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
}

## Remove feed_token from the request URI
# In:  /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&feed_token=unfiltered&token=unfiltered&...
# Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&feed_token=[FILTERED]&token=unfiltered&...
map $gitlab_temp_request_uri_2 $gitlab_temp_request_uri_3 {
  default $gitlab_temp_request_uri_2;
  ~(?i)^(?<start>.*)(?<temp>[\?&]feed[\-_]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
}

## Remove token from the request URI
# In:  /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&feed_token=unfiltered&token=unfiltered&...
# Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&feed_token=[FILTERED]&token=[FILTERED]&...
map $gitlab_temp_request_uri_3 $gitlab_filtered_request_uri {
  default $gitlab_temp_request_uri_3;
  ~(?i)^(?<start>.*)(?<temp>[\?&]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
}

## A version of the referer without the query string
map $http_referer $gitlab_filtered_http_referer {
  default $http_referer;
  ~^(?<temp>.*)\? $temp;
}

## Normal HTTP host
server {
  ## Either remove "default_server" from the listen line below,
  ## or delete the /etc/nginx/sites-enabled/default file. This will cause gitlab
  ## to be served if you visit any address that your server responds to, eg.
  ## the ip address of the server (http://x.x.x.x/) 0.0.0.0:80 default_server; # default_server; # default_server;
  server_name git.domain.dk; ## Replace this with something like gitlab.example.com
  server_tokens off; ## Don't show the nginx version number, a security best practice

  ## See app/controllers/application_controller.rb for headers set

  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html
  real_ip_header X-Real-IP; ## X-Real-IP or X-Forwarded-For or proxy_protocol
  real_ip_recursive off;    ## If you enable 'on'
  ## If you have a trusted IP address, uncomment it and set it
  set_real_ip_from xx.xx.xx.xx; ## Replace this with something like 192.168.1.0/24

  ## Individual nginx logs for this GitLab vhost
  access_log  /var/log/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/nginx/gitlab_error.log;

  location / {
    client_max_body_size 0;
    gzip off;

    ## https://github.com/gitlabhq/gitlabhq/issues/694
    ## Some requests take more than 30 seconds.
    proxy_read_timeout      300;
    proxy_connect_timeout   300;
    proxy_redirect          off;

    proxy_http_version 1.1;

    proxy_set_header    Host                $http_host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $remote_addr;
    proxy_set_header    X-Forwarded-Proto   $scheme;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Connection          $connection_upgrade_gitlab;
    proxy_pass http://gitlab-workhorse;
  }

  error_page 404 /404.html;
  error_page 422 /422.html;
  error_page 500 /500.html;
  error_page 502 /502.html;
  error_page 503 /503.html;
  location ~ ^/(404|422|500|502|503)\.html$ {
    # Location to the GitLab's public directory,
    # for Omnibus this would be: /opt/gitlab/embedded/service/gitlab-rails/public.
    root /opt/gitlab/embedded/service/gitlab-rails/public;
    internal;
  }

    add_header          Strict-Transport-Security "max-age=31536000" always;

    listen [xxxx:xxxx:xxxx:xxxx::1]:443 ssl http2; # managed by Certbot
    listen xx.xx.xx.xx:443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/git.domain.dk/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/git.domain.dk/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = git.domain.dk) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  listen xx.xx.xx.xx:80 http2;
  listen [xxxx:xxxx:xxxx:xxxx::1]:80 http2;
  server_name git.domain.dk;
    return 404; # managed by Certbot


}

and below is the redacted output from gitlab-ctl dff-config:

diff --git a/etc/gitlab/gitlab.rb b/opt/gitlab/etc/gitlab.rb.template
index 7e9566b..2b0b48c 100644
--- a/etc/gitlab/gitlab.rb
+++ b/opt/gitlab/etc/gitlab.rb.template
@@ -29,7 +29,7 @@
 ##! On AWS EC2 instances, we also attempt to fetch the public hostname/IP
 ##! address from AWS. For more details, see:
 ##! https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
-external_url 'https://git.domain.dk'
+external_url 'GENERATED_EXTERNAL_URL'
 
 ## Roles for multi-instance GitLab
 ##! The default is to have no roles enabled, which results in GitLab running as an all-in-one instance.
@@ -96,15 +96,15 @@ external_url 'https://git.domain.dk'
 ###! Docs: https://docs.gitlab.com/omnibus/settings/smtp.html
 ###! **Use smtp instead of sendmail/postfix.**
 
-gitlab_rails['smtp_enable'] = true
-gitlab_rails['smtp_address'] = "mail.domain.dk"
-gitlab_rails['smtp_port'] = 465
-gitlab_rails['smtp_user_name'] = "user@domain.dk"
-gitlab_rails['smtp_password'] = "xxx"
-gitlab_rails['smtp_domain'] = "domain.dk"
-gitlab_rails['smtp_authentication'] = "login"
-gitlab_rails['smtp_enable_starttls_auto'] = false
-gitlab_rails['smtp_tls'] = true
+# gitlab_rails['smtp_enable'] = true
+# gitlab_rails['smtp_address'] = "smtp.server"
+# gitlab_rails['smtp_port'] = 465
+# gitlab_rails['smtp_user_name'] = "smtp user"
+# gitlab_rails['smtp_password'] = "smtp password"
+# gitlab_rails['smtp_domain'] = "example.com"
+# gitlab_rails['smtp_authentication'] = "login"
+# gitlab_rails['smtp_enable_starttls_auto'] = true
+# gitlab_rails['smtp_tls'] = false
 # gitlab_rails['smtp_pool'] = false
 
 ###! **Can be: 'none', 'peer', 'client_once', 'fail_if_no_peer_cert'**
@@ -120,8 +120,8 @@ gitlab_rails['smtp_tls'] = true
 
 ###! If your SMTP server does not like the default 'From: gitlab@gitlab.example.com'
 ###! can change the 'From' with this setting.
-gitlab_rails['gitlab_email_from'] = 'gitlab@domain.dk'
-gitlab_rails['gitlab_email_display_name'] = 'domain GitLab'
+# gitlab_rails['gitlab_email_from'] = 'example@example.com'
+# gitlab_rails['gitlab_email_display_name'] = 'Example'
 # gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com'
 # gitlab_rails['gitlab_email_subject_suffix'] = ''
 # gitlab_rails['gitlab_email_smime_enabled'] = false
@@ -633,7 +633,7 @@ gitlab_rails['gitlab_email_display_name'] = 'domain GitLab'
 
 ### OmniAuth Settings
 ###! Docs: https://docs.gitlab.com/ee/integration/omniauth.html
-gitlab_rails['omniauth_enabled'] = false
+# gitlab_rails['omniauth_enabled'] = nil
 # gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
 # gitlab_rails['omniauth_sync_email_from_provider'] = 'saml'
 # gitlab_rails['omniauth_sync_profile_from_provider'] = ['saml']
@@ -708,7 +708,6 @@ gitlab_rails['omniauth_enabled'] = false
 
 ###! The duration in seconds to keep backups before they are allowed to be deleted
 # gitlab_rails['backup_keep_time'] = 604800
-gitlab_rails['backup_keep_time'] = 3600
 
 # gitlab_rails['backup_upload_connection'] = {
 #   'provider' => 'AWS',
@@ -1104,12 +1103,12 @@ gitlab_rails['backup_keep_time'] = 3600
 ##! Docs: https://docs.gitlab.com/ee/administration/packages/container_registry.html
 ################################################################################
 
-registry_external_url 'https://registry.domain.dk'
+# registry_external_url 'https://registry.example.com'
 
 ### Settings used by GitLab application
-gitlab_rails['registry_enabled'] = true
-gitlab_rails['registry_host'] = "registry.domain.dk"
-gitlab_rails['registry_port'] = "5000"
+# gitlab_rails['registry_enabled'] = true
+# gitlab_rails['registry_host'] = "registry.gitlab.example.com"
+# gitlab_rails['registry_port'] = "5005"
 # gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
 
 ###! Notification secret, it's used to authenticate notification requests to GitLab application
@@ -1298,7 +1297,7 @@ gitlab_rails['registry_port'] = "5000"
 ##! wait up to that amount of time and report a timeout error if the
 ##! process has not shut down.
 # gitlab_workhorse['shutdown_timeout'] = nil
-# gitlab_workhorse['listen_network'] = "socket"
+# gitlab_workhorse['listen_network'] = "unix"
 # gitlab_workhorse['listen_umask'] = 000
 # gitlab_workhorse['listen_addr'] = "/var/opt/gitlab/gitlab-workhorse/sockets/socket"
 # gitlab_workhorse['auth_backend'] = "http://localhost:8080"
@@ -1440,7 +1439,7 @@ gitlab_rails['registry_port'] = "5000"
 ##!     for other processes, with ~1 GB per Puma worker).
 ##! The result is clamped to a minimum of 2 (required for the web editor).
 ##! Set this explicitly if auto-detection produces incorrect results.
-puma['worker_processes'] = 0
+# puma['worker_processes'] = 2
 # puma['min_threads'] = 4
 # puma['max_threads'] = 4
 
@@ -1501,7 +1500,7 @@ puma['worker_processes'] = 0
 # sidekiq['log_format'] = "json"
 # sidekiq['shutdown_timeout'] = 4
 # sidekiq['interval'] = nil
-sidekiq['concurrency'] = 10
+# sidekiq['concurrency'] = 20
 
 ##! GitLab allows route a job to a particular queue determined by an array of ##! routing rules.
 ##! Each routing rule is a tuple of queue selector query and corresponding queue. By default,
@@ -1909,7 +1908,7 @@ sidekiq['concurrency'] = 10
 
 ##! When bundled nginx is disabled we need to add the external webserver user to
 ##! the GitLab webserver group.
-web_server['external_users'] = ['nginx']
+# web_server['external_users'] = []
 # web_server['username'] = 'gitlab-www'
 # web_server['group'] = 'gitlab-www'
 # web_server['uid'] = nil
@@ -1922,7 +1921,7 @@ web_server['external_users'] = ['nginx']
 ##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html
 ################################################################################
 
-nginx['enable'] = false
+# nginx['enable'] = true
 # nginx['client_max_body_size'] = '0'
 # nginx['redirect_http_to_https'] = false
 # nginx['redirect_http_to_https_port'] = 80
@@ -3684,24 +3683,3 @@ nginx['enable'] = false
 # oak['components']['openbao']['ssl_certificate_key'] = '/etc/gitlab/ssl/openbao.example.com.key'
 ## Redirect HTTP to HTTPS (enabled automatically when using Let's Encrypt):
 # oak['components']['openbao']['redirect_http_to_https'] = true
-
-
-## CUSTOM
-alertmanager['enable'] = false
-gitlab_exporter['enable'] = false
-gitlab_kas['enable'] = false
-node_exporter['enable'] = false
-postgres_exporter['enable'] = false
-prometheus_monitoring['enable'] = false
-prometheus['enable'] = false
-puma['exporter_enabled'] = false
-redis_exporter['enable'] = false
-sidekiq['metrics_enabled'] = false
-
-gitlab_rails['env'] = {
-  'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
-}
-
-gitaly['env'] = {
-  'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
-}

Thanks. I asked Claude to analyze the artifacts.

The GraphQL-Ruby lines in your production.log (WorkItemID! vs ID! etc.) are just deprecation warnings about legacy type coercion in the schema. Ignore those, they’re not your bug.

The real signal is in the browser console: TypeError: NetworkError when attempting to fetch resource for the currentUser and workItemTypes queries. That message means the fetch never got a response at all, the connection got dropped or reset before GitLab could answer. That’s a proxy/backend issue, not a GraphQL logic issue.

Looking at your gitlab-ctl diff-config output, one line stands out:

puma['worker_processes'] = 0

A value of 0 forces Puma into single process mode. The work item page fires several GraphQL queries in parallel (you can see currentUser and workItemTypes both failing at once). With only one worker to serve them, some requests get queued or reset while waiting on the others, which lines up with your “reload a few times and it goes away” symptom.

Worth checking:

1. Set puma['worker_processes'] to something explicit, 2 is a safe minimum, or match it to your CPU core count.
2. Run gitlab-ctl reconfigure and restart.
3. Reproduce and watch whether the NetworkErrors are gone.

Also noticed sidekiq['concurrency'] = 10 instead of the default 20. Not your immediate problem, but worth a second look if that was tuned down intentionally or just carried over.

Hi,

Increasing the puma worker processes to 4 (the number of cores) did not solve the probkem. The reasons for the modificaions you mention is to conserve RAM as described at Running GitLab in a memory-constrained environment | GitLab Docs

Which resources are available to GitLab on this host? (CPU, memory, disk, etc.) If they are low, that might explain the data response latencies.

The server has 4 cores, 8GB RAM and 260GB storage. Currently gitlab-runsvdirconsumes 4.3GB RAM and there is about 2GB free, there is > 6GB swap available. Of the storange about 63GB is used.

I’ve ran on a 4vcpu and 8gb ram without issues, but then I don’t have too many users. You may wish to compare your config with mine. Note that I have the puma worker processes at 2. When commented out, it used 4 by auto-detecting.

gitlab_kas['enable'] = false
gitlab_rails['env'] = {
  'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
}
gitaly['env'] = {
  'MALLOC_CONF' => 'dirty_decay_ms:1000,muzzy_decay_ms:1000'
}
puma['worker_processes'] = 2
puma['per_worker_max_memory_mb'] = 1024
sidekiq['concurrency'] = 10
prometheus['enable'] = false
prometheus_monitoring['enable'] = false

I disabled a lot of the services I don’t need, so you may want to check/verify if you need them or not. The puma worker processes at zero never worked for me. The above are the only tuning options I did, I know your config above has a lot more done to it, so you may wish to simplify it a bit like mine and see if it gets any better.

Alas, that did not help, but thank you for the suggestion!

We recently had a longer thread about performance issues, both CE and EE related. Maybe some tips on troubleshooting, monitoring, correlating the errors to time series spikes and logs, etc. will be helpful. Gitlab Upgrade from CE to EE results in instance hickups (502 errors)/ Gitlab Premium upgrade evaluation

I seen to have solved tge issue by removing http2 from the listen directive in my nginx configuration.