Notifications Stoped to works after upgrade

:hugs: Please review whether this category fits your question about GitLab community topics. If not, please change the category dropdown to switch the guided template automatically.

For support questions about GitLab product usage and features, please check these categories instead:

  1. How to Use GitLab
    Local instance
  2. GitLab CI/CD
    No problem with this
  3. GitLab Duo (AI)
    We are facing issues after the migration to the new version
  4. DevSecOps
  5. Infrastructure as Code & Cloud Native
  6. Observability

Dear Comunity, since I executed the migraton of my gitlab instance to 16.10 and 16.11

I’m facing issue to send notifications for all service provided by gitlab.

I don’t know how to start to investigate it but I already sent some tests via console and I’m receiving it normally:

This is the second one that I sent via console:

The user confirms that received normally.

But when I send it by interface closing or opening some issue for example, it is not working.

Maybe your question was answered already, or you are hitting a bug?

I already resolved the issue:

{"severity":"WARN","time":"2024-05-10T09:18:33.521Z","retry":3,"queue":"mailers","store":null,"class":"ActionMailer::MailDeliveryJob","args":["[FILTERED]"],"jid":"a1363464436eece4abcf5f6f","created_at":"2024-05-10T09:17:03.649Z","correlation_id":"01HXGWNAYQJFV6N293MMWYPJ4V","meta.caller_id":"MailScheduler::NotificationServiceWorker","meta.remote_ip":"77.20.183.9","meta.feature_category":"team_planning","meta.user":"paolodasilva","meta.user_id":563,"meta.client_id":"user/563","meta.root_caller_id":"graphql:updateIssue","worker_data_consistency":"delayed","wal_locations":{},"wal_location_source":"primary","idempotency_key":"resque:gitlab:duplicate:mailers:5cdec3af298d04d32f411cb7907037d4d93d4cffb6595510deb1c56051b467a4","size_limiter":"validated","enqueued_at":"2024-05-10T09:18:33.452Z","retry_count":2,"retried_at":"2024-05-10T09:17:45.952Z","job_size_bytes":467,"pid":1180144,"message":"ActionMailer::MailDeliveryJob JID-a1363464436eece4abcf5f6f: fail: 0.067087 sec","job_status":"fail","scheduling_latency_s":0.001248,"redis_calls":4,"redis_duration_s":0.000691,"redis_read_bytes":2,"redis_write_bytes":307,"redis_cache_calls":2,"redis_cache_duration_s":0.000417,"redis_cache_write_bytes":84,"redis_queues_metadata_calls":1,"redis_queues_metadata_duration_s":0.000143,"redis_queues_metadata_read_bytes":1,"redis_queues_metadata_write_bytes":123,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000131,"redis_shared_state_read_bytes":1,"redis_shared_state_write_bytes":100,"db_count":23,"db_write_count":1,"db_cached_count":5,"db_txn_count":0,"db_replica_txn_count":0,"db_primary_txn_count":0,"db_main_txn_count":0,"db_ci_txn_count":0,"db_main_replica_txn_count":0,"db_ci_replica_txn_count":0,"db_replica_count":0,"db_primary_count":23,"db_main_count":23,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":5,"db_main_cached_count":5,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_txn_max_duration_s":0.0,"db_primary_txn_max_duration_s":0.0,"db_main_txn_max_duration_s":0.0,"db_ci_txn_max_duration_s":0.0,"db_main_replica_txn_max_duration_s":0.0,"db_ci_replica_txn_max_duration_s":0.0,"db_replica_txn_duration_s":0.0,"db_primary_txn_duration_s":0.0,"db_main_txn_duration_s":0.0,"db_ci_txn_duration_s":0.0,"db_main_replica_txn_duration_s":0.0,"db_ci_replica_txn_duration_s":0.0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.011,"db_main_duration_s":0.011,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.05193,"mem_objects":20816,"mem_bytes":1614888,"mem_mallocs":5408,"mem_total_bytes":2447528,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.067087,"completed_at":"2024-05-10T09:18:33.520Z","load_balancing_strategy":"primary_no_wal","exception.class":"Premailer::Rails::CSSHelper::FileNotFound","exception.message":"File with URL \"https://gitlab.ebrains.eu/assets/notify_enhanced-5ae4450e4ed24b373f6b44dd773f778232a806b86d0c7f58fb94bfb81750c79a.css\" could not be loaded by any strategy.","exception.backtrace":["premailer-rails (1.10.3) lib/premailer/rails/css_helper.rb:51:in load_css’","premailer-rails (1.10.3) lib/premailer/rails/css_helper.rb:38:in`

After the upgrade my css URLs was messed and I had to move the URL manually and this https://gitlab.ebrains.eu/assets/notify_enhanced-5ae4450e4ed24b373f6b44dd773f778232a806b86d0c7f58fb94bfb81750c79a.css was affected.

I just moved this URL:

-rw-r–r-- 1 root root 5.5K May 7 19:30 notify_enhanced-ca8035fd4f17f43ca4989ad164737b68c3b685fc5735d77bf7529b1cf6641445.css

To:

-rw-r–r-- 1 root root 25K May 7 19:30 notify_enhanced-5ae4450e4ed24b373f6b44dd773f778232a806b86d0c7f58fb94bfb81750c79a.css

root@gitlab:/opt/gitlab/embedded/service/gitlab-rails/public/assets# mv notify_enhanced-ca8035fd4f17f43ca4989ad164737b68c3b685fc5735d77bf7529b1cf6641445.css notify_enhanced-5ae4450e4ed24b373f6b44dd773f778232a806b86d0c7f58fb94bfb81750c79a.css

and it works

So one Note:

I don’t know how it happens but please we have to ensuring that the upgrade will not mess the CSS files.