Gitlab CE email notifications not working

Summary

Recently backed up and restored gitlab to fresh aws ec2 instance. Gitlab is working fine now. All processes are up. I also see a new user which is support@mygitdomain.com user created. When I try to invite a new user via email, I do not get any email notifications. I also tried configuring AWS SMTP credentials but these dont work either. Also email alerts for commits and merge requests do not work either

Steps to reproduce

Restore backup to new gitlab instance and upgrade to new version

What is the current bug behavior?

User is added successfully on the gitlab UI. But I do not get any email notifications.

What is the expected correct behavior?

Email notification when added to a repo

Relevant logs

{"severity":"WARN","time":"2020-09-10T17:40:25.685Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","retry":3,"jid":"d85f2e0490a007d7a1a8fc54","created_at":"2020-09-10T17:35:38.658Z","meta.user":"koushik","meta.caller_id":"SessionsController#create","correlation_id":"c31bcb51a426d88c0235ab2faf07774e","enqueued_at":"2020-09-10T17:39:54.841Z","failed_at":"2020-09-10T17:36:09.623Z","retry_count":2,"retried_at":"2020-09-10T17:38:12.902Z","pid":15722,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-d85f2e0490a007d7a1a8fc54: fail: 30.842951 sec","job_status":"fail","scheduling_latency_s":0.001299,"duration_s":30.842951,"cpu_s":0.034351,"completed_at":"2020-09-10T17:40:25.685Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","db_duration_s":0.001873} {"severity":"WARN","time":"2020-09-10T17:40:25.687Z","error_class":"EOFError","error_message":"end of file reached","context":"Job raised exception","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"jid":"d85f2e0490a007d7a1a8fc54","created_at":"2020-09-10T17:35:38.658Z","meta.user":"koushik","meta.caller_id":"SessionsController#create","correlation_id":"c31bcb51a426d88c0235ab2faf07774e","enqueued_at":"2020-09-10T17:39:54.841Z","failed_at":"2020-09-10T17:36:09.623Z","retry_count":2,"retried_at":"2020-09-10T17:38:12.902Z","error_backtrace":["lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:32:inperform’",“lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:40:in perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in call’”,“lib/gitlab/sidekiq_middleware/worker_context/server.rb:13:in call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in call’”,“lib/gitlab/sidekiq_versioning/middleware.rb:9:in call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:8:in call’”,“lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in call’”,“lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in block in call’”,“lib/gitlab/with_request_store.rb:17:in enabling_request_store'","lib/gitlab/with_request_store.rb:10:in with_request_store’”,“lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in call'","lib/gitlab/sidekiq_middleware/memory_killer.rb:20:in call’”,“lib/gitlab/sidekiq_middleware/server_metrics.rb:35:in call'","lib/gitlab/sidekiq_middleware/monitor.rb:8:in block in call’”,“lib/gitlab/sidekiq_daemon/monitor.rb:49:in within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:7:in call’”,"lib/gitlab/sidekiq_logging/structured_logger.rb:18:in call'"]} {"severity":"WARN","time":"2020-09-10T17:40:46.663Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","retry":3,"jid":"9f3cccf9b10ae542f6bc7c4f","created_at":"2020-09-10T17:40:16.040Z","meta.user":"koushik","meta.project":"*******************","meta.root_namespace":"*****","meta.caller_id":"Projects::ProjectMembersController#create","correlation_id":"5ame4sTLAn","enqueued_at":"2020-09-10T17:40:16.041Z","pid":15722,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-9f3cccf9b10ae542f6bc7c4f: fail: 30.620572 sec","job_status":"fail","scheduling_latency_s":0.001113,"duration_s":30.620572,"cpu_s":0.028155,"completed_at":"2020-09-10T17:40:46.663Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","db_duration_s":0.002413} {"severity":"WARN","time":"2020-09-10T17:40:46.664Z","error_class":"EOFError","error_message":"end of file reached","context":"Job raised exception","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"jid":"9f3cccf9b10ae542f6bc7c4f","created_at":"2020-09-10T17:40:16.040Z","meta.user":"koushik","meta.project":"xyz

Details of package version

<!--
For Debian, Ubuntu based systems, paste the output of:
`dpkg-query -l "gitlab-*"`
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version                         Architecture                    Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii  gitlab-ce                                             13.3.5-ce.0                     amd64                           GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                                                                 (no description available)
### Environment details

external_url 'https://xyz.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "email-smtp.us-west-2.amazonaws.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = xxxx
gitlab_rails['smtp_password'] = xxxx
gitlab_rails['smtp_domain'] = xyz.com - > Should this be the same domain as the gitlab external url?
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
nginx['enable'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/xyz.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/xyz.com.key"
letsencrypt['enable'] = true

Looks you missed:

gitlab_rails['gitlab_email_from'] = 'test@test.com'