Gitlab CE email notifications not working

Summary

Recently backed up and restored gitlab to fresh aws ec2 instance. Gitlab is working fine now. All processes are up. I also see a new user which is support@mygitdomain.com user created. When I try to invite a new user via email, I do not get any email notifications. I also tried configuring AWS SMTP credentials but these dont work either. Also email alerts for commits and merge requests do not work either

Steps to reproduce

Restore backup to new gitlab instance and upgrade to new version

What is the current bug behavior?

User is added successfully on the gitlab UI. But I do not get any email notifications.

What is the expected correct behavior?

Email notification when added to a repo

Relevant logs

{"severity":"WARN","time":"2020-09-10T17:40:25.685Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","retry":3,"jid":"d85f2e0490a007d7a1a8fc54","created_at":"2020-09-10T17:35:38.658Z","meta.user":"koushik","meta.caller_id":"SessionsController#create","correlation_id":"c31bcb51a426d88c0235ab2faf07774e","enqueued_at":"2020-09-10T17:39:54.841Z","failed_at":"2020-09-10T17:36:09.623Z","retry_count":2,"retried_at":"2020-09-10T17:38:12.902Z","pid":15722,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-d85f2e0490a007d7a1a8fc54: fail: 30.842951 sec","job_status":"fail","scheduling_latency_s":0.001299,"duration_s":30.842951,"cpu_s":0.034351,"completed_at":"2020-09-10T17:40:25.685Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","db_duration_s":0.001873} {"severity":"WARN","time":"2020-09-10T17:40:25.687Z","error_class":"EOFError","error_message":"end of file reached","context":"Job raised exception","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"jid":"d85f2e0490a007d7a1a8fc54","created_at":"2020-09-10T17:35:38.658Z","meta.user":"koushik","meta.caller_id":"SessionsController#create","correlation_id":"c31bcb51a426d88c0235ab2faf07774e","enqueued_at":"2020-09-10T17:39:54.841Z","failed_at":"2020-09-10T17:36:09.623Z","retry_count":2,"retried_at":"2020-09-10T17:38:12.902Z","error_backtrace":["lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:32:inperform’",“lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:40:in perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in call’”,“lib/gitlab/sidekiq_middleware/worker_context/server.rb:13:in call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in call’”,“lib/gitlab/sidekiq_versioning/middleware.rb:9:in call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:8:in call’”,“lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in call’”,“lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in block in call’”,“lib/gitlab/with_request_store.rb:17:in enabling_request_store'","lib/gitlab/with_request_store.rb:10:in with_request_store’”,“lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in call'","lib/gitlab/sidekiq_middleware/memory_killer.rb:20:in call’”,“lib/gitlab/sidekiq_middleware/server_metrics.rb:35:in call'","lib/gitlab/sidekiq_middleware/monitor.rb:8:in block in call’”,“lib/gitlab/sidekiq_daemon/monitor.rb:49:in within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:7:in call’”,"lib/gitlab/sidekiq_logging/structured_logger.rb:18:in call'"]} {"severity":"WARN","time":"2020-09-10T17:40:46.663Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","retry":3,"jid":"9f3cccf9b10ae542f6bc7c4f","created_at":"2020-09-10T17:40:16.040Z","meta.user":"koushik","meta.project":"*******************","meta.root_namespace":"*****","meta.caller_id":"Projects::ProjectMembersController#create","correlation_id":"5ame4sTLAn","enqueued_at":"2020-09-10T17:40:16.041Z","pid":15722,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-9f3cccf9b10ae542f6bc7c4f: fail: 30.620572 sec","job_status":"fail","scheduling_latency_s":0.001113,"duration_s":30.620572,"cpu_s":0.028155,"completed_at":"2020-09-10T17:40:46.663Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","db_duration_s":0.002413} {"severity":"WARN","time":"2020-09-10T17:40:46.664Z","error_class":"EOFError","error_message":"end of file reached","context":"Job raised exception","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::MailDeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"jid":"9f3cccf9b10ae542f6bc7c4f","created_at":"2020-09-10T17:40:16.040Z","meta.user":"koushik","meta.project":"xyz

Details of package version

Provide the package version installation details
<!--
For Debian, Ubuntu based systems, paste the output of:
`dpkg-query -l "gitlab-*"`
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version                         Architecture                    Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii  gitlab-ce                                             13.3.5-ce.0                     amd64                           GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                                                                 (no description available)
### Environment details
  • Operating System: Ubuntu 18.04
  • Installation Target, remove incorrect values:
    • Bare Metal Machine
    • VM: AWS EC2
    • Installation Type, remove incorrect values:
    • Upgrade from version 12.9.2. Restored from backup zip onto new instance. Then updated gitlab to latest version from 12.9.2
  • Is there any other software running on the machine: NO
  • Is this a single or multiple node installation? Single Server
  • Resources
    • CPU: 2 vCPU
    • Memory total: 8GB RAM

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://xyz.com'
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "email-smtp.us-west-2.amazonaws.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = xxxx
gitlab_rails['smtp_password'] = xxxx
gitlab_rails['smtp_domain'] = xyz.com - > Should this be the same domain as the gitlab external url?
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
nginx['enable'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/xyz.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/xyz.com.key"
letsencrypt['enable'] = true