GitLab CE 15.9.0 - Microsoft Graph and incoming Service Desk e-mail

Is there any way to see why an e-mail is not being picked up from a sub-addressing mailbox?

I have e-mail configured with MS Graph using an authentication token, the authentication is working and admin e-mails (such as password reset etc) are being sent.

For the incoming mail, again authentication is taking place and the mailbox is being checked; however no e-mail is being delivered.

in the logs I have the following:

{"severity":"INFO","time":"2023-03-14T14:57:52.621+00:00","context":{"email":"git@domain.com","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2023-03-14T14:57:52.635+00:00","context":{"email":"git@domain.com","name":"inbox"},"uid":"AAMkADcxNjY1ZThkLWRhZTUtNGU4Ni05Zjc1LWY2ZTAzNWY4MmNiNwBGAAAAAACkUHjwNsGJQLz9ReGIboyPBwBoYTmieCuKQKPbQimJXiVdAAAAAAEMAABoYTmieCuKQKPbQimJXiVdAAAIXLHSAAA=","action":"asking arbiter to deliver","arbitrator":"MailRoom::Arbitration::Redis"}
{"severity":"INFO","time":"2023-03-14T14:57:52.636+00:00","context":{"email":"git@domain.com","name":"inbox"},"action":"Getting new messages","unread":{"count":1,"ids":["AAMkADcxNjY1ZThkLWRhZTUtNGU4Ni05Zjc1LWY2ZTAzNWY4MmNiNwBGAAAAAACkUHjwNsGJQLz9ReGIboyPBwBoYTmieCuKQKPbQimJXiVdAAAAAAEMAABoYTmieCuKQKPbQimJXiVdAAAIXLHSAAA="]},"to_be_delivered":{"count":0,"ids":[]}}

The e-mail is being seen as unread, but not being seen as needing to be delivered.

the To: address for the waiting email is: git+sys-admin-systems-support-2-issue-@domain.com, which matches that specified within the service desk.

Is there anyone who can shed some light on what I am missing, or how I can investigate this further.

I had similar issue and I added to gitlab.rb, this two lines:

gitlab_rails['incoming_email_delivery_method'] = "sidekiq"
gitlab_rails['service_desk_email_delivery_method'] = "sidekiq"

But form version 15.9.2 onward I have problem with SSL and Microsoft Graph. Error form /var/log/gitlab/mailroom/current:

2023-03-15_13:41:51.79345 /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:66: warning: already initialized constant Net::ProtocRetryError
2023-03-15_13:41:51.79348 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/net-protocol-0.2.1/lib/net/protocol.rb:68: warning: previous definition of ProtocRetryError was here
2023-03-15_13:41:51.79350 /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:206: warning: already initialized constant Net::BufferedIO::BUFSIZE
2023-03-15_13:41:51.79351 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/net-protocol-0.2.1/lib/net/protocol.rb:214: warning: previous definition of BUFSIZE was here
2023-03-15_13:41:51.79351 /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:503: warning: already initialized constant Net::NetPrivate::Socket
2023-03-15_13:41:51.79352 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/net-protocol-0.2.1/lib/net/protocol.rb:541: warning: previous definition of Socket was here
2023-03-15_13:41:51.90425 /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired) (Faraday::SSLError)
2023-03-15_13:41:51.90429       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:44:in `ssl_socket_connect'
2023-03-15_13:41:51.90430       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1009:in `connect'
2023-03-15_13:41:51.90432       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
2023-03-15_13:41:51.90433       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:932:in `start'
2023-03-15_13:41:51.90433       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block'
2023-03-15_13:41:51.90433       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:102:in `perform_request'
2023-03-15_13:41:51.90434       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:66:in `block in call'
2023-03-15_13:41:51.90434       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/adapter.rb:45:in `connection'
2023-03-15_13:41:51.90434       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:65:in `call'
2023-03-15_13:41:51.90435       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/request/url_encoded.rb:25:in `call'
2023-03-15_13:41:51.90437       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/rack_builder.rb:153:in `build_response'
2023-03-15_13:41:51.90437       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/connection.rb:444:in `run_request'
2023-03-15_13:41:51.90437       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oauth2-1.4.11/lib/oauth2/client.rb:107:in `request'
2023-03-15_13:41:51.90438       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oauth2-1.4.11/lib/oauth2/client.rb:177:in `get_token'
2023-03-15_13:41:51.90438       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oauth2-1.4.11/lib/oauth2/strategy/client_credentials.rb:22:in `get_token'
2023-03-15_13:41:51.90438       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/microsoft_graph/connection.rb:78:in `setup'
2023-03-15_13:41:51.90439       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/microsoft_graph/connection.rb:20:in `initialize'
2023-03-15_13:41:51.90439       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:69:in `new'
2023-03-15_13:41:51.90439       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:69:in `connection'
2023-03-15_13:41:51.90441       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:31:in `run'
2023-03-15_13:41:51.90441       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/coordinator.rb:22:in `each'
2023-03-15_13:41:51.90442       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/coordinator.rb:22:in `run'
2023-03-15_13:41:51.90442       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/cli.rb:58:in `start'
2023-03-15_13:41:51.90442       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/bin/mail_room:5:in `<top (required)>'
2023-03-15_13:41:51.90443       from /opt/gitlab/embedded/bin/mail_room:23:in `load'
2023-03-15_13:41:51.90443       from /opt/gitlab/embedded/bin/mail_room:23:in `<main>'
2023-03-15_13:41:51.90443 /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired) (OpenSSL::SSL::SSLError)
2023-03-15_13:41:51.90444       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/protocol.rb:44:in `ssl_socket_connect'
2023-03-15_13:41:51.90446       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:1009:in `connect'
2023-03-15_13:41:51.90446       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
2023-03-15_13:41:51.90446       from /opt/gitlab/embedded/lib/ruby/2.7.0/net/http.rb:932:in `start'
2023-03-15_13:41:51.90447       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block'
2023-03-15_13:41:51.90447       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:102:in `perform_request'
2023-03-15_13:41:51.90447       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:66:in `block in call'
2023-03-15_13:41:51.90448       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/adapter.rb:45:in `connection'
2023-03-15_13:41:51.90448       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-net_http-3.0.2/lib/faraday/adapter/net_http.rb:65:in `call'
2023-03-15_13:41:51.90448       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/request/url_encoded.rb:25:in `call'
2023-03-15_13:41:51.90449       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/rack_builder.rb:153:in `build_response'
2023-03-15_13:41:51.90450       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.7.4/lib/faraday/connection.rb:444:in `run_request'
2023-03-15_13:41:51.90450       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oauth2-1.4.11/lib/oauth2/client.rb:107:in `request'
2023-03-15_13:41:51.90451       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oauth2-1.4.11/lib/oauth2/client.rb:177:in `get_token'
2023-03-15_13:41:51.90451       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/oauth2-1.4.11/lib/oauth2/strategy/client_credentials.rb:22:in `get_token'
2023-03-15_13:41:51.90452       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/microsoft_graph/connection.rb:78:in `setup'
2023-03-15_13:41:51.90452       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/microsoft_graph/connection.rb:20:in `initialize'
2023-03-15_13:41:51.90452       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:69:in `new'
2023-03-15_13:41:51.90453       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:69:in `connection'
2023-03-15_13:41:51.90455       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:31:in `run'
2023-03-15_13:41:51.90455       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/coordinator.rb:22:in `each'
2023-03-15_13:41:51.90455       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/coordinator.rb:22:in `run'
2023-03-15_13:41:51.90456       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/cli.rb:58:in `start'
2023-03-15_13:41:51.90456       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/bin/mail_room:5:in `<top (required)>'
2023-03-15_13:41:51.90456       from /opt/gitlab/embedded/bin/mail_room:23:in `load'
2023-03-15_13:41:51.90456       from /opt/gitlab/embedded/bin/mail_room:23:in `<main>'
2023-03-15_13:41:51.91015 Runit: waiting 5 seconds before restarting mail_room

I did an update to version 15.9.3 - issue is still there. Reconfigure incoming mail to use IMAP and than back - issue is still there.
Any ideas?

Grega

Hi Grega,

Thanks for the suggestion; that has resolved my issue; I had the lines in my gitlab.rb, but had them specified to the use default “webhook”. I updated the lines and this resolved the issue.

The log implies that your certificate has expired (but I am presuming that is not the case)?

As you are using MS Graph, has your client secret for the app in Azure expired?

If this is not how you are connecting to the server, a little more information on the method you are using would be helpful.

Andrew

I’m using microsoft_graph option on shared mailbox. Same configuration on different server - test environment, works.

If I change secret to wrong one, error is the same.
We notice error on monday, we did update on previous monday. We don’t use this function to much.
I even change the mailbox but erros is the same.

I will compare root certificates on OS.

We are using on-prem setup on oracle linux 8, omnibus setup.

Grega

YES! I found the root of a problem.
We had expired intermediate certificate in folder: /opt/gitlab/embedded/ssl/certs
Removed and then gitlab-ctl reconfigure and it works again.
Grega

1 Like