All incoming mail stopped working after update to v15.3.1 last night

Everything worked perfectly fine until I received the v15.3.1 update last night. Since then service desk and ticket creation via e-mail services are not working.

I checked the mailroom.log and it says:

"action":"Disconnected. Resetting...","error":"Error in IMAP command received by server."

docker exec -it gitlab gitlab-rake gitlab:incoming_email:check’s output:

Checking Incoming Email ...

Incoming Email: ... Checking Reply by email ...

IMAP server credentials are correct? ... Checking email1@example.com
Checking email2@example.com
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped

Checking Reply by email ... Finished
Checking Incoming Email ... Finished

The configuration has not changed.

I have checked the mail server logs but I did not find anything helpful. Could you help me please how can I make mailing work again?

I’ve the same Issue. Mails leave in the inbox with seen flag set. I can see an exception in mailroom’s log. I think, the exception is thrown, because gitlab expects a .gitlab_incoming_email_secret in the work dir.

I’m apt dist-upgrade my vm since gitlab 8 or 9 I guess. So there is propably something missing or deprecated in /etc/gitlab/gitlab.rb

Exception

2022-08-23_15:42:42.81953 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `read': No such file or directory @ rb_sysopen - .gitlab_incoming_email_secret (Errno::ENOENT)
2022-08-23_15:42:42.81953       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `token'
2022-08-23_15:42:42.81953       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:100:in `config_request_jwt_auth'
2022-08-23_15:42:42.81954       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:82:in `block in deliver'
2022-08-23_15:42:42.81954       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:442:in `block in run_request'
2022-08-23_15:42:42.81955       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:459:in `block in build_request'
2022-08-23_15:42:42.81956       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:40:in `block in create'
2022-08-23_15:42:42.81956       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `tap'
2022-08-23_15:42:42.81957       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `create'
2022-08-23_15:42:42.81957       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:455:in `build_request'
2022-08-23_15:42:42.81958       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:437:in `run_request'
2022-08-23_15:42:42.81958       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:281:in `post'
2022-08-23_15:42:42.81958       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:78:in `deliver'
2022-08-23_15:42:42.81959       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox.rb:118:in `deliver'
2022-08-23_15:42:42.81959       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:32:in `block in run'
2022-08-23_15:42:42.81960       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `map'
2022-08-23_15:42:42.81961       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `process_mailbox'
2022-08-23_15:42:42.81961       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:49:in `wait'
2022-08-23_15:42:42.81962       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
2022-08-23_15:42:42.81973 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `read': No such file or directory @ rb_sysopen - .gitlab_incoming_email_secret (Errno::ENOENT)
2022-08-23_15:42:42.81975       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `token'
2022-08-23_15:42:42.81975       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:100:in `config_request_jwt_auth'
2022-08-23_15:42:42.81976       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:82:in `block in deliver'
2022-08-23_15:42:42.81977       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:442:in `block in run_request'
2022-08-23_15:42:42.81977       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:459:in `block in build_request'
2022-08-23_15:42:42.81977       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:40:in `block in create'
2022-08-23_15:42:42.81978       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `tap'
2022-08-23_15:42:42.81979       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `create'
2022-08-23_15:42:42.81979       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:455:in `build_request'
2022-08-23_15:42:42.81980       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:437:in `run_request'
2022-08-23_15:42:42.81980       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:281:in `post'
2022-08-23_15:42:42.81981       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:78:in `deliver'
2022-08-23_15:42:42.81981       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox.rb:118:in `deliver'
2022-08-23_15:42:42.81981       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:32:in `block in run'
2022-08-23_15:42:42.81982       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `map'
2022-08-23_15:42:42.81982       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `process_mailbox'
2022-08-23_15:42:42.81983       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:49:in `wait'
2022-08-23_15:42:42.81984       from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
2022-08-23_15:42:42.82013 #<struct MailRoom::Delivery::Postback::Options url="https://gitlab.cross-solution.de/api/v4/internal/mail_room/incoming_email", token=nil, username=nil, password=nil, logger=#<MailRoom::Logger::Structured:0x00007f37646b51a0 @level=0, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f37646b5178 @datetime_format=nil>, @formatter=nil, @logdev=#<Logger::LogDevice:0x00007f37646b5128 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename="/var/log/gitlab/mailroom/mail_room_json.log", @dev=#<File:/var/log/gitlab/mailroom/mail_room_json.log>, @binmode=false, @mon_data=#<Monitor:0x00007f37646b50d8>, @mon_data_owner_object_id=1460>>, content_type="text/plain", jwt=#<MailRoom::JWT:0x00007f37643d9670 @header="Gitlab-Mailroom-Api-Request", @secret_path=".gitlab_incoming_email_secret", @issuer="gitlab-mailroom", @algorithm="HS256">>

I have just found the same error message too. I do not know what to do with it yet. I downgraded to v15.2.2 but no luck yet.

I try to investigate network errors but the only thing that changed was the version yesterday so I hoped downgrade will help.

Please let me know if you find something

Thanks for your reply!

I tried to connect to imap servers from the gitlab container via telnet (for example telnet imap.gmail.com 993) it does not seem to work I can not get a prompt. If I create a new ubuntu:22:04 container on the same docker network network telnet imap.gmail.com 993 works fine.

It explains the Error in IMAP command received by server. error message but I do not know yet how to troubleshoot this.

I can connect to port 587 and 25 from the gitlab container but 993 is not working.
telnet smtp.gmail.com 587
telnet smtp.gmail.com 25

If I use nmap -p993 imap.myhost.com it shows the port is open.

Tcpdump on the host shows sent requests and received responses between the two hosts.

I tought it was some ssl issue but I can connect with openssl s_client -connect example.myhost.com:993 and get the cert and it looks all right.

I dont think its a network problem. It is this .gitlab_incoming_email_secret file that gitlab expects. I don’t know what needs to be in there though. If you just touch /var/opt/gitlab/gitlab-rails/working/.gitlab_incoming_email_secret to create the file, then the exception does not happen. But the mails are just retrieved and deleted from the inbox.

I could help myself by configuring a forwarding for incoming mails. I wait until someone knows. what to do.

/var/opt/gitlab/gitlab-rails/working/ is the working dir in omnibus. At least on my instance. Check it on your side,

I have just noticed after the downgrade gitlab started to process new incoming messages but it does not care about the ones arrived before the downgrade.

As I can see now the problem is in v15.3.1 and downgrade to v15.2.2 solves the problem but the messages received during the v15.3.1 won’t be processed.
Is there anything I can help the community to fix this issue?

Maybe this will be helpful:

The workaround I did:

1. Stop the Gitlab container
2. Make a backup from the whole directory
3. Tag the previous image with docker tag xxxxxxxxxxxx gitlab/gitlab-ce:last-working
4. Replace the image in docker-compose.yml to image: gitlab/gitlab-ce:last-working
5. Start the Gitlab container

Thanks for sharing all the details. This sounds like a bug in 15.3, upgrading from 15.2. Can you please open an issue? Thanks!

Sure! Thank you for your reply!

@cbleek and all.

I opened an issue and got help.

If you edit your gitlab.rb file and add these two lines

gitlab_rails['incoming_email_delivery_method'] = "sidekiq"
gitlab_rails['service_desk_email_delivery_method'] = "sidekiq"

It will work more than likely.

@dnsmichi thank you for your help.

In addition to reverting the delivery method to sidekiq, you probably need to (manually) mark the mails as unread. On our IMAP account, the mails were still present in the INBOX but marked as read; as soon as we reset the flag, they were imported within some minutes.

I marked them as unread and now all my mails in place Thank you for your help!

worked for me, too. Thank you.

Added incoming_email_delivery_method and service_desk_email_delivery_method to /etc/gitlab/gitlab.rb and everything is fine again.

gitlab_rails['incoming_email_delivery_method'] = "sidekiq"
gitlab_rails['service_desk_email_delivery_method'] = "sidekiq"

+1 to the solution, EE Premium 15.3.2 Omnibus, using MS Graph API Incoming mail. It took a while to process backlogged messages but it worked great!

+99 to this solution on CE

I know this post is a bit old, but I’ll give it a try.

I tried the "sidekiq" fix with no success - does the use of sidekiq require any other settings?

Hi prathers,

I have the same problem - and for me the fix doesn’t work too.
Did you fix your problem?

Best regards
Stefan

The problem persist unfortunately. I have no other ideas

For me it is working now. It was no problem with incoming mail. Our IT changed someting in the office 365 settings. I found it when I switch from SMTP Auth to OAuth for outgoing emails.

When we solved the problem with outgoing mails (easier to debug) it solved our problem with incoming mails automatically.

No, unfortunately this was a change made by Microsoft that started coming into effect slowly since October. It hit us in November. Using SMTP Auth with a normal username/password or even a generated token if 2FA was being used on the email account is no longer possible. Which means moving to OAUTH/GraphAPI to get email delivery to work. I believe Google is also doing the same thing if they haven’t already started.