Everything worked perfectly fine until I received the v15.3.1 update last night. Since then service desk and ticket creation via e-mail services are not working.
I checked the mailroom.log and it says:
"action":"Disconnected. Resetting...","error":"Error in IMAP command received by server."
I’ve the same Issue. Mails leave in the inbox with seen flag set. I can see an exception in mailroom’s log. I think, the exception is thrown, because gitlab expects a .gitlab_incoming_email_secret in the work dir.
I’m apt dist-upgrade my vm since gitlab 8 or 9 I guess. So there is propably something missing or deprecated in /etc/gitlab/gitlab.rb
Exception
2022-08-23_15:42:42.81953 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `read': No such file or directory @ rb_sysopen - .gitlab_incoming_email_secret (Errno::ENOENT)
2022-08-23_15:42:42.81953 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `token'
2022-08-23_15:42:42.81953 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:100:in `config_request_jwt_auth'
2022-08-23_15:42:42.81954 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:82:in `block in deliver'
2022-08-23_15:42:42.81954 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:442:in `block in run_request'
2022-08-23_15:42:42.81955 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:459:in `block in build_request'
2022-08-23_15:42:42.81956 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:40:in `block in create'
2022-08-23_15:42:42.81956 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `tap'
2022-08-23_15:42:42.81957 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `create'
2022-08-23_15:42:42.81957 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:455:in `build_request'
2022-08-23_15:42:42.81958 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:437:in `run_request'
2022-08-23_15:42:42.81958 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:281:in `post'
2022-08-23_15:42:42.81958 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:78:in `deliver'
2022-08-23_15:42:42.81959 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox.rb:118:in `deliver'
2022-08-23_15:42:42.81959 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:32:in `block in run'
2022-08-23_15:42:42.81960 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `map'
2022-08-23_15:42:42.81961 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `process_mailbox'
2022-08-23_15:42:42.81961 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:49:in `wait'
2022-08-23_15:42:42.81962 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
2022-08-23_15:42:42.81973 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `read': No such file or directory @ rb_sysopen - .gitlab_incoming_email_secret (Errno::ENOENT)
2022-08-23_15:42:42.81975 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `token'
2022-08-23_15:42:42.81975 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:100:in `config_request_jwt_auth'
2022-08-23_15:42:42.81976 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:82:in `block in deliver'
2022-08-23_15:42:42.81977 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:442:in `block in run_request'
2022-08-23_15:42:42.81977 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:459:in `block in build_request'
2022-08-23_15:42:42.81977 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:40:in `block in create'
2022-08-23_15:42:42.81978 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `tap'
2022-08-23_15:42:42.81979 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `create'
2022-08-23_15:42:42.81979 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:455:in `build_request'
2022-08-23_15:42:42.81980 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:437:in `run_request'
2022-08-23_15:42:42.81980 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:281:in `post'
2022-08-23_15:42:42.81981 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:78:in `deliver'
2022-08-23_15:42:42.81981 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox.rb:118:in `deliver'
2022-08-23_15:42:42.81981 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:32:in `block in run'
2022-08-23_15:42:42.81982 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `map'
2022-08-23_15:42:42.81982 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `process_mailbox'
2022-08-23_15:42:42.81983 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:49:in `wait'
2022-08-23_15:42:42.81984 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
2022-08-23_15:42:42.82013 #<struct MailRoom::Delivery::Postback::Options url="https://gitlab.cross-solution.de/api/v4/internal/mail_room/incoming_email", token=nil, username=nil, password=nil, logger=#<MailRoom::Logger::Structured:0x00007f37646b51a0 @level=0, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f37646b5178 @datetime_format=nil>, @formatter=nil, @logdev=#<Logger::LogDevice:0x00007f37646b5128 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename="/var/log/gitlab/mailroom/mail_room_json.log", @dev=#<File:/var/log/gitlab/mailroom/mail_room_json.log>, @binmode=false, @mon_data=#<Monitor:0x00007f37646b50d8>, @mon_data_owner_object_id=1460>>, content_type="text/plain", jwt=#<MailRoom::JWT:0x00007f37643d9670 @header="Gitlab-Mailroom-Api-Request", @secret_path=".gitlab_incoming_email_secret", @issuer="gitlab-mailroom", @algorithm="HS256">>
I tried to connect to imap servers from the gitlab container via telnet (for example telnet imap.gmail.com 993) it does not seem to work I can not get a prompt. If I create a new ubuntu:22:04 container on the same docker network network telnet imap.gmail.com 993 works fine.
It explains the Error in IMAP command received by server. error message but I do not know yet how to troubleshoot this.
I can connect to port 587 and 25 from the gitlab container but 993 is not working. telnet smtp.gmail.com 587 telnet smtp.gmail.com 25
If I use nmap -p993 imap.myhost.com it shows the port is open.
Tcpdump on the host shows sent requests and received responses between the two hosts.
I tought it was some ssl issue but I can connect with openssl s_client -connect example.myhost.com:993 and get the cert and it looks all right.
I dont think its a network problem. It is this .gitlab_incoming_email_secret file that gitlab expects. I don’t know what needs to be in there though. If you just touch /var/opt/gitlab/gitlab-rails/working/.gitlab_incoming_email_secret to create the file, then the exception does not happen. But the mails are just retrieved and deleted from the inbox.
I could help myself by configuring a forwarding for incoming mails. I wait until someone knows. what to do.
/var/opt/gitlab/gitlab-rails/working/ is the working dir in omnibus. At least on my instance. Check it on your side,
I have just noticed after the downgrade gitlab started to process new incoming messages but it does not care about the ones arrived before the downgrade.
As I can see now the problem is in v15.3.1 and downgrade to v15.2.2 solves the problem but the messages received during the v15.3.1 won’t be processed.
Is there anything I can help the community to fix this issue?
In addition to reverting the delivery method to sidekiq, you probably need to (manually) mark the mails as unread. On our IMAP account, the mails were still present in the INBOX but marked as read; as soon as we reset the flag, they were imported within some minutes.
For me it is working now. It was no problem with incoming mail. Our IT changed someting in the office 365 settings. I found it when I switch from SMTP Auth to OAuth for outgoing emails.
When we solved the problem with outgoing mails (easier to debug) it solved our problem with incoming mails automatically.
No, unfortunately this was a change made by Microsoft that started coming into effect slowly since October. It hit us in November. Using SMTP Auth with a normal username/password or even a generated token if 2FA was being used on the email account is no longer possible. Which means moving to OAUTH/GraphAPI to get email delivery to work. I believe Google is also doing the same thing if they haven’t already started.