IMAP Idle Timeout

[PeteSL]

Is your feature request related to a problem? Please describe.
Some IMAP servers (Exchange Online/Office 365/Microsoft 365 definitely) quit pushing notifications after a period of time. The RFC for IMAP Idle states "clients using IDLE are advised to terminate IDLE and reissue it at least every 29 minutes to avoid being logged off." It has been recommended elsewhere that this should be adjustable down to every 10 minutes if so desired (and O365 seems like it needs it that rapidly).

Describe the solution you'd like
Currently, handleIdle ignores a socket receive timeout and simply goes back to listening. Instead, it should abort and reissue the IDLE command. If there are errors, they can be handled by the application.

Describe alternatives you've considered
Add mail.imap.idletimeout which will set an underlying timer to interrupt the readResponse to get it to reissue the IDLE command.

Additional context
Preference is to simply reissue the IDLE command (after aborting it) upon a socket receive timeout since that does not involve any extra threads.

[PeteSL]

In IMAPFolder.handleIdle, if a socket timeout exception do an idleAbort, a readResponse, a startIdle, and, if the previous 3 succeded, set r = null to continue waiting for an IDLE response. This will work regardless of whether idle() is called directly or IdleManager is in use since this is done as a "maintenance" procedure and will not signal any event monitoring.

[jbescos]

Hi @PeteSL , thanks for your deep description. Please, go ahead and create a PR if you know the steps. We will need to test this too, preferable with a unit test, or otherwise with a small reproducer.

[PeteSL]

I apologize, I don't know the steps to create a PR. In handleIdle, you can see where SocketTimeoutException is handled with a log entry basically saying the timeout is ignored. That is where I -think- you can do the idleAbort(), readResponse(), startIdle() commands before setting r = null; I would be remiss in saying I know what exceptions to catch and what to do once caught without considerable further investigation. I think it can be done just by analyzing the Response (step 2) and that startIdle() returns true but that is just going from my memory this morning.

[jbescos]

I am investigating this. The code already handles the abort of the idle in the case it is not a SocketTimeoutException here:

angus-mail/providers/imap/src/main/java/org/eclipse/angus/mail/imap/IMAPFolder.java

Line 3278 in c399fb4

} else {

Basically your suggestion would be to remove the if section in here:

                        /*
                         * If it was a timeout and no bytes were transferred
                         * we ignore it and go back and read again.
                         * If the I/O was otherwise interrupted, and no
                         * bytes were transferred, we take it as a request
                         * to abort the IDLE.
                         */
                        Exception ex = r.getException();
                        if (ex instanceof InterruptedIOException &&
                                ((InterruptedIOException) ex).
                                        bytesTransferred == 0) {
                            if (ex instanceof SocketTimeoutException) {
                                logger.finest(
                                        "handleIdle: ignoring socket timeout");
                                r = null;    // repeat do/while loop
                            } else {
                                logger.finest("handleIdle: interrupting IDLE");
                                IdleManager im = idleManager;
                                if (im != null) {
                                    logger.finest(
                                            "handleIdle: request IdleManager to abort");
                                    im.requestAbort(this);
                                } else {
                                    logger.finest("handleIdle: abort IDLE");
                                    protocol.idleAbort();
                                    idleState = ABORTING;
                                }
                                // normally will exit the do/while loop
                            }

From the comment above, it seems this behavior is intentional. It was made in this commit.

It looks the current behavior is:

1. SocketTimeoutException -> Nothing happens related to idle.
2. Next iteration in loop throws ConnectionException and it falls down into this section:

            } catch (ConnectionException cex) {
                // Oops, the folder died on us.
                throw new FolderClosedException(this, cex.getMessage());

[PeteSL]

No, the assumption is the connection has been closed and it hasn't been. The server can quit servicing the IDLE without closing the connection and without providing any response (per RFC and this is what happens with Exchange Online). In fact, this code assumes the folder is still open and we just haven't heard from the server (per RFC). In essence, this is a "keep-alive" sequence recommended by the RFC to be done every 29 minutes or less of inactivity. The sequence is (after a successful IDLE command to start with) issue a DONE command and then reissue the IDLE command after the server responds. This all depends on the connection and folder still being open. This does, however, allow for the connection or folder to have been closed which will cause handleIdle() to throw an exception based on the ConnectionException thrown here.

What I am recomending is replacing logger.finest("handleIdle: ignoring socket timeout"); with:

idleAbortWait(); // This issues the DONE command and clears responses
if (!startIdle(im)){ // This restarts the idle command.
    close(false, true);
    throw new ConnectionException(this, "Folder closed during Idle.");
}

r will still be set to null causing the loop to continue without affecting either IdleManager or a manual idle() call. If there are any connection errors or folder errors, the folder is closed and a ConnectionException is thrown so that error handling occurs. The startIdle handling is a guess but I think those 4 lines replacing the logger line would work.

[PeteSL]

I am currently testing the following code as a workaround (adds unnecessary thread if socket timeout keepalive ia implemented) which would work with both angus.mail and javax.mail IMAP implementations (irrelavent code is not included). processInbox reads messages in the folder and acts upon them (not related to this issue). MiscTools.scheduledTasks is a ScheduledExecutorService defined elsewhere.

    @Override
    public void run()
    {
        Thread curThread = Thread.currentThread(); // For idle Interrupt
        
        // Idle thread processing
        for (;;)
        {
            try (Store mailsrvr = parent.sn.getStore(storeurl))
            {
                mailsrvr.connect();
                for (;;)
                {
                    ScheduledFuture<?> inttask = null;
                    try (Folder inbox = mailsrvr.getFolder("INBOX"))
                    {
                        inbox.open(Folder.READ_WRITE);
                        for (;;)
                        {
                            while (inbox.getMessageCount() > 0)
                                processInbox(inbox); // Do this before going to idle in case there is mail waiting
                            if (hasTimeout)
                                ((IMAPFolder)inbox).idle(true);
                            else
                            {
                                inttask = MiscTools.scheduledTasks.schedule(() -> {curThread.interrupt();}, 10, TimeUnit.MINUTES);
                                ((IMAPFolder)inbox).idle(true);
                                if (!Thread.interrupted() && !inttask.cancel(false))
                                {
                                    inttask.get();
                                    Thread.interrupted(); // Clear interrupted flag if unable to cancel
                                }
                            }
                        }
                    }
                    catch (FolderClosedException e)
                    {
                        if (inttask != null && !inttask.cancel(false))
                            inttask.get();
                        Thread.interrupted(); // Clear interrupted flag if unable to cancel
                        if (e.getMessage().contains("Exception: java.net.SocketException"))
                            throw new java.net.SocketException(e.getMessage());
                        classlogger.log(Level.INFO, "Closed in idle loop", e);
                    }
                }
            }
            catch (StoreClosedException e)
            {
                classlogger.log(Level.INFO, "Mailbox closed, will retry in 30 seconds", e);
            }
            catch (java.net.SocketException e)
            {
                continue; // Connected socket disconnected does not set Connected = false, close does.
            }
            catch (IOException | MessagingException e)
            {
                classlogger.log(Level.WARNING, "IMAP Error", e);
            }
            catch (Exception e)
            {
                classlogger.log(Level.WARNING, "Unexpected Exception", e);
            }
            catch (Error e)
            {
                classlogger.log(Level.SEVERE, "Unexpected Error", e);
            }
            try
            {
                TimeUnit.SECONDS.sleep(30L);
            }
            catch (InterruptedException e)
            {
            }
        }
    }

This interrupts the thread listening so the socket is closed. A FolderClosedException is thrown and that is where we do a socket closed check (ConnectionException is not passed to the creation of FolderClosedException so the only way to detect is to parse the exception message). It is possible to have an IDLE event and then the thread get interrupted which is handled in the for() loop. Since the socket closed abruptly, Store thinks it is still connected so we have to close the Store before reconnecting.

[jbescos]

No, the assumption is the connection has been closed and it hasn't been. The server can quit servicing the IDLE without closing the connection and without providing any response (per RFC and this is what happens with Exchange Online). In fact, this code assumes the folder is still open and we just haven't heard from the server (per RFC). In essence, this is a "keep-alive" sequence recommended by the RFC to be done every 29 minutes or less of inactivity. The sequence is (after a successful IDLE command to start with) issue a DONE command and then reissue the IDLE command after the server responds. This all depends on the connection and folder still being open. This does, however, allow for the connection or folder to have been closed which will cause handleIdle() to throw an exception based on the ConnectionException thrown here.

What I am recomending is replacing logger.finest("handleIdle: ignoring socket timeout"); with:

idleAbortWait(); // This issues the DONE command and clears responses
if (!startIdle(im)){ // This restarts the idle command.
    close(false, true);
    throw new ConnectionException(this, "Folder closed during Idle.");
}

r will still be set to null causing the loop to continue without affecting either IdleManager or a manual idle() call. If there are any connection errors or folder errors, the folder is closed and a ConnectionException is thrown so that error handling occurs. The startIdle handling is a guess but I think those 4 lines replacing the logger line would work.

OK, I get your point. I am going to work no this. The difficult part will be the test.

[jbescos]

I made a draft, but this is what I expect to change:
#179

I will need to test this.

[PeteSL]

I have a test bed to test against with Exchange Online (MS 365). If you can supply me with a jar, I will test it there. A valid test would be, with any IMAP server, set imap.timeout=600000 (or imaps.timeout=600000) to set the socket timeout to 10 minutes, debug=true, and let run. You should see the idle command being reissued every 10 minutes and send a message to the mailbox at 10+ minutes should show the connection is still active. Test with idle(true) and IdleManager (both should act the same). You could use a shorter timeout for your test. I am using 10 minutes with Exchange Online and it is working well.

[jbescos]

I have a test bed to test against with Exchange Online (MS 365). If you can supply me with a jar, I will test it there. A valid test would be, with any IMAP server, set imap.timeout=600000 (or imaps.timeout=600000) to set the socket timeout to 10 minutes, debug=true, and let run. You should see the idle command being reissued every 10 minutes and send a message to the mailbox at 10+ minutes should show the connection is still active. Test with idle(true) and IdleManager (both should act the same). You could use a shorter timeout for your test. I am using 10 minutes with Exchange Online and it is working well.

Thank you. Let me do my tests first and when I am happy with it, I will share you a jar. In this way I will not bother you that much :)

[jbescos]

I am starting to find things.

We cannot add the code in the section of here:

angus-mail/providers/imap/src/main/java/org/eclipse/angus/mail/imap/IMAPFolder.java

Line 3274 in c399fb4

if (ex instanceof SocketTimeoutException) {

That is happening after the server sends BYE, which is when the server disconnects.

Do you know whether the server is going to send something when it wants to quit pushing?. Or the client just waits for the socket timeout to idle again?.

[PeteSL]

The server is not sending BYE. The client needs to wait for the socket timeout to issue abort the idle and then reissue the idle (keepalive). Are you sure that readIdleResponse isn't setting BYE when getting the InterruptedIOException (which SocketTimeoutException is a subclass of)? The reason I updated my workaround as not working is because if you interrupt the thread listening on a socket, the socket is closed so I can't use Thread.interrupt(). However, a socket timeout does not close the socket so the idle "keepalive" sequence should be able to occur. If readIdleResponse() is setting BYE on a socket timeout, it shouldn't be since a socket timeout does not indicate a socket closure. I'll take a look as well as to how BYE is getting set, if it is. But if it is getting set, that loop would fail, right, since BYE has already been set? Hmmm.

By the way, the only way to implement the keepalive sequence is based on a socket timeout since you cannot interrupt the listening thread without closing the socket.