A solution to Mail.app’s IMAP woes
The writer at gnegg posts a long and pained description 
of his experience with Mail.app’s IMAP implementation.
Although he likes lots of things about Mail, he also notes some well-known IMAP frustrations, and then adds,
Even worse: Mail.app does not send an IMAP move command to move the messages to the trash (or just mark them as deleted). It actually manually copies the messages over! Message by Message. From the local copy to the server. Then it deletes them. And then begins the awful “Writing Changes to disk”, completely killing the performance of my MacBook.
His conclusion — “Mail.app is perfect for reading and writing your daily mail. It fails miserably at all mail administration jobs” — sees him using Mail for reading and writing, but keeping a copy of Thunderbird for all the mail admin tasks that he wants to perform.
Tags: Apple Mail, Apple Mail Tips, bugs, imap, mail.app, thunderbirdRelated posts
May 22nd, 2006 at 1:32 am
He makes a very good point at the very end though — it’s risky keeping both Thunderbird and Mail around. You have a lot of faith that Mail is going to pick up all of the local changes.
(additionally, certain local changes Thunderbird has trouble picking up… In particular, if you turn an IMAP flag OFF on the server, Thunderbird sometimes doesn’t notice; I don’t know if there is an existing bug report for this… On the other hand, Mail won’t be turning off any IMAP flags (excepted for a Junk mail flag) so this isn’t that big of a deal)
May 22nd, 2006 at 4:58 am
If Apple would only respond to some of the IMAP issues I would be using Mail.app in no time at all… This post only confirms the problems I had and explains why I had them.
May 22nd, 2006 at 7:12 am
I’m not sure what you mean by turning off an IMAP flag on the server in a way TB sometimes doesn’t notice. Can you describe that in more detail? Thanks.
May 22nd, 2006 at 7:25 am
This guy clearly doesn’t know what he is talking about: there is no “IMAP move command” (this is simple to verify - just search through the RFC for “move”, and you won’t find anything). This is one of the major flaws in the IMAP protocol - the only way to move a message to another folder is to create a copy of the message in the destination folder, and then delete the original.
It’s actually even worse, because you can’t delete an individual message. You can mark messages with the \Deleted flag, but to actually remove the message you have to purge all the \Deleted messages in the folder.
With the trash, you can be clever and just pretend that any \Deleted messages are in a different folder, even though they aren’t, but you can’t do this with other folders.
May 22nd, 2006 at 7:44 am
For example, let’s say I’m in Thunderbird and I tag a message with $Label1, which might be called “Important” in Thunderbird.
Now let’s say I use PINE or another IMAP client (Apple Mail, even, if it supported IMAP tags) to turn OFF $Label1. The next time I open Thunderbird, it seems to not notice that the server status has changed and it sticks with its cached copy, which has the “Important” flag still turned on.
Interestingly, if the flag is NOT turned on in Thunderbird and I turn the flag on with another mail client, Thunderbird DOES seem to notice that.
This may have been fixed recently, but it wasn’t as of the first release of TBird 1.5.
May 22nd, 2006 at 7:59 am
Tony, I think you’re being too harsh on IMAP, and I think you are mistaken.
This is the document to which you are referring:
IETF RFC 2060
You should take a look at commands like APPEND (6.3.11), COPY (6.4.7), and EXPUNGE (6.4.3) command.
It actually is a little more helpful to view the man page of the mailutil command:
mailutil.1
As you can see, there are a number of ways to “move” messages on the server. It’s true that these methods are not “atomic” in that they copy messages, mark the old ones as deleted, and then expunge them. However, this is all done at the server level using highly efficient algorithms that work directly on the disk. It’s much better than having to shuffle messages off of the server and then back on.
So I *THINK* you’re getting tied up in some semantics issues. Regardless, I’m sure you see the point of the original poster: IMAP commands (even including searching and selecting messages) should be done on the server where there is fast access to the mail.
I’m not quite sure how this is “worse.” Like you say, the alternative is to move messages into a “Trash” folder (which still is available on IMAP).
IMAPv4r1 is very versatile. It is designed SPECIFICALLY to address all of your concerns while also providing lots of neat ways to handle mail while ALSO storing it in a central location on the server.
I think if you really sit down and think about it, all of the things you list as limitation of IMAP are just different ways of doing things. Personally, I’m happier with doing things a little differently because I pick up the added performance of IMAP as well as the ability to manage my mail in one place at one time and share that management everywhere.
Additionally, most modern IMAP clients are designed to allow you to use IMAP just like POP… So you don’t even notice that you’re doing things “differently.” This is the flexibility of the IMAPv4r1 protocol. It’s really meant to be an API to allow a mail client to transparently implement an efficient server-based mail system.
May 22nd, 2006 at 8:49 am
Thanks for the detailed explanation, Ted. Now I understand exactly what you meant.
And thanks to Tony for explaining the lack of an atomic move operation and other IMAP protocol deficiencies. I remember discussions about those particular issues on Mulberry’s mailing list.
Anyone (e.g. Tony :-)) aware of alternatives to the IMAP protocol being proposed? My at-a-distance impression is that the IETF has been more interested in things like moving certain IMAP extensions out of draft stagnation than in coming up with a better protocol to replace it. In other words, it seems the hope and effort is still to “save” IMAP more than eventually abandon it in favor of something superior.
May 22nd, 2006 at 8:56 am
(Tim: apologies for not top-quoting ;))
I develop with IMAP nearly every day, and have done for years now. I am not too harsh at all, and I certainly am not mistaken.
I have read the RFC many, many, times. EXPUNGE (like CLOSE) permanently removes messages marked as \Deleted, returning information about which messages those were (CLOSE doesn’t do this part). That’s it. COPY copies a message, leaving the original alone (i.e. it’s a copy, not a move). APPEND adds a new message, it has nothing to do with moving.
Actually, no, there are no ways. You can only create a new message (either reading and writing with FETCH and APPEND, or more efficiently via COPY) and delete the old one.
Well, how efficient things are depends on how well the server is written, obviously. This is a silly thing to say.
I don’t know what you mean by “selecting” a message. There are too many variables involved to know whether searching would be faster with a local cache or on the server.
I’m not quite sure how this is “worse.†Like you say, the alternative is to move messages into a “Trash†folder (which still is available on IMAP).
This is worse because it is not possible for an IMAP client to delete an individual message (well, unless you check which messages are \Deleted, change that, \Delete your message, expunge, then restore the flags of the other messages, and have some way of being sure that no-one else is changing the flags at the same time). Deleting an individual message is a useful operation when writing IMAP clients, particularly when moving or altering messages.
IMAP tries to be too versatile. It allows too much flexibility in how the protocol can be implemented, which means that coding for it is overly complex. Despite this, it lacks the ability to do many desirable tasks (alter a message, delete an individual message, move a message). There’s insufficient room here to go into all of IMAP’s flaws.
Actually, Crispin didn’t have any of my concerns in mind…
No, they are limitations. Trust me, I work with IMAP server and client code nearly every day, I have “sat down and thought about it” many times.
I think this is taking too much space for comments, so if you want to debate this further, I can blog about this myself and leave Tim’s space alone.
May 22nd, 2006 at 9:04 am
WRT to a new protocol - I think perhaps it’s too late to introduce new email protocols, unless they are backwards compatible with POP or IMAP. To really get adoption, you need to convince Microsoft to support the protocol in “Windows Mail” (the new OE) and Outlook, as well as a reasonable number of others (say Apple Mail and Thunderbird). I think the political obstacles are too high.
Not only that, but you then have to convince ISPs (etc) to provide additional servers. Unless there’s a clear benefit to them, or huge numbers of users are demanding it, that’s unlikely to happen, IMO.
June 30th, 2006 at 8:05 am
Tony: Not that long ago, all those were arguments against IMAP. Or ssh. Or the internet protocol. Yet all of these protocols are very successful now.
There are plenty of good reasons one might use an inferior technology when a superior one is available, but it is (IMHO) almost never worthwhile to limit yourself to an inferior technology simply because the superior technology might not take off. Even if the path you choose does eventually die, so what? You’ve spent a few years being less frustrated and more productive than everyone else, and that’s worth something.
June 30th, 2006 at 10:53 am
You don’t have to be good to be successful. VHS & Betamax are the classical example.
Using new technology in communication isn’ the same as in other areas. Interoperability is essential. It does me no good to switch to some great new mail protocol if I can’t find an ISP that will provide me with access via that protocol (and assuming that I don’t want to run my own mail server).
July 7th, 2006 at 10:40 pm
Hi there,
thanks for referring to my blog. Unfortunately, as I have changed the engine, the old links are not valid any more. If you want to read the posting, maybe consider visiting its new location.
Philip
July 7th, 2006 at 10:49 pm
btw, Tony, about your argument with the MOVE command.
Even if we consider that there exists no atomic move command (you see, I’ve written my blog entry not for a IMAP developer, but for a user of Mail.app, so I was not exactly detailed), what do you think is more efficient considering you want to delete a mail message with a 10 MB attachement and further considering you are behind an analog connection:
1) Do it locally:
- Download the Message (10 MB downloaded - takes ages)
- Mark Message as deleted
- Remove marked message
- Upload the Message to the trash folder (10 MB uploaded)
2) Do it on the server
- Copy message to trash folder
- Mark original as deleted
- Remove marked message
Now tell me what’s faster: Loading 20 MB of data over a analog connection or just moving around 20 MB on the server?
That was my point. Not semantics or protocol details.
Philip
July 7th, 2006 at 11:07 pm
Philip, thanks for the heads-up. I’ll edit the post to make sure the link stays live.
December 19th, 2006 at 11:17 am
I’m developing a IMAP-backended webmail and just got stuck with the problem of moving a message. I wouldn’t like go through all the “copy to temp mailbox, \delete, expunge, copy all back, remove temp mailbox” process. Does anyone have a good suggestion on how to do it?
I’m using PHP, but doing the whole imap control through sockets, ’cause PHP’s IMAP implementation seems to be quite lazy and bad written ’till now. I’m just guessing based on my personal benchmarks and have never looked at the code, though, but it actually performed drastically worse than my own PHP code.
Amplexos. (btw: I used some non-breaking spaces entities to format the paragraphs. in the live preview they look ok, but I don’t know how they’ll look like in the final version, so I’m sorry if they appear as the horrible-looking entities).