John Gruber, Mark Pilgrim, Mail.app and openness
John Gruber and Mark Pilgrim are having a very public and very excellent conversation about Apple and file formats, proprietary and open.
Mark Pilgrim announced 
that he was switching from OS X to Ubuntu, citing the ever-advancing proprietary creep in Apple as the main reason for his switch. Apple “just doesn’t get it” when it comes to open file formats.
Uproar. Not least because of Pilgrim’s reputation as a long-standing Mac guru.
John Gruber responded to the post, arguing that “Apple gets it / Apple doesn’t get it” is too crude a view:
The question isn’t “Does Apple get it?â€, but “Does Apple get it enough?†…. [W]hile it is easy to find ways to complain that Apple is not open enough — under-documented and undocumented security updates and system revisions, under-documented and undocumented file formats — it would be hard to argue with the premise that Apple today is more open than it has ever been before. (Exhibit A: the Web Kit project.)
But there are things that could be better, should be better, but aren’t, and it’s hard to ascribe these policies to anything other than management that is, at best, indifferent to issues related to openness.
Interesting as this all is (and there is a lot more of it—you should read the posts on both sites), I am posting this because it turns out that Mail.app played a crucial role in Pilgrim’s decision to switch.
In his response to John’s response, Mark writes that Mail 2.0 finally forced his decision to switch:
Tags: Apple, Apple Mail, emlx, john gruber, mail.app, mark pilgrim, mbox, open format, open source, openness, proprietary file formatsAnd then came Tiger, and Mail.app 2.0. In Mac OS X 10.4, Apple deliberately changed Mail.app to use their proprietary .emlx data format, apparently to work around the limitations of Spotlight. Mail.app 2.0 helpfully auto-converted all my wonderful mbox files into Apple’s shitty undocumented format. I’m now in the process of undoing the damage….
This was really the last straw for me. I was already feeling vaguely dissatisfied with Apple; now I feel actively betrayed. By the time I even realized what had happened (a year after buying OS X 10.4), it was too late. Now I’m forced to migrate all my mail yet again from yet another proprietary format, and the best documentation I’ve found so far is on LiveJournal. Jesus H. Christ, somebody deserves to be fired for that.
Related posts
June 19th, 2006 at 11:23 pm
You didn’t quote it but did you catch this comment on his rebuttal about Mail.app:
MARK: “I tried an emlx-to-mbox converter program, but it has bugs that ruin certain mail messages and corrupt the resulting mbox file.”
Now, I haven’t used Mail.app since 2004, but can’t Mail export to mbox files simply by dragging a mailbox to the desktop?
June 19th, 2006 at 11:31 pm
What I don’t get is why someone who cares about storing data would use a mail _reader_ to store email. I store my email in maildir format, and since Mail.app cannot read that directly I use an IMAP server to read it, but at any time I can switch to mutt to read my mail, which is safe in an open format.
June 20th, 2006 at 12:09 am
Mark Pilgrim is being overly dramatic.
As Jeff pointed out above, Mark can get all his emails out of Mail in mbox format just by dragging the mailbox to the desktop. No need for third party mail converters.
Even *if* the drag export function didn’t exist, Mark Pilgrim’s comments about being “physically sick” about this are just overly dramatic. Mail is fully AppleScriptable; someone as skilled as Mark could knock out an exporter in whatever open format he liked in less than an hour.
In general, I find Mark Pilgrim’s whole argument based on unfounded, unfair premises and an overly rosy view of the current state of Ubuntu. For instance, if he isn’t satisfied with the iPhoto XML file, it’s easy to knock out an AppleScript to pull all your data out in whatever format you like. I fail to see how writing code to pull data out of a Linux photo app’s SQL datastore, as Mark proposes to do, is any different.
June 20th, 2006 at 12:33 am
Alan,
Could you give me details about your setup.
June 20th, 2006 at 12:43 am
Oh, and here is another missive about Mail.app:
http://www.oreillynet.com/mac/blog/2006/05/good_bye_my_lover.html
June 20th, 2006 at 12:48 am
Done that one already :)
Thanks though. I’m always on the lookout for that kind of thing.
June 20th, 2006 at 1:01 am
Alan brings up a TERRIFIC point.
What is especially nice about maildir is that it stores every message within a single text file, and each of those files resides within a directory named to match the corresponding IMAP folder.
In other words, maildir is an ideal format for Spotlight.
It’s something to keep in mind. This also lets you use other clients (like PINE and Thunderbird) pretty easily.
(of course, getting an IMAP account on a remote server has its merits too; except that Spotlight cannot see remote servers)
June 20th, 2006 at 1:46 am
Yeah, I read the whole drama this weekend. While Gruber is insightful in few areas, MAN was his rebuttal long-winded! Goodness. Pilgrim’s counter-point about Gruber focusing on minutae was accurate, and Pilgrim’s desire for formats that last and are not DRM’d is a somewhat valid one.
One commenter on Pilgrim’s site nailed it: something like, “you’ve been spoiling to switch to Linux, and now you’ve accumulated enough misplaced reasons for doing so. You’re just bored with OSX.”
There’s simply no way to know what state our data will be in in 50 years, regardless of platform.
And why keep all those old emails, anyway? Deletion=cleansing!
June 20th, 2006 at 2:04 am
Tim,
Sorry for the dupe. I searched Hawk Wings on his first name (Francois) but I didn’t get a match. Maybe it has something to do with the way the “c” in his first name looks in your linked article?
June 20th, 2006 at 3:28 am
I just wrote a big explanation, and the comment just disappeared with a weird php error :( So I’ll redo it very shortly:
fetching and sorting mail:
fetchmail -> procmail -> maildir (everything available by default in OS X)
reading mail:
mutt (directly on maildir) or Mail.app using IMAP
IMAP server: dovecot (easy to configure and compiles out of the box, reads mail as maildir)
advantages: mail stored in nice format, readable using several tools, synchronized across several machines (the IMAP server at my work desktop, my laptop, using Apple.mail IMAP synchronization), nice spotlight integration as I get all messages and attachments
cons: mail stored twice on server (once as maildir, once in Mail.app)
June 20th, 2006 at 3:54 am
Interesting post Alan. I tried to get something like this working last year but I seemed to be having problems with Dovecot allowing me to authenticate, even though it was running on the same machine I was using. Maybe I was just having problems with Pam?
In any case, my plan was this:
fetchmail > procmail/bogofilter > spam/ham mboxes
June 20th, 2006 at 4:10 am
If you need help, you can send me an email. My configuration file is trivial, the only tricky bit is the PAM setup, if I remember correctly.
June 20th, 2006 at 4:20 am
Yeah, I think that is what got me. IIRC, Apple introduced PAM with Tiger. There must be some nice features that people like about PAM because NetBSD is using it now too.
June 20th, 2006 at 10:04 am
That doesn’t work for me (Mail Version 2.0.7 (746.2/750)). What works is to select a batch of messages within a mailbox, then run Files > Save As… (shift-command-s), and choose Raw Message Source from the Format: menu and an output folder/filename.
June 20th, 2006 at 10:13 am
Pretty sure PAM’s been around since 10.2, if not even earlier. I’m too lazy to dig up old backups and check. It’s definitely in 10.3.
June 20th, 2006 at 11:10 am
SJK,
You’re right. I have found references to PAM being in 10.2 but I haven’t been able to find a mention of it in earlier versions.
June 23rd, 2006 at 5:54 am
oh please.
Apple “gets it”. Apple doesn’t *want it*. Fer crissakes. Apple isn’t linux, ubuntu and it sure as hell isn’t Open Source. Never has been, prolly never will be.
Apple wants to “control the user experience”. If they gave a hoot about what we want (hell if they believed for a second we even *know what we want*), there’d be a big-ass honking “send us feedback” from on their website.
/rant off
hehe (Sorry Tim!)
June 23rd, 2006 at 7:40 am
No worries, Boris.
Hawk Wings aims to provide a complete service, including room for therapeutic sounding-off ;-)
June 23rd, 2006 at 8:48 am
Rants often “entertain” me, unless they’re ignorantly whiny, redundantly boring, personal attacks, etc.
. . .
I thought parts of the +2-year-old “Fundamental issues with open source software development” article I came across the other day were an interesting counterpoint to some issues recently discussed about whether or not Apple should/shouldn’t/will/won’t Open Source certain OS X software. It doesn’t mention anything about proprietary vs. open file formats, though.
November 16th, 2006 at 8:03 pm
[...] Remember a few months ago there was an apparent stampede of people, headed by Mark Pilgrim, who were abandoning Macs for Ubuntu? (Although some later came back.) [...]