Killing one bird with two-and-a-half stones in Mac OS X Mail

This story is important to me because it was the first software I ever wrote that shipped to more than just a few friends or colleagues. What started as some scripting turned into Cocoa development, which is not something I ever expected to find myself doing. It never became a full-time thing for me, but it was a fun journey.

Note: I joined the Mail team in June 2000, three months before the Public Beta. We shipped the first “real” version of Mac OS X (10.0) nine months later.

On March 24, 2001, Apple released Mac OS X 10.0 (internally codenamed Cheetah). The initial version was slow, incomplete, and had very few applications available at launch, mostly from independent developers. While many critics suggested that the operating system was not ready for mainstream adoption, they recognized the importance of its initial launch as a base on which to improve. Simply releasing Mac OS X was received by the Macintosh community as a great accomplishment, for attempts to overhaul the Mac OS had been underway since 1996, and delayed by countless setbacks.

macOS (previously Mac OS X)—Wikipedia

An accurate and fair description! I previously wrote about the first day we first printed something in Mac OS X a month before the ship date! Everyone knew the operating system had large areas of missing functionality, and Mail was no exception.

One of the big holes we wanted to fill in Mail was the ability to import messages from other email clients. Everyone was keen on using Mail because it sported the fancy new Aqua look and feel (pinstripes! drawers!). Without an importer, you were left running your existing client, probably in Classic, and having your email split between clients. At that time, IMAP wasn’t yet a big thing, so you couldn’t just configure your account and have your messages magically appear.

We felt we couldn’t ship without an importer, yet there was a long list of things we felt we couldn’t ship without. Since everyone was busy with even more important stuff, this became a great summer intern project. Well, unless you’re the intern!

Mail’s first importer

Even though there was a loosely defined file format for mailboxes called mbox, it wasn’t a standard. Simply put, an mbox file is the raw source of every message in a mailbox, strung together into one large file. There is an RFC—essentially a proposal for a standard—which stated:

The mbox database format is not documented in an authoritative specification, but instead exists as a well-known output format that is anecdotally documented, or which is only authoritatively documented for a specific platform or tool.

It turns out that Eudora and Netscape were the only major clients that used mbox format, BUT they had their own variations:

Eudora used an mboxo variation where a sender’s email address is replaced by the constant string “???@???”. Most mbox clients store incoming messages as received. Eudora separates out attachments embedded in the message, storing the attachments as separate individual files in one folder.

The Mozilla family of email clients (Mozilla, Netscape, Thunderbird, et al.) use an mboxrd variation with more complex From line quoting rules.

Mbox—Modifier mbox—Wikipedia

So, Netscape required some ugly trickery, and bringing in attachments for Eudora wasn’t going to happen. It didn’t help that mbox is already a pretty difficult format for parsing. It’s not like XML, where data is labelled and structured. It’s just a stream of text. The end goal was to massage the mbox files from these two clients into the mbox format that Mail used (which followed the “standard”).

Then there were many confounding factors. Sometimes the mailboxes in the file system didn’t map to how they were organized in email clients. There was sometimes a separate database that would track this sort of information. Did we have time for even more error-prone reverse engineering?

And speaking of reverse engineering, there were two very popular clients that didn’t use the mbox file format at all: Microsoft Outlook and Claris Emailer. Even though we had the source code for Claris Emailer, it was still quite a daunting task, with only a few weeks to complete it.

The end result was an importer that was rather disappointing, to be honest. And I think the intern would probably agree with me. Too much gnarly stuff and too little time. For two clients, there was no attachment import. For Outlook, everything was imported into one mailbox because we didn’t crack the code in time. In some cases, mailbox names and the organization of them was lost. There were issues with encoding. The whole project was a minefield of edge cases.

Icky workaround

Back then, Mail would store local mailboxes in ~/Library/Mail/Mailboxes and would look there on launch to load existing mailboxes. If Mail encountered a corrupt mailbox or message, you could rule it out by moving some mailboxes aside and relaunching.

So, there was an opening here to trick Mail into importing your mailboxes. Provided you organized mailboxes and formatted them the way Mail was expecting, it would “import” them. For example, if you had a mailbox you created in Mail called “Apple”, this is how it would look in the filesystem.

Then, if you had an mbox file from another email client, then formatted in a way that Mail understood, you could create a folder in the filesystem (e.g. Pear.mbox), drop in a file named “mbox” and then relaunch Mail. Cross your fingers.

Mail’s second importer

Applescript can sometimes be an effective glue when no good solution exists. My thought was that if Mail were scriptable, we could simply grab messages from one client and stuff them into Mail, all with a script. But Mail was effectively not scriptable at that time. I pushed for getting some basic support for 10.0, but there were too many other things on the agenda and adding scriptability to Cocoa applications was very much a work in progress.

My hope was we could pursue this after 10.0 shipped, and then we wouldn’t have to rely on reverse engineering file formats. We could support attachments, mailbox hierarchies, and perhaps avoid some of the problems the first importer had.

BUT!

After work on 10.0 was closed down, I had an epiphany. Like many epiphanies, they are obvious in retrospect. Given we had this icky workaround to “import” mailboxes, perhaps I could use Applescript to grab data from other email clients, manually construct mailboxes Mail could understand, then quit and launch Mail. Voila!

When I proposed the idea, it was assigned to me. Surprise! So I spent an intense month on scripts for five major email clients: Entourage, Claris Emailer, Outlook Express, Netscape, and Eudora. The last two either had no Applescript support or it was so poor as to be usable. So, I had to replicate the reverse engineering that the built-in importer did for them, but I had some time to improve on it quite a bit. So it was worth the effort.

While it fixed many problems with the built-in importer, it still didn’t support getting attachments. There was no progress indication, and the import was all-or-nothing. So if you had 10 years of email in Eudora, plan on waiting… and praying… indefinitely.

I initially wanted to create one script, but Applescripts had a 32k limit back then, so it would’ve been too large! So, I ended up with 5 scripts. It was possible to put common code into a shared library, but this was not working for me very well at the time. This was a huge pain because the same code had to be copy-and-pasted between scripts and the same bugs had to be fixed in multiple places.

As an aside, debugging is pretty terrible for those who don’t know Applescript, even if you used one of the third-party script editors. You put in some logging, run it, and look at a big pile of output and try to guess what went wrong.

AppleCare agreed to host the scripts, and they shipped the same day that Mac OS X shipped. They let me ship them uncompiled so people could see the code and adapt it to situations or clients I hadn’t had time to get to. There were even versions for Mac OS 9, so you could run them on that side of the fence, and then move the output to the right location when you booted back into Mac OS X.

Here’s the page from archive.org a few months after they shipped: Mail Import Scripts.

Here are the original scripts in all their terrible splendour:

• Import Entourage 1.0.5
• Import Emailer 1.0.5
• Import Outlook Express 1.0.5
• Import Eudora 1.0.5
• Import Netscape 1.0.5

Future updates to the importer

The importer went through many changes after the initial version posted on the AppleCare website and in the next four versions of Mail I worked on.

Updated scripts

Three months later, AppleCare posted new versions of the scripts that addressed some of the biggest shortcomings. Attachments were now imported! The code for Emailer and Eudora was particularly gnarly.

• Import Entourage 1.1
• Import Emailer 1.1
• Import Outlook Express 1.1
• Import Eudora 1.1
• Import Netscape 1.1 (This one got corrupted somehow!)

I added super cheezy progress indication by opening a blank TextEdit window and then appending text to it. It was a live-updating log file. It was one part clunky, one part sweet. Applescript has never been known for progress indication. This was before Applescript Studio (ASS) existed… and then got killed.

And for Emailer, the script would copy the attachments folder over into Mail, then manually construct messages that contained links to the attachments. That script is some of the gnarliest Applescript I’ve ever written. It was a mess, but it worked and was better than nothing!

I added support to select individual mailboxes to import, so it was no longer an all-or-nothing proposition.

Puma (10.1)

An improved importer was on the feature list for 10.1, codenamed Puma. Since it was only a few months after the updated scripts, the main change was that the scripts were integrated into the Mail interface. The old importer was gone, and the new one merely launched the scripts and guided the whole process.

Jaguar (10.2)

I wrote a new generic mbox importer for clients other than the five main ones for this release. It would sniff around in a folder hierarchy and determine if anything could be imported. One common use case was people wanting to import messages from Outlook for Windows.

Additionally, this was the first release that enabled importing mailboxes in Mail’s own format! A small thing, but better than making people shuffle things around in the file system and hope they did it correctly.

Panther (10.3)

This release featured a native Cocoa importer that used Apple Event APIs to extract data with other email clients. The biggest benefit was an immense speed boost. The biggest benefit to me was that it was now fully debuggable. It was my first multi-threaded code and my first significant Cocoa project.

This release included Applescript support that was finally ready for prime time. The grand irony was that, now that we had good Applescript support, we never used it for the importer. Since I had now written this native importer, it was faster and better to use Mail’s APIs to create mailboxes and stuff them with messages.

We would’ve had to add a lot more Applescript support to do an end-to-end importer entirely in Applescript as I had originally envisioned. You would need to be able to create mailboxes, create received messages (with attachments), and append them to mailboxes. No email client I’m aware of allows you to do that, even to this day. I feel like the Panther importer was the best solution all along, and if anyone had thought of this back then, it could have been a 10.0 feature.

Also, in this release, we included a host of sample Applescripts. It included the famous Crazy Message Text script written by Sal Sogohian. I’m pretty certain I wrote the rest. They were great tests of the Applescript support in Mail and great examples for scripters. You could now create scripts and attach them to rule actions, put scripts in the Scripts menu, and even attach keyboard shortcuts to them. Even the preferences were scriptable. We had gone from being terrible to being in the top tier for scriptability.

It was too depressing to try the scripts I wrote as I expect changes in Applescript and Mail in almost 20 years have certainly broken them. I did try Crazy Message Text, and it was indeed broken! But I fixed it up. When you run it, it creates a new Message and pops in whatever text you want in various crazy colours, sizes, and fonts.

If you’re bored, I zipped them up here.

The account preferences needed to be rewritten to support the scriptability of preferences. Of course, bringing that up meant I was once again volunteering for something. So that became my first project that every Mail user would use. Scary! The worst that could happen with the importer is it would fail, and your email couldn’t be imported.

Leopard (10.4)

I left Mail before Leopard shipped. However, the importer lived on as is for quite some time. Eventually, the client-specific importers were removed, and all that remains are the mbox and Mail importers. Here is a sequence of screenshots from versions after I left, up until the present.

Conclusion

From the outside, it can sometimes feel like Apple is a chaotic mess. How could they have two importers that overlap each other? And the built-in one they are telling you not to use. Go download these Applescripts instead. It’s laughable to look back on that now.

The truth is that most of the time, things in software don’t go as planned, and you decide to ship something ugly and confusing because it’s better than not shipping it.

As I recall, we didn’t get any major criticism for it, and the scripts were widely appreciated. There was intense scrutiny of Apple after so many failed attempts to bring out a new operating system. At the same time, there were a lot of understanding users out there that enjoyed Mac OS X so much that they looked beyond its many faults. After a few releases, I looked back at the other email clients and was happy to discover that our importer was among the best out there!

Sometimes it’s better to focus on improving your software and not obsessing too much about what your competitors are doing. It can cause you to focus on doing what they do, only better. This can blind you to going down paths they never considered.

Think Different and stuff.

Download all the scripts

Here’s a link to an archive of all the scripts. Note that most of them likely won’t work and definitely won’t compile unless you still have an ancient version of some of these email clients lying around.