Mail team (2000-2004)

2019-07-01

Table of Contents

All in-depth articles about my Apple career:

Interviewing

Sometime in the summer of 2000, I interviewed with the Mail team.

I never intended to work in software engineering but I desperately wanted to work on Mac OS X, which was working on releasing the Public Beta within a few months. So a bunch of interviews with the Mail team.

The short summary is I sweated and laughed a lot. I alternated between being certain I was getting the job to being sure I wasn’t.

My most intimidating interview was with Scott Forstall, who was then was the head of applications and frameworks and the manager of the Mail team reported to him. This is what the entire interview was like:

Scott: It’s your first day in your new job, what are you going to do?
Me: Uh, let me make up some stuff on the fly while I sweat and write things on the whiteboard
Scott (expressionless and not acknowledging what I said): Ok, then what would you do?
Me: See above
Scott: See above

What made it extra fun was that the job description was fairly generic and I’d never held a position like this before in my life. I seem to recall this was my first interview so I was not feeling great out of the gate!

Another uncomfortable interview was with Julie Zelenski, who, like Scott, was a master of the expressionless face. Although Julie was even harder to read and had an almost bored expression on her face. I don’t remember what she asked.

An interview with Ben Haller also made me sweat. His expression always seemed to be communicating that what I was saying was not exactly what he was looking for.

My interview with Paul Marcos was a blast and I recall it stretched out for more than an hour. We were in his office and he was actively working on the preferences panel for Mail that they were redoing. Since I was peering over at it, he talked about it a bit and then started asking for my opinions on things.

I was also interviewed by Kwok Lau, head of Software Engineering Operations and thought she was a wonderful person (and she was!). Finally, I was interviewed by Bertrand Serlet, head of the Mac OS X engineering team. His French accent was quite tricky even though I knew French. I liked his energy and enthusiasm for the technology. I don’t remember much other than he didn’t make me sweat too much. 🙂

How do you think this should be worded? Should we have a preference for this? How does this look visually? Could this be organized better?

I was not expecting that I could ever contribute to a software product in such a way. That I could propose things or criticize things and effect change on a product. Amazing. My whole career up to that point had been in technical support style positions where you are basically cleaning up the mess after a product ships and no one ever asks for your input.

After the interviews, I was definitely interested but felt that more of them went poorly than went well. As an aside, I was also interviewing for another position in a newly created group called Customer Seeding.

Amazingly, I was offered both positions. The Mail team was exciting to me, but intimidating and I worried I was in over my head. The Customer Seeding team was more support-like and so it was more in my comfort zone. Plus, being only the second hire on the team after the manager was an opportunity to shape things.

Scott Forstall was relentless in trying to convince me. He would contact me often and I visited the Mail team a couple times as I was deciding. He would see me there and pull me aside to pressure me more. I’ll write more about Scott later, but one of his greatest qualities was that he took recruiting very seriously and if he liked someone, he would do everything to get them.

I can’t deny that it was flattering to me. I didn’t know how much I would interact with Scott day-to-day (turns out it was a lot!), but I liked him and it weighed into my decision. A small aside is that an unnamed person told me before I was hired that I should never trust what Scott says. Intrigue.

Paul Marcos was a big factor in my decision. I got along with him instantly and thoroughly enjoyed debating various technical topics with him during the interview. I learned a lot in the interview and he later revealed himself to be the best mentor I could ever have hoped for.

Mail was always my first choice but I wanted to be sure so I thought about it quite awhile before I accepted the job. The next four years would prove that I made one of my best career decisions ever.

QA

My position on the Mail team was really two jobs in one:

Serve as the lone quality assurance engineer that was fully dedicated to Mail. This included managing incoming bug reports, analyzing, root causing, and prioritizing them (QA)
Be an on-call support person for all of software engineering as well as important people in other organizations. I was the public face of Mail and also monitored the “help” mailing list (Technical support)

I’d never done software testing before, so to be the lone dedicated tester on one of the signature apps for Mac OS X was very intimidating! Now, there were others that did testing across the system and there was this dogfood testing concept, but ultimately I was responsible when Mail was broken.

Even though I lacked experience, there was also an element of looking at things with fresh eyes. Also, there really wasn’t much uniformity and infrastructure in place, so I felt like it was a time to figure out how software should be tested. I felt an incredible amount of freedom to try things out, keep things that worked, and discard things that didn’t.

Bertrand Serlet, our fearless leader, believed that everyone was an engineer and that Quality Assurance (QA) engineering was engineering all the same and in our HR system, we were classified as software engineers. A psychological benefit of that is that I always felt like a true peer of everyone in engineering and I was generally always treated that way. That type of close relationship with QA engineers is vital to making a great product.

Test plans

The first time anyone running the overall project ever asked me for anything was when they asked for my test plans. I didn’t really know what they were expecting so I looked at what others had done. This is when I first realized that test plans are a list of things that seemed worthwile to test at the time that someone demanded you have a test plan. Then, you let them get horribly out of date and don’t ever use.

I learned that no one ever really looked at what you wrote in your test plan. I would miss major things and no one ever noticed. I would throw in random lines like “make sure this works on other planets” and no one noticed. So, it was basically like a checkbox item. Red tape. Didn’t sit well with me.

Eventually, I grew to where I always wrote test plans and I wrote them for me. And I used them. To the outside world, they may have looked the same, but there were key differences.

They were very high level and not very detailed
The goal was to list all the areas I wanted to be sure to check at least once before the product shipped. As products became more complex, it became easier to forget to check something.
Nothing was described specifically, which allowed randomness to be inserted at the time something was actually tested. Each time I checked an area, I could check it differently to get better coverage.

Automated testing

After a few years, there started to be more demands from the higher ups and the biggest was to introduce automated testing into our workflows. I very quickly developed a dislike for it and also saw how other teams fared. The Finder team dedicated one of two QA engineers entirely to automated testing and I was shocked at the ineffectiveness of it.

I became more of a student of QA at this point, learning about what techniques existed and which were considered effective and ineffective. But even that didn’t always jive with my day-to-day experience. If I looked at things from an efficiency standpoint, looking at every hour I had to test software, which things were the most effective?

Manifestoes

I evangelized a lot of what I learned about testing, but never made much headway with managment. Eventually, many years later, my experiences in these early days experimenting with testing methods led to a number of documents I wrote about the testing process.

The seeds that were planted in 2000-2004 germinated for the next decade and more as I continued to test software in an increasingly rigid and larger company. These manifestoes have proven to be useful to those that have stumbled upon them so I have them available on this website and have listed them below.

Ok, these are all very long! But I’d been writing them in my head for 10+ years so when I got around to writing them, it was really just a matter of hours before each of them were written.

I could split them up, but, like software, all of this is deeply interconnected. You can’t improve your testing procedures for a feature without thinking about the usability mistakes you made the last time you did a feature like this. It’s one big interconnected mess.

Kind of like software. 😉

Technical Support

My first 12 years at Apple, I held three different jobs that were just amazing. Mail edges out the other two, but it’s very a very close contest.

I’ll cover the technical support part of the job here. The reason for needing this was a mandate that everyone in software engineering live and work on Apple Mail as their only email client. Some of the executives were also heavily “encouraged” to do so.

This was my first exposure to the dogfood concept. Essentially, having everyone use the product would tease out edge cases that stemmed from the wide variety of workflows and usage patterns that people have for email.

Mail was not in very good shape when I started. I don’t think any of the engineers at the time would take offense at that statement. The codebase stemmed from the NeXT email client that would read messages from a local UNIX spool file. POP was added later and IMAP was just getting fleshed out when I started.

The support part of the job started out to be well over half the job. My phone was very active, people would stop by for help, or send email. There were so many benefits to this being part of my job, which I’ve split into two buckets: software engineering and important people.

Software engineering

I got to meet everyone in software engineering due to this job. The benefits of that were reaped even many years after I left the job. Other than the networking aspect, I got to learn how people used Mail more than anyone else at the company. This helped tremendously when fielding bug reports, testing, and also participating in design discussions for the next versions of the product.

Some people that became long-time friends sprung out of this part of my job. Having a phone and in-person technical support background helped me out a lot here and I later found this was a big reason they hired me. Quite honestly, I had virtually no experiene doing any of the other parts of my job.

I learned the importance of having a public face for a team. If people know that a team listens and is responsive, they are more likely to write good bug reports and be cooperative when more data gathering might be a drain on their time.

I also learned the app inside and out, in ways I probably never would have if my job had only been to test the application. I was very proud of the skills I developed and so grateful to my team members for all the time they gave to me to learn what I needed to learn. It’s good to work with people that understand the need to invest in people and have it pay off later.

Important people

I had prior experience dealing with executives and potentially difficult people in jobs prior to Apple, so it was not intimidating to provide support to high-level people including executives. Again, this was profound for networking benefits. If I can send an email to an executive and have a higher chance of being listened to because of a positive support experience they had with me, that benefitted me greatly.

I also have some good stories that came out of it.

The head of the Mac OS 9 (“Classic”) team was challenging to help because there appeared to be a lot of bitterness there towards the Mac OS X team. So, while I’m trying to help, I would get an earful about how crap Mac OS X was compared to Mac OS 9. Not everything was unfounded, but it was challenging to be diplomatic because I understood how people feel when a company buys another company and essentially gets replaced by it.

Some of the people I dealt with like Fred Anderson (CFO) and Avie Tevanian (SVP of Software Engineering) were always very nice and enjoyable to talk to. It was nice to get to spend time with people several rungs of the ladder above you and to be granted some time for chit-chat or brainstorming.

Sometimes I’d deal with executive admins rather than executives. On the plus side was a wonderful woman named Bambi Fernandez. She was Avie’s admin and she sometimes had issues of her own or she would supervise me in Avie’s office. She would have a list of mailboxes with juicy, enticing names like “board meeting notes” and “emails from Steve”—I would threaten to click on the mailboxes and she would freak out but I never did.

Steve’s admin was someone whose name I’ve since forgotten, but I got many an earful from her over the years. I had experience dealing with yelling people so I was cool with it, but nonetheless, this is Steve’s admin not just a random yelling person.

One of my favorite stories was going to Jon Rubenstein’s office to troubleshoot. He was the executive in charge of Hardware Engineering at the time. One day I was sitting at his desk with him behind me and I was mousing and clicking and noticed that the mouse felt weird. Heavier. Then I noticed no cord coming out of it. This was before the wireless mouse had been announced.

I was surprised to see this so I looked back at him and he was just holding his index finger in front of his mouth in a shushing gesture. Good times.

Conclusion

I later realized that combining so many skills into one job made a whole that was better than the sum of its parts. Unfortunately, as software gets more complex, you can’t realistically have someone doing all these jobs at once.

When I started on Mail, we hadn’t even shipped Public Beta yet, so we effectively had a few hundred Mail users in the entire world. Nowadays, there are probably thousands of active bug reports and tens of thousands of internal users. The app is far more complex.

I have some thoughts on scalability, but it’s a difficult task. In any event, the support part of my job was really fun and challenging and I feel proud of the work I did.