Interview feedback and the Thumb-o-meter™
The typical timeline of interviews at Apple was: (1) a candidate was interviewed by one or two employees at a time, and (2) feedback was discussed afterwards in a conference room or sent to manager and/or recruiter via email. Each approach has its own benefits and drawbacks.
In-person feedback
In my earlier days at Apple, feedback was almost always provided (in person) in a post-interview feedback session in a conference room. Typically, we’d take turns talking about the candidate, mentioning the topics we covered and how they did.
Benefits:
This provided a nice back-and-forth between interviewers as we tried to build a complete picture of the candidate. Everyone would know what happened in every interview session. This would trigger further questions and answers between the interviewers. This was also an opportunity to try to reconcile differing opinions about various qualities of the candidate. One of the biggest benefits of these sessions was the hiring decision was made right there. Everyone whose opinion was considered was in the room.
It was a great consensus-building activity. We required unanimous support for a hiring decision. If there was some dissent, we would consider bringing the candidate back for a second day, bring in some new interviewers, and hammer them on what was found lacking.
I remember one particular interview where everyone raved about the candidate, but I thought he was arrogant and he rubbed me the wrong way. He described himself as an “expert” in both Objective-C and C++, so we brought in two experts in those areas. They gave him more advanced questions and also played devil’s advocate on some of the accomplishments on his resume. He came unglued and we didn’t hire him.
Drawbacks:
The biggest problem with group feedback sessions is that your opinion may be colored by what others say before you speak. If five people in a row speak badly of a candidate, you may revise your glowing review downwards. It’s human nature.
There are other downsides like the time a feedback session can take out of the schedule of a large group of people as well as the difficulty in finding a time everyone is available. Some of the feedback sessions could exceed an hour.
Solution?
One practice we used to counter the main drawback was something we called the “Thumb-o-meter™”. It originated at NeXT and was carried into the Mac OS X group where most of those engineers landed.
The concept was simple. Before the feedback session began, we would count to three and then hold up our thumb rating, channeling rock-scissors-paper. Anything from full thumbs up to full thumbs down was acceptable. Here are what five reference thumb positions stood for, based on a clock:
- Noon: Hire immediately
- 10:30: Hire if no one has objections
- 9:00: Indifferent
- 8:30: Concerns about hiring
- 6:00: Don’t hire
After people held up their thumbs, someone would jot down what everyone voted, usually in 30 minute increments at most. This had one critical benefit. If everyone was thumbs up or down, then there was no need to take any more time from the interviewers. Meeting over! In some cases, if only one or two people weren’t on board, the feedback could start with them. If they are not swayed up or down, then we could proceed with the full feedback session.
But the biggest benefit was that this was a check against people changing their verbal feedback. It was also kind of hokey and fun. Someone would jot down the ratings quickly before anyone could sneakily drift their thumb in either direction. It was self-policing.
Over the last 10 years I was at Apple, feedback migrated much more towards providing it via email to the manager and/or recruiter.
Benefits:
Due to teams being larger, not all in the same physical location, and the difficulty of finding conference room space and compatible times, email was often the ideal way to handle feedback. Even a video conference would cause schedule and time disruptions.
Even if someone was trying to take notes at an in-person feedback, having a written record of feedback can be essential if there is debate about a candidate.
Provided the manager or recruiter don’t read the feedback until they have finished theirs, this can be a way to capture more honest feedback that is not influenced by peer pressure.
Drawbacks:
With email, everything is many-to-one, so there is no dialog between interviewers. Sure, you can try to get clarification from different interviewers, but it’s more ad hoc and uneven.
One of the biggest impacts is that the interviewers feel more disconnected from the hiring process. You don’t see what others have said before a decision is made. It’s a bit like voting and then watching the results on election night.
Subjectively, I feel like the new hires that come from the email feedback process have generally not been as strong. I think when you are writing feedback, you are more careful and collected and might be tempted to be more positive about a candidate so you’re not the odd person out. My theory.
Also, with email feedback you’re spending far less time considering a candidate as you are not listening to the feedback of others and thinking mostly about your own. I think this type of feedback leads to less thoughtful hiring decisions.
Solution?
There are a couple of technical solutions that could improve some of the drawbacks of email feedback. You could set up a website to submit feedback. Once all feedback is submitted, then the site unlocks and every interviewer can see everyone else’s notes. Perhaps it could be like a message board where you could have a dialog with each item of feedback.
You could also set up a room in Slack (or equivalent) specifically for feedback about a particular candidate. People could be free to pop in and out of the room, asking and answering questions, as their schedule and time allows. Basically a somewhat time-disjointed group feedback session.
Conclusion
There’s no one way to handle interview feedback, so I’m just providing an overview of two ways that Apple chose to handle this over the years. If you can find a conference room and everyone is available and can devote a block of time, I think an in-person meeting (or video conference) is well worth the time invested. Hiring the wrong person can be a drain on the team and the organization.
However, we have to recognize that all getting together at once is increasingly less possible, so perhaps employing some new technical strategies could be the best option of all.
