Lies, bias, and a collection of things I’ve learned from 100’s of hours of interviewing for product design roles and mentoring designers.
Quick note: You might notice some in-progress / incomplete sections in the write-up below. This document is edited and updated constantly. So if you don’t see what you’re looking for, check back at a later date!
Start here:
Hiya, welcome! You’re probably here because you’re about to graduate from a master's program, bootcamp, or are just wanting to ditch your current role.
And you’re thinking:
“Shit… I guess I need to find a job...”
“This upside down monitor is really bringing down my mood.”
You’re probably feeling a bit stressed. OK A LOT STRESSED. If you’re new to product design, you don’t know what to expect as you delve into the professional world. If you’re already a pro, you’re probably wondering how to get to the next level.
This playbook is created entirely to help you with that. It’s a collection of things I’ve learned by interviewing for product design roles.
You’re also probably thinking: ”Why should I listen to this guy’s advice?”
All I can say is that: These are methods and strategies that helped me progress through interviews; from random startups to large tech giants. I can also say firsthand that it’s helped mentees I’ve worked with, to land gigs, negotiate salaries, and vet employers.
With that in mind, I promise to do my best to cite sources an provide with evidence for all the things I recommend.
Complete transparency ……I’ve learned a lot over the 9+ years I’ve been in this career and I still have a lot to learn. So I encourage you to, take what’s helpful to you and ditch the rest.
Good to go? Let’s rock and roll.
Step one, welcome to the theater:
Back in 2015, the TSA missed 95% of mock explosives or banned weapons smuggled through by undercover investigators. It's why whenever I fly with my wife, we joke that we’re “going to the theater.” (indeed the finest security theater, my good chap.)
In that same way, most companies aren’t very good at finding the right candidates, let alone crafting a solid interview process. It’s common practice to just blindly copy the process of tech giants like Google, in hopes of emulating their success.
Surprised? I was too.
App critique, portfolio, design exercises…
Do these steps prove anything? Are recruiters better than a coin flip at determining a quality candidate? Barely…
Can an employer tie your app critique performance to your success in the workplace?
Can you show me stats that show a good portfolio means a good designer?
Can you show me stats that you even hire quality candidates?
No. You can’t. Because much of it is just “design theater”.
The irony is insane. We’re a bunch of designers who can’t even follow the practices of “measure and test” that we espouse.
And after all that… With all these companies copying Google… What did Google have to say about their own hiring practices?
Well here’s what their own head of HR had to say in 2013:
Years ago, we did a study to determine whether anyone at Google is particularly good at hiring. We looked at tens of thousands of interviews, and everyone who had done the interviews and what they scored the candidate, and how that person ultimately performed in their job. We found zero relationship. It’s a complete random mess…
- Laszlo Bock, senior vice president of people operations at Google (source)
So what are interviewers “measuring?”
Many things. But one of the biggest is "themselves." Most interviewers unconsciously look for their own traits in candidates. Here’s a study on this talking about how perceived similarity “significantly influences job suitability evaluations.”
Because, hey! they’re qualified, so if a candidate is like them, then they must be qualified too. But that’s not a thing most people are able to or willing to reflect on or control.
(here’s another article going into that bias)
So they tell themselves that they’re just “very selective”. The sad truth is that often times they just haven’t found a candidate that reflects their own views.
A friend of mine pointed out that author, Nassim Taleb, coined this effect: **Wittgenstein’s Ruler.**
*”Unless the source of a statement has extremely high qualifications, the statement will be more revealing of the author than the information intended by him. This applies to matters of judgment.
According to Wittgenstein’s ruler: Unless you have confidence in the ruler’s reliability, if you use a ruler to measure a table you may also be using the table to measure the ruler. The less you trust the ruler’s reliability, the more information you are getting about the ruler and the less about the table.”
What should they be measuring?
Math time! Here’s the best study I’ve found on how to predict performance in different jobs. It’s a meta-analysis of selection tests used by employers, drawing on hundreds of studies performed over 100 years.
Our strongest indication is surprisingly IQ tests. Even beating out a traditional interviews for their ability to predict job performance. The an R-value of 0.65 suggests a positive correlation between the variables, but the strength of the correlation is not considered very strong.
The general guideline for interpreting the strength of the correlation coefficient is as follows:
0.0 to 0.3: Weak correlation
0.3 to 0.7: Moderate correlation
0.7 to 1.0: Strong correlation
Therefore, the R-value of 0.65 falls within the moderate correlation range. It implies that there is a discernible relationship between the variables, but it may not be strong enough to make definitive predictions or draw conclusive conclusions.
Let me say that again: The strongest indicator of job performance (IQ tests), according to 100 years worth of meta analysis, may not be strong enough to make definitive predictions or draw conclusive conclusions.
Where does that leave us?
Well shit, if we know that:
Most design interview processes are bias…
And that the best methods we’ve determined for evaluating candidate’s job success only have a moderate correlation with actual job performance…
Then….
We can accept that interviews were never an objective measure.
That doesn’t mean that interviews can’t be valuable, it just means they’re not infallible.
It’s freeing to accept that fact, and it’s helpful for dealing with issues like impostor syndrome.
From here, we can think about using those biases to our advantage.
If we know that: Interviewers often make snap judgments on a candidate based on their speech and perceived social class.
Then: We can practice our speech and the way we present ourselves to others.
Then: We can book our interviews for earlier in the week and earlier in the day
If we know that interviews weight past experience heavily in their decision. Then: we can look at our experience and then target jobs in that niche.
So put on your costume and let’s go learn our lines.
We’ve got a show to put on.
Let’s start the show!
We’re all at different places in our journey, feel free to dive into the area that suits you best.