Hiring Interviews Are Terrible: Smart UX Teams Structure Them

Introduction

Does anyone like the hiring interview? For job applicants, it's a stressful experience. You meticulously prepare your resume and portfolio materials, arrange your schedule to accommodate interviews, are peppered with random questions or whiteboarding-design challenges, and face the collective judgment of strangers. And when you disappointingly don't receive a job offer? You are left wondering what you could have done differently. Was it your portfolio? Your answers to their questions? Or perhaps your identity was unconsciously held against you?

For UX team leaders, it's also a stressful experience. You must justify opening the position to your boss, evaluate a small mountain of applications to find promising candidates, schedule and interview multiple candidates, negotiate a job offer, and hope they accept — all while work is piling up. And when that newly hired team member disappointingly underperforms or quits in 18 months? You're also left wondering what you could have done differently. Was your intuition off that day? Was it the pressure to fill the role quickly? Or perhaps your judgment was clouded by their work experience at recognizable companies or by their beautiful portfolio?

Hiring interviews are critical for teams because the consequences of success and failure are substantial. Identifying a qualified candidate is like striking gold: research by Ernest O'Boyle Jr. and Herman Aguinis suggests that job performance follows a Pareto or power-law distribution, with high-performing employees being prodigiously more productive than their average peers. On the other hand, the U.S. Department of Labor estimates that a bad hire can cost 30% of the role's first-year earnings. For example, a UX Designer role at $100,000 per year could result in $130,000 in direct costs (including salary and severance).

Despite the stress they generate, the effort they demand, and their high stakes, many hiring interviews are conducted haphazardly and lead to bad outcomes for both job seekers and UX teams. Like with user interviews or usability testing, thoughtful planning and careful moderation are essential for gathering valuable insights from hiring interviews. Ultimately, the success of a hiring interview comes down to the amount of structure utilized in the interview.

Unstructured Hiring Interviews

Unstructured hiring interviews have no predefined questions and emphasize open-ended dialog between the candidate and the interviewer. These interviews can differ wildly, but they frequently have these attributes in common:

• Interviewers may vary for each candidate.
• Interviewers may have questions they prefer to ask, but they may not ask each candidate those questions.
• The questions could cover strengths and weaknesses, work experience, credentials, hypotheticals, or worse — brainteasers (How many golf balls would fit into a 747 airplane?).
• What constitutes an adequate answer is up to the interviewer's individual opinion and there's no standardized guidance on "very poor" or "very good" answers.
• Candidates are also free to ask the interviewer questions and the interview proceeds based on whatever questions get brought up by either participant.

Afterward, the interviewer's feedback to the UX hiring manager is often a "thumbs up" or "thumbs down" recommendation based on their holistic experience with the candidate. Notes or justification for their decision may not always be captured and shared.

Structured Hiring Interviews

Structured hiring interviews are radically different from unstructured ones:

• Interviewers use open-ended questions based on critical job behaviors and skills.
• The number and people involved as interviewers are consistent across each candidate.
• Interviewers draw upon a predefined pool of questions that are asked in the same order with the help of interview guides.
• The questions are either behavioral or situational in nature and they require the candidate to draw upon their experiences to convey their competency or problem-solving ability.
• The interviewer gives a numerical rating to each answer before moving on to the next question.
• Interview-guidance documents help interviewers reference the questions and identify very poor or very good answers.
• Small talk and the candidate's questions are put on hold until the interviewer's assessment is complete.

Afterward, the interviewer sends a collection of numerical ratings and notes to the UX hiring manager. Interviewers do not compare notes or compare candidates with each other until all interviews are finished.

Why Unstructured Hiring Interviews Seem Effective

Unstructured hiring interviews have numerous advantages:

• They require minimal preparation from the interviewer.
• They cater to the natural human desire for autonomy and the belief that we have above-average skills when evaluating others.
• They grant flexibility and discretion for the interviewer to tailor questions to the unique background of each candidate.
• They're cheap to conduct, especially with video-conferencing tools' popularity.
• They deliver reasonably fast feedback to the UX hiring manager.
• They require minimal training or administration by UX leaders. (Although operationally focused staff like DesignOps or ResearchOps are slowly becoming common in mature UX organizations, it's still customary for hiring procedures to be delegated to HR staff and ignored until needed.)
• They allow ample time for the interviewer to answer the candidate's questions and impress the candidate with details about the role, the UX team, and the company.
• They are based on the most natural method people use to acquaint themselves with a stranger — the conversation.

Various psychology surveys over the years have shown that only a third of companies incorporate structure into their interview process. Unstructured hiring interviews are popular because they're the format we're most accustomed to as candidates, interviewers, or hiring professionals. However, despite their inherent advantages and popularity, unstructured hiring interviews are an unwise practice for any UX team.

Why You Should Avoid Unstructured Hiring Interviews

Over 100 years of applied-psychology research has consistently shown that unstructured hiring interviews are worse than structured hiring interviews in various ways:

• The goal of any screening method is to predict a candidate's future behavior and job performance successfully. Unstructured hiring interviews have worse predictive validity compared to structured hiring interviews. Analysis conducted by Allen Huffcutt and Winfred Arthur showed that structure was an essential ingredient of a hiring interview's ability to predict future job performance. A literature review by Julia Levashina and colleagues from Purdue and Michigan State University also gives a comprehensive summary of past research and the components that contribute structure to hiring interviews.
• Unstructured hiring interviews are unreliable, which means interviewers are inconsistent in their evaluations and are less likely to agree upon qualified candidates.
• Because unstructured interviews take an improvisational approach to questioning, it's impossible to properly evaluate all candidates on job-relevant topics. For example, candidate A may be asked about their usability-testing technique, while candidate B is not.
• Research has shown that diversity improves business performance. A recent report by McKinsey found that the most diverse companies are 36% more profitable than the least diverse. Unfortunately, unstructured interviewing harms diversity by introducing bias against historically marginalized groups. A research study by François de Kock and Daniël Hauptfleisch showed that structure-reduced interviews favored applicants who were racially similar to their interviewers. Another study by Sima Wolgast and colleagues showed that using a structured process in a hiring scenario reduced ethnic discrimination.

Unstructured interviews are also detrimental to job seekers. Numerous cognitive biases plague these interviews and lead interviewers to judge candidates on irrelevant details:

• Halo/Horns bias: The interviewer may evaluate the candidate unfavorably on just one aspect, such as a gap in work history, and then reject or discount any subsequent positive information.
• Anchoring bias: If the candidate stumbles over their introduction or the interviewer's first question, the interviewer may form a negative first impression that can be difficult to overcome for the rest of the interview.
• Recency bias: A weak answer to the interviewer's last question can result in a more-negative overall evaluation.
• Attribution bias: If a candidate is a few minutes late to a remote interview, the interviewer may make negative assumptions about the candidate's character (they're unreliable) instead of considering external issues (their computer lost power due to a thunderstorm).
• Confirmation bias: An interviewer who already has an existing negative opinion about something that relates to the candidate may focus on confirming that preexisting belief during the interview instead of asking job-relevant questions.
• Affinity bias: An interviewer may overvalue candidates because they share similar interests, education, or work history.
• Similarity-attraction bias: Your interviewer may overvalue candidates who look like them or have the same cultural background
• Perception bias: The interviewer may unconsciously base their evaluation on stereotypes or assumptions about the candidate's identity.
• Affect-heuristic bias: If the interviewer is in a bad mood on the interview day, their negative emotions may skew their judgment.
• Conformity bias: Interviewers who rated a candidate highly may subsequently lower their evaluations if they discover other interviewers rated the same candidate poorly.
• Contrast-effect bias: The interviewer may evaluate applications relative to other candidates in the applicant pool instead of the job's standard requirements.
• Overconfidence bias: Interviewers tend to overestimate their performance when evaluating people, which leads them to continue using biased approaches.

How sad (or infuriating!) is it to lose out on a promising UX job because of bias instead of an honest assessment of your abilities? These flaws in the human mind misrepresent candidates and introduce unhelpful noise in decision making. Incorporating structure into interviews and relying less on holistic judgments can avoid these problems and give everyone a fairer shot. And being fair isn't just a moral or legal concern. UX teams need team members from diverse backgrounds to design compelling products and services for a global audience.

How to Incorporate Structure in Your Interviews

It is much easier to reduce bias in your hiring interview technique than in your interviewers. UX teams should follow these 10 steps to conduct better hiring interviews:

1. Perform a job analysis of the UX role and generate a list of about 25–35 task statements. Refer to the job description, recall accomplishments of high-performing employees, or ask employees in that role to brainstorm essential tasks, critical situations, and productive behaviors. What do you want to attempt to predict about your candidates? Consider a skill-mapping exercise to reveal your team's current expertise gaps and how a new hire might fill them.
2. Survey your team and team leaders and rank the task statements by importance to the role.
3. Take the top 10–20 tasks and develop situational or behavioral questions, which are the most effective question types for predicting future job performance:
   1. Behavioral questions ask the candidate to draw upon their life experience. For example: "Tell me about a time where you had to present controversial UX research findings to stakeholders. How did you prepare, and what did you do?"
   2. Situational questions present a hypothetical yet relevant and realistic dilemma to the candidate. For example: "Two months ago, you joined an app team as their first UX designer. You recently completed a qualitative usability test with users, revealing several high-severity usability findings from a recent app redesign. You are scheduled to present your results to the development team in a week. You suspect these findings pose a risk to the redesign's success, but you also know that the app team worked many nights to complete the redesign. What would you do?"
4. Create a scoring rubric for each question. Write at least three statements describing a very good answer and at least three statements describing a very poor answer for each question. Rubrics help interviewers rate answers against a consistent, objective standard.
5. Pilot-test your interview questions with a team member in a mock interview and adjust wording for clarity.
6. Document all questions and scoring rubrics into print or digital interviewer guides. Assign at least 5 questions to each guide; the questions can be identical or unique for each interviewer. Make space for notetaking and, for each question, include a 5-point Likert-type scale to document each interviewer's assessment of the answer received for that question. Interviewers should rate the answer before moving on to the next question.
7. Select 3–4 interviewers (including yourself). Brief them on the interview guides, their use, and the importance of structured interviewing. Encourage them to take objective notes on the candidate's answers and not rely on their subjective opinions.
8. Schedule a 1-hour, one-on-one interview with each candidate and interviewer. Ensure all interviewers can interview each candidate.
9. Once a candidate has finished interviewing, arrange a 30-minute followup with all interviewers within 24 hours. Review each question and ask interviewers to share insights based on their notes. Finally, ask everyone else to imagine working with this candidate. Then have them submit a final intuition score from 1 to 5. While it may seem counterintuitive that such a score will reduce bias, Daniel Kahneman's research suggests intuitive judgments can still be effective when informed by relevant objective facts. Maintaining a degree of holistic assessment in the hiring process also increases adoption from skeptical UX interviewers who may initially dislike the reduced autonomy of structured interviewing.
10. Calculate each candidate's score by averaging interviewer ratings across all questions. Calculate the average intuition score across all interviewers. Then add both values and make a job offer to the candidate with the best overall score. Refer to the spreadsheet at the end of this article on how to do these calculations.

Are Unstructured Hiring Interviews Ever Okay to Use?

Unstructured interviewing does have its place — at the end of the interview. After the interviewer has completed their evaluation, they should reserve the last 10 minutes of an hour-long interview session for the candidate's questions. The interview's conclusion is the perfect time to answer the candidate's various questions and take a traditional, conversational approach without biasing the interview evaluation.

Identify Smart UX Teams by their Interview Technique

Job seekers, pay attention to your candidate experience (or even directly ask the hiring manager): Does this UX team incorporate best practices in their hiring interviews? Even if structured hiring interviews may feel unusual, take heart that you're applying to be part of a team that values recruiting talented new people and combating unconscious bias. Your user experience during hiring interviews is a preview of your user experience as an employee. Ask yourself: if this UX team manages such an important decision as hiring arbitrarily, how will it handle other important topics? Who will be my future colleagues and are they guided by sound principles?

Conclusion

Many UX professionals I've met are proud that UX is an evidence-based profession that designs successful products by championing empathy and resisting the human tendency to cater to one's desires. In our work, we use techniques that feel unusual, but we engage in them regardless because they're effective. UX teams should apply that same data-driven thinking to how they interview and evaluate candidates. By structuring your hiring interviews, you will reduce the chances of costly bad hires and cultivate a high-performing design team. Job seekers can also recognize structured interviewing as an important clue that a UX team cares about hiring talented people while promoting diversity, equity, and inclusion.

Despite the arrival of AI/ML technologies that attempt to screen job applicants algorithmically, it's hard to envision a future where UX teams hire anyone without human interaction or evaluation. Hiring interviews, for all their challenges, will always be with us. Let's start doing them better.

References

1. Ernest O'Boyle Jr. and Herman Aguinis. 2012. The best and the rest: Revisiting the norm of normality of individual performance. Personnel Psychology 65, 1 (2012), 79–119. DOI:http://dx.doi.org/10.1111/j.1744-6570.2011.01239.x

2. Allen I. Huffcutt and Winfred Arthur. 1994. Hunter and Hunter (1984) Revisited: Interview validity for entry-level jobs. Journal of Applied Psychology 79, 2 (1994), 184–190. DOI: http://dx.doi.org/10.1037/0021-9010.79.2.184

3. Julia Levashina, Christopher J. Hartwell, Frederick P. Morgeson, and Michael A. Campion. 2013. The Structured Employment Interview: Narrative and Quantitative Review of the research literature. Personnel Psychology 67, 1 (2013), 241–293. DOI:http://dx.doi.org/10.1111/peps.12052

4. François S. de Kock and Daniël B. Hauptfleisch. 2018. Reducing racial similarity bias in interviews by increasing structure: A quasi-experiment using multilevel analysis. International Perspectives in Psychology 7, 3 (2018), 137–154. DOI:http://dx.doi.org/10.1037/ipp0000091

5. Dixon-Fyle, S., Dolan, K., Hunt, V., & Prince, S. (2021, November 12). Diversity wins: How inclusion matters. McKinsey & Company. Retrieved January 7, 2022, from https://www.mckinsey.com/featured-insights/diversity-and-inclusion/diversity-wins-how-inclusion-matters

6. Sima Wolgast, Martin Bäckström, and Fredrik Björklund. 2017. Tools for fairness: Increased structure in the selection process reduces discrimination. PLOS ONE 12, 12 (2017). DOI:http://dx.doi.org/10.1371/journal.pone.0189512

7. Paul R. Sackett, Charlene Zhang, Christopher M. Berry, and Filip Lievens. 2021. Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology (2021). DOI:http://dx.doi.org/10.1037/apl0000994

8. Daniel Kahneman. 2013. Thinking, fast and slow, Farrar, Straus and Giroux.