H O W
T O
D E - B I A S
Y O U R
H I R I N G
A practical guide to building an ethical,
empirical and effective
Traditional hiring has had its day.
We know the traditional ways of hiring are not perfect because they have led us here, to a place
of great inequality and inefficiency.
People from minority backgrounds are disproportionately overlooked, talent is lost and systemic inequality is perpetuated.
To build a better, more equal world, we must push back against conventional wisdom and biased, antiquated practices with a smarter solution - applying research over resumes, science over the status quo and insights over gut instincts.
IT'S TIME TO TRANSFORM
HOW YOU HIRE, FOR GOOD.
At Applied, we believe hiring should be: Empirical: by using data to hire, not bias Ethical: by improving diversity and inclusion
Efficient: by making smarter, faster, more impactful decisions.
Applied is the essential platform for de-biasing hiring; born out of the UK Government's Behavioural Insights Team (also known as the Nudge Unit) - a leading global consultancy turning behavioural science research into policy.
By anonymizing applications, leveraging a skill-based methodology, and building
transparency and analytics into every step of the process, our platform surfaces genuine talent that would have otherwise been overlooked.
Applied Hiring Process
Attract Apply Assess Interview #1 Interview #2 Hire/reject
Use networks Resume / cover letter 6-10 second review Unstructured gut-driven Unstructured gut-driven No feedback
Attract Apply Assess Interview #1 Interview #2 Hire/reject
Diverse job boards 3-5 work samples Anonymous review Structured case study Structured simulation Personalised feedback
Traditional Hiring Process
Likely to be biased Not predictive Highly predictive
THE GUIDE
1
2
3
4
5
The science behind bias
Screening
Feedback
Interviewing
No matter how well-intentioned you may be, we’re all subject to unconscious biases that affect our decision-making.
A common misconception about unconscious bias is that it’s something perpetrated by a handful of bad apples.
In reality, we’re all biased, it’s simply a part of being human.
If anybody ever tells you they don’t have unconscious biases, they’re lying!
Unconscious bias is, essentially, any prejudices we have that we’re unaware of (hence unconscious).
We naturally categorise others based on physical qualities and background - from ethnicity to education.
Not all of these biases are necessarily discriminatory.
Hiring, for example, can be swayed by biases we have towards certain candidates, rather than against other candidates.
This is likely due to our natural gravitation towards the familiar and the power of certain attributes to entirely cloud our judgment.
FACT: Traditional hiring is prone to
bias
Unconscious bias: a quick definition
Confirmation Bias Feeling a connection to those similar to us Stereotypes and assumptions about different groups Projecting positive qualities onto people without actually knowing them Looking to confirm our own opinions
and pre-existing ideas.
Perception Bias
If we want to effectively remove unconscious bias from our hiring processes, we first have to understand why it occurs...
Our brain makes 1000’s of decisions every day.
And this means that it can also misfire 1000’s of times.
In our day-to-day lives, we make big, important decisions, such as planning a project at work.
And we also make minor, everyday decisions, like deciding which route to take home. The more decisions we make, the more fatigued we become. So we can’t dedicate the same amount of processing power to each and every one of these decisions.
To lighten the load, our brain’s decision-making is broken down into two systems - one fast thinking, and one slow thinking, a theory popularized by Daniel Kahneman’s
Thinking Fast and Slow.
System 1
System 2
Fast, intuitive and emotional
Slow, conscious and effortful System 1: used for everyday, intuitive
decisions. These are the sort of decisions that if you had to think long and hard about every one of them, you’d likely have a meltdown - like how much milk to put into your tea, what top to wear in the morning, or when to cross the road.
System 2: used for bigger, more
important decisions like planning a trip or working on a big presentation. System 2 thinking tends to be slower, considered, and more mentally taxing.
Having these two systems is what allows us to stay productive and sane. However, we often use system 1, when we should be using system 2.
Our brains use shortcuts and patterns to draw conclusions, by using the
information subconsciously stored in our mental lockers: this includes things like our past experiences, what we’ve seen in the media and our upbringing.
The consequences of unconscious bias are real and measurable.
Our tendency to make subconscious associations and resist the unfamiliar means that candidates from minority backgrounds end up being disproportionately overlooked.
Correspondence studies have been used to gauge the effects of bias and the discrimination it causes.
Researchers send identical applications off to employers, usually changing just one variable, such as a candidate’s name or gender.
Unconscious bias in hiring
UK Studies
Extra applications needed to receive 1 callback (%)
Source: Centre for Social Investigation, Oxford University (2019)
Middle Eastern/North African
High-skilled occupations All other occupations All occupations Nigerian Pakistani 120 100 80 60 40 20 0
A 2019 study from the University of Oxford looking at ethnicity found that candidates from minority ethnic backgrounds had to send 80% more
applications to get the same results as a White-British person.
Discrimination across ethnic groups
Source: Wood et al. (2009)
0% 20% 40% 60% 80% 100%
White vs Pakistani/ Bangladeshi
72% Successful 44% Successful
White vs Black African
68% Successful 37% Successful 37% Successful 67% Successful 35% Successful 66% Successful 45% Successful
White vs Black Carribean
White vs Chinese
68% Successful
White vs Indian
These results reflect the earlier finding of a 2009 report.
Each ethnicity was tested against white candidates to mitigate any variation across different
employers (this is why you can see varying success rates for white candidates in the chart). The result: white
candidates were favoured in around 47% of the tests.
And Indian candidates would have to send twice as many applications to get the same callbacks as white-named applications.
US Studies
Source: Bertrand & Mullainathan (2003)
Resume applications needed to receive 1 callback
10
15
Candidates with White-Sounding names Candidates with Black-Sounding namesLooking specifically at the effect of Black-sounding
names on callbacks, one study involving over 5000 CVs found that applicants with white-sounding names needed to send roughly 10 CVs to get a callback, whilst applicants with African-American names
needed to send about 15.
3
Even when employers have pro-diversity in their job descriptions, this doesn’t mean that their process is any less biased.
A study from 2016 found that when applying to an organisation that
presents itself as valuing diversity, minority
background candidates were less likely to ‘whiten’ their CV. However, these
organisations’ diversity statements were not actually associated with reduced discrimination. 30 25 20 15 10 5 0 Source: Kang et al (2016)
Callback rates for black candidates
Whitened name and experienced No whitening Whitened first name Whitened experience
All job ads Job ads with pro-diversity language
Source: Quillian et al (2017) 1975 1980 1985 1990 1995 2000 2005 2010 2015 Response Ratio White/African American Year of study
Despite the boom of the D&I industry over the past decade, these sorts of outcomes aren’t improving.
Intentions don’t matter. If we don’t change the way we hire, systemic inequality will not improve.
5
Bias training doesn’t work - we have
to design bias out of hiring
We’ve all seen the headlines - unconscious bias training doesn’t work. Why? Because it’s near-impossible to de-bias human beings.
Im p lic it A ss o ci at io n T es t Sc o re 0.8 0.6 0.4 0.3 0.0
Baseline Week 4 Week 8
Time
Intervention Control
Source: Long-term reduction in implicit race bias: A prejudice habit-breaking intervention (2013)
Does training reduce bias?
A meta-analysis of 426 studies found that while there was a reduction in bias immediately after training, this disappeared after about 8 weeks.
It might be possible to change human nature with enough training… but we simply don’t have time to find out.
You can’t de-bias a person but you can de-bias a process.
What the training-based approach misses is that unconscious bias isn’t something to be defeated.
It’s simply an inevitable part of the human experience that we must design around in order to make better decisions.
Resumes may be the bread and butter of most hiring processes, but they’re seriously ineffective at identifying top candidates.
The more we know about a candidate, the more grounds for bias there is. We know from the studies above that just changing your name can affect your chances of being shortlisted but still, we continue to use this antiquated, bias-riddled screening practice.
A reckoning for resumes
% of additional resumes that need to be sent by candidates with a non white-sounding names
Canada Australia UK 80 70 60 40 30 20 0 10 USA 50 If we can use CVs to measure bias, surely that should tell us all we need to know about them.
To move forward, we must leave behind flawed signifiers and apply science over speculation and insights over gut instinct.
The case against resumes:
The average resume review takes just 7.4 seconds Candidates viewed first receive a 5% ranking increase
Around 33% of job seekers falsify information on their resumes every year Candidates with non-white sounding names need to send 75% more resumes Education and experience are poor predictors of ability
Name Age Photo
Gender/ethnicity
To combat biases triggered by resumes, some organisations have started to anonymize resumes.
This usually involves manually crossing out identifying information before reviewing begins. This includes:
Anonymizing resumes
Anonymizing resumes is undoubtedly a step in the right direction.
However, what’s left on an anonymized resume (education and experience) still doesn’t matter as much as you think.
? 0 0.1 0.2 0.3 0.4 0.5 Predictive validity EDUCATION YEARS OF EXPERIENCE REFERENCE CHECKS COGNITIVE ABILITY TESTS STRUCTURED INTERVIEWS WORK SAMPLE TESTS
CVs
Source: Schmidt & Hunter (1998)
The landmark
Schmidt-Hunter’s meta-analysis looked at 50+ years worth of studies to determine which selection methods were the most predictive of ability.
As you can see - education and experience are some of the weakest predictors of real-life ability.
If we simply assume that the best candidates come from the most prestigious universities, then candidates from underprivileged backgrounds are always going to be overlooked.
If you go to one of the best universities, you’ll likely get the best experience as a result.
And so the top jobs will continue to go to those from the most privileged backgrounds unless we change the way we hire.
There’s also the fact that women have a tendency to downplay their achievements. On average, women rate their performance less favourably than equally
performing men.
It’s not that experience doesn’t have any value.
But the value comes from the skills gained via experience.
So, why not test for these skills directly rather than guessing at them via proxies?
As the chart above shows, these are the most predictive form of assessment we have at our disposal, can be easily anonymized and don’t require any specific experience, just pure ability.
Instead of CVs, we recommend asking candidates 3-5 work sample questions.
7.4 seconds
average time taken to review a resume
10
75%
They work by taking a realistic task or scenario that candidates would encounter in the role and asking candidates to either perform the task or explain how they would go about doing so.
The idea is to simulate the role as closely as possible by having candidates perform small parts of it.
Instead of candidates talking through their achievements or why they’d be a great hire, work samples directly test the relevant skills.
Candidates aren’t required to talk about their skills, they’re instead asked to demonstrate their skills.
Work samples are similar to your typical ‘situational question’ except they pose scenarios hypothetically - focussing on potential over experience.
Why use work samples?
Work samples are interview-style questions designed to the specific
skills required for the job. Sales DevelopmentRepresentative
We run free
training days in order to help Talent professionals de-bias their recruitment processes and understand how
behavioural science impacts diversity & inclusion.
Once people understand the science, the chances of them becoming a customer are pretty good! *We provide candidates with a link for more information about our training days.
You have built a list of 1000 Heads of Talent in the US. Write an email explaining who Applied are, and inviting them along. Remember at this stage we aren't selling them the platform, just trying to get them to come along to the training day.
Research, Communication
Question:
Technical skills (e.g. SEO, Javascript) Value-based skills (e.g. passion)
Working characteristics (e.g. leadership, organisation)
To make fair, data-driven hiring decisions, you’ll first need to define what you’re going to be testing for.
Usually, hirers might outline some vague characters or experience requirements they’d desire, this isn’t going to cut it!
We recommend choosing 6-8 skills, these can be made up of a mix of:
Pose a realistic scenario
Think of a 6-8 real-life scenario that could (or already has) come up on the job that would require each of the skills.
You can test multiple skills with a single question.
These can be either everyday tasks the candidate would be doing in the role or rarer scenarios/problems that could feasibly arise.
If there are tasks that are writing-based, you can ask candidates to perform them (e.g. writing an email or short blog extract).
If the role doesn’t include these kinds of tasks, you can simply ask
candidates to explain their approach to them.
Prioritisation task - give
candidates a list of tasks and ask which they’d prioritise and why). Written task - email to a
customer, marketing email, blog post)
Explaining a process - ask candidates how they’d tackle a given project or task)
Crisis management - give candidates tricky scenarios and ask how they’d approach it Explaining concepts - ask candidates to explain, in their own words, basic concepts involved in the job (e.g. inbound marketing).
How to create work samples
Decide on the core skills required for the job
One of the biggest pitfalls of traditional hiring is that there’s no way to quantify how suitable someone is.
This is why scoring criteria is a must.
If we want to make hiring decisions using data, we first need to collect that data.
For each of your work samples, you’ll need to create a simple ‘review guide’.
At Applied, we use a 1-5 star scale - with bullet points noting what a good, bad and mediocre answer would include.
Score against criteria
1 star ⭐
3 star ⭐ 5 star ⭐
Review guide
Score:
/5
At Applied, we use a three-step process to completely de-bias the screening process...
1
Anonymization
Remove all non-relevant information on
applications. Including address, hobbies, name of university, names of previous companies.
2
Chunking
Slice up applications into different sections, and then compare each section for all
candidates next to each other.
3
Randomization
Randomize the order in which the sections of candidate applications are reviewed each time.
Ordering effects are biases that alter our perception based on the order in which we view something. Candidates scored first generally tend to score higher and certain attributes can overshadow how we judge someone.
2. How to build a job
description
When it comes to interviewing candidates, removing bias completely can be challenging.
According to one study, roughly 5% of decisions were made within the first minute of the interview, and nearly 30% within five minutes.
As soon as candidates set foot through the door (or pop up on Zoom), your brain starts making associations and misfiring in all sorts of ways.
Did the candidate go to the same university as you? Did they make a good first impression?
Did you interview them at the start of the day, on a full stomach? Did you find them attractive?
All of these factors can and often do affect how we perceive candidates.
Blindfolding interviewers is obviously not an option but there are steps you can take to ensure hiring decisions are as objective as they can be.
Although you can’t turn off the unconscious biases that will be triggered when meeting a candidate, that doesn’t mean you can’t debias the interview process itself.
Is it possible to de-bias a face-to-face
interview?
A structured interview is where all candidates are asked the same questions in the same order.
Structured interviews have been proven to be more effective than unstructured interviews. Why is this? Well, by making interviews more uniform, it makes it easier to objectively compare candidates.
By knowing what you’re looking for and how you’ll test this before the interview starts, you’re able to identify who meets the criteria and who doesn’t.
The more structure you add to your interviewing process, the more bias you can remove.
A structured interview will only get you so far unless you’re also asking the right questions.
Forget about candidates’ backgrounds. It’s about what they can do, not where they’ve been.
Why waste time asking candidates about their skills when you can test for them instead?
Here are some of the most effective styles of interview question for skill-testing...
Use structured interviews
Ask questions that test skills, not background
Work samples
Work samples can be used for interviews as well as screening.
You simply explain the context of the scenario and ask candidates how they’d respond.
Think of work samples as a more
predictive version of your typical ‘tell me a time when’ question, except posed hypothetically.
Client Growth Manager
You haven't had a sale in x months but you know it's just been an unusual period and you think you're in a good place for the future. Your sales manager wants to help you look at your pipeline and activity levels. What do you do?
Case studies
Case studies present candidates with a bigger task to think through.
This is an opportunity to present candidates with a real (or near enough real) project that they’d actually be working on.
After giving them the context, you can ask candidates a series of follow-up questions to see how well they understood the task and what their approach would entail.
Although you’re in a structured interview, you can still ask additional questions to help candidates explore their ideas - just make sure every candidate is afforded this help. Role: Question: Follow-up 1: Follow-up 2: Digital Marketer
Below is some fake data to discuss. To meet our commercial targets we think we need to increase our demo requests from
90/month to 150/month. Below are some fake funnel metrics and website GA data. With a view to meeting this objective, talk through the above data and what it might mean.
What additional data would you need to work out how to meet the objective?
Just like the work samples at the screening stage, each of your interview questions will need scoring criteria.
For the most accurate, objective scores, have three team members score each interview round.
Not only does this mean that any individual’s biases will be averaged out, but it’ll also yield you the most objective scores.
This is due to a phenomenon known as ‘crowd wisdom’ - the general rule that collective judgment is more accurate than that of an individual (two heads really are better than one).
Use scoring criteria and interview
panels
11
At Applied, we recommend having a new panel for each interview round, with a lead hirer remaining throughout.
The more diverse your panels, the more objective the scores will be. We’d advise involving team members from other functions since this diversity of perspective can add real value
Interview round 1
Interview round 2
Work samples screening
Hiring Manager Hiring Manager Hiring Manager
Won’t candidates be intimidated?
Being interviewed by one person is daunting enough as it is. So, the best practice is to divide the interview questions among the interviewers, this way there are no silent judges. It’s also advised to explain to
Since our memories are easily tainted by unconscious biases, interviewers should be encouraged to either take notes throughout or simply score candidates in the interview itself.
Needless to say, interviewers should make note of their scores without conferring with one another.
Here’s what interview scoring should look like:
What made you want to apply for this job? Why now? Why a startup?
Interviewer notes
No real reason
Is excited by the growth journey Wants to be a part of the startup world
Clear passion for what we do
Likes the challenge of working in a startup Displays a growth mindset 1 star ⭐ 3 star ⭐ 5 star ⭐
Define a great, mediocre and bad answer. Your guide ensures that reviewers score answers consistently and fairly.
Review guide
Score:
Loves what we do
Wants to develop their skills in a face-paced environment
Eager to work in a startup and in a smaller team
Looking to grow with the company and get experience of other functions
Passionate about our mission
Take notes as candidates answer question. This will make it easier to score when the interview is over and reduces bias.
Question
Are candidates passionate about your mission?
Will they embody the values necessary for the team to achieve this?
Cultural fit has the potential to be problematic when it comes to de-biasing your hiring process.
It’s is often used as a smokescreen for biased decision making. Can you test for culture and claim to have fair hiring?
This depends on what you’re calling ‘culture’.
If you see culture as being something fixed, to be adhered to, then you’re likely to hire people who look and sound like those who are already in the organisation A company founded by a certain demographic, who hires predominantly people of that same demographic will tend to have a culture geared towards that type of person.
Testing for culture then becomes a measure of ‘how like us’ a candidate is. This is how we end up with organisations - and even whole industries - being dominated by a handful of demographics.
Culture should be something that evolves over time and built upon, then it’s not a matter of candidates ‘fitting in’.
It's a matter of what they can add.
Whilst ‘culture add’ is a more diversity-centric way to think about culture, you might want to consider ditching culture-based assessments altogether.
Instead, simply test for mission and values alignment.
What’s the difference between this and culture?
Mission/values alignment removes personality as a decision-making factor.
Hiring for culture fit
What someone likes to do in their free time, their sense of humour or how fun they’d be at your weekly social shouldn’t affect their chances of being hired. What does matter, is their interest in what your organization does and the values guiding your team.
Turn ‘culture’ tests into work samples
What are you hoping to get out of this role? How does it contribute to your longer-term ambitions?
Why do you want to join the team? Why now?
If you treat mission/values alignment as a skill, you can test it using work sample-style questions like you would any other skill, it can then be tested and quantified objectively.
3. How to attract
candidates
2. How to build a job
description
Usually, candidates are lucky to get a “sorry, you were unsuccessful” email, never mind personalized, useful feedback.
Feedback costs nothing and is something your competitors probably aren’t doing. Below are the results from our own research. The chart shows the candidate experience scores given by unsuccessful candidates.
This is the power of feedback.
And if you’ve followed our process this far, you’ve already done most of the legwork.
Feedback is they key to nailing your
employer brand
1 2 3 4 5 6 7 8 9 10 ⭐ Rating N um be r of c an di da te s 350 300 250 200 150 100 50 0Use candidates’ scores to give feedback
Your feedback should be strictly objective and skill-based.
Since we’ve eliminated candidates’ backgrounds from our decision-making, the feedback shared with them should reflect this.
Let them know which areas they were strong in, and which they could improve. Below is a snapshot of what feedback looks like through the Applied platform.
Whilst this would be extremely laborious to recreate through Google sheets alone, you can get a sense of how we keep feedback impersonal and solely skill-focused.
If your work samples and interview questions were all scored, you can use this to paint a picture of how the candidate performed across the various stages and against the required skills.
All candidates need to know is how they performed against the initial job criteria, and how they fared against other candidates (at an aggregate level of course). Providing feedback won’t claw back the pain of rejection, but it can help to demonstrate that you appreciate and respect the time candidates put into
3. How to attract
candidates
2. How to build a job
description
If your goal is equality and diversity in recruitment, you’ll need to actually track these metrics in order to
optimise your hiring process. This begins with an equal opportunities form.
Although your process should be anonymous, you’ll still need to collect diversity metrics to ensure its
fairness.
Whilst it’s up to you which details you ask for, you should be clear about what they’ll be used for and that they’ll only ever be used at an aggregate level.
1) At the very start of your process, collect your candidates' equal opportunities information. We recommend the following: gender, age grouping, ethnicity, disability status, socioeconomic indicators.
2) Measure these at each stage of the application process. Ensure that this information is only ever shared on aggregate and not on an individual basis.
3) Monitor your funnel for each role and look for drop-offs that may need to be addressed.
You can't change what you can't
measure
How to track diversity
Once you’re able to track candidates’ progression throughout the hiring funnel, you can identify any drop-off points.
If there’s a specific question (or even an entire step of the process) that seems to be
disadvantaging a particular group, you’ll be able to spot and address this.
Some hiring processes in the U.S, for example, have been shown to prefer a masculine style of
leadership.
Unlike a marketing department, Return on Investment (ROI) is rarely measured in a talent acquisition team. When it is measured it is often linked to candidate volumes, rather than the actual performance of the hired candidates.
The ultimate feedback on your recruitment process is the performance review scores that hired candidates get 6-12 months after they have started. This is the holy grail but can be quite hard to implement in practice. However, there is a whole host of intermediate steps we recommend you take to understand the effectiveness of your process:
1. Ensure that you are using a data-driven assessment process. This enables you to compare like-for-like across candidates.
2. For each question or application chunk record the average, standard deviation and look at the distribution of scores. You want the average score to fall in the middle of the scale, with a nice normal distribution. 3. If using multiple reviewers, monitor the subjectivity of questions by calculating the average difference for review scores of the same answers. 4. Record interview conversion rates to understand if you are shortlisting effectively.
5. Compare scores between rounds to see if they predict future performance.
30
Appendix
6. Use data to prove
you're doing it right
12) Avivah WC., Havard Business Review (2014). In Search of a Less Sexist Hiring Process 1) D. Kahneman (2011). Thinking, Fast and Slow
2) Centre for Social Investigation at Nuffield College, University of Oxford (2019). Are employers in Britain discriminating against ethnic minorities?
3) M. Wood, J. Hales, S. Purdon, T. Sejersen & O. Hayllar, Department for Work and Pensions (2009). A test for racial discrimination in recruitment practice in British cities 4) M. Bertrand & S. Mullainathan, National Bureu of Economic Research (2003). Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination
5) S. Kang, K. Decelles, A. Tilcsik, S. Jun, Administrative Science Quarterly (2016). Whitened Resumes: Race and Self-Presentation in the Labor Market
6) L. Quillian, D. Pager, O. Hexel & A. H. Midtbøen, PNAS (2017). Meta-analysis of field experiments shows no change in racial discrimination in hiring over time
7) P. Forscher, C. K. Lai, J. R. Axt & C. R. Ebersole, Journal of Personality and Social Psychology (2019). A Meta-Analysis of Procedures to Change Implicit Measures
8) Multiple studies - CSI (2019), PSC & OCHRO (2017), S. Chowdhury & E. Ooi & R. Slonim (2017), M. Bertrand & S. Mullainathan (2003)
9) Schmidt, Frank. (2016). The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings 10) C. L. Exley & J. B. Kessler (2019). The Gender Gap in Self-Promotion
11) Workopolis (2015). How quickly do interviewers really make decisions?
