Knowing how to ask insightful interview questions is a great skill. But knowing how to accurately evaluate the responses to those questions is an even better skill. Here’s what I mean.
Consider the question “Can black hole evaporation be reconciled with quantum mechanics?” I know it’s a bit out there, but I’ve been told (by people who know) that it’s a great physics question. I would appear highly intelligent if I were to ask it of a bunch of physicists at a cocktail party. Unfortunately, I don’t know the answer (I’m not sure physicists do either), which limits my ability to evaluate and assess any responses.
I’m not able to differentiate between a great answer, a
plausible answer, and a terrible answer. And I might even get sucked into believing a completely implausible answer just because it’s delivered in an eloquent and confident manner.
(Much like the impressive and convincing demeanor the majority of job candidates assume during an interview.) I’m not saying that knowing the answer in advance is a prerequisite for every query you make. There are plenty of times when it’s perfectly OK to ask a question if you’re unsure, or even clueless, as to the answer. However, interviewing job candidates is when you definitely want guidelines on what good and bad answers sound like before you ask the question. This is a tool that you want to put in the hands of every member of your hiring team.
The concept of using Answer Guidelines as part of the hiring process is revolutionary in the world of hiring. In fact, Leadership IQ is the only group I know of that teaches this technique. You can find a lot of people who want to teach you how to create interview questions (albeit incorrectly). But
there’s nobody else out there teaching how to create an answer key to those questions so you can accurately and consistently score the responses you get.
Let me share with you a recent experience with a Leadership IQ client that reinforces the need for Answer Guidelines. To give you some context, this was a large chemical company with lots of employees, many of them engineers. (I am being
deliberately vague because I don’t want them identified.) Leadership IQ had been engaged to conduct a Brown Shorts project for this client. At the point when this story takes place, we’d completed the Discovery process and identified their Brown Shorts. We wrote a set of Brown Shorts Interview Questions and created custom Answer Guidelines to score the candidate responses to those questions. All these materials had been reviewed and approved by the organization’s executive team. So we were just entering the final phase of the project, the point where we go in and train all their hiring managers on how to implement these tools. (It’s all well and good to have great tools. However, if you don’t know how to use them …) But before we started the formal training, I conducted a little experiment. As you’ll see, it’s an easy exercise. I encourage you to try it out with your own hiring team. First, I shared with the group the Brown Shorts Interview Question that had been developed during Discovery. “Could you tell me about a time when you didn’t know how to do something that a customer was asking you to do?”
Everybody loved the question. But despite the group’s enthusiasm for it, I felt certain that there were a number of different ideas floating around regarding what a good answer would sound like. I continued the exercise by asking the group to silently read an actual answer that had been documented during a recent interview. I didn’t want any second party vocal inflection or phrasing to influence individual thinking. And I asked all the managers to refrain from any discussion after they were done reading it.
Then, based on what they had just read, I asked this group to rate the candidate with a view toward how well that person fit their organizational culture. They were instructed to use a numerical seven-point rating scale, where only the endpoints were labeled with 1 for “Poor Fit” and 7 for “Great Fit.” (See Figure 4.1.) There were no rating definitions given for points 2 through 6. I’ll explain why the rating scale was designed this way in Chapter 5.
Figure 4.1: The Seven-Point Scale
Just to review, here’s the Brown Shorts question again: “Could you tell me about a time when you didn’t know how to do something that a customer was asking you to do?” And here’s the answer that the candidate gave:
At my last job I was inundated with requests that were outside my area of expertise or influence. I am always pretty cautious when it comes to stepping outside my comfort zone, so most of the time I just turned the situation over to someone more experienced. After all, I want to make sure I’m protecting the company’s back because I don’t want to touch a project for which I’m unqualified and then have it do damage to the client.
The client’s interests are always of paramount importance. And it’s critical that an engineer adhere to accepted practices and the proper processes. If I’m in a situation where I don’t know those processes, it’s better for me to pass the request to someone else that does.
The managers participating in this exercise struggled to rate this candidate. The final numbers showed something really interesting; the scores ranged from 1 to 7, with everything in between. In a room of 50 managers, every number on that seven-point scale was chosen by at least two people. And remember, these managers were all employed by the same company, they all shared similar professional backgrounds, and
they all worked for the same group of executives. But despite all these similarities, their assessment of the candidate showed that my initial suspicion had been right—every person in that room had a wildly different idea about what a good (or bad) answer sounded like. And did they ever become animated when I shared that fact with them!
I overheard one discussion between two guys sitting at the same table. I was told that their offices are located next door to each other and that they’d been good friends for years. Their conversation went something like this:
Manager A: “What’s the matter with you? How could you score that answer a 6? This candidate would be a total failure here.
He’s all wrong for us!”
Manager B: “Sure, that’s what you say. Because why would we want an engineer who actually followed a protocol!”
I asked the folks who gave the candidate a high score to tell me some of the things they liked about the answer:
“This person is really focused on the customer. And you can tell extra care is being taken not to do anything that might damage our reputation with the customer.”
“This is someone who appreciates the proper engineering mind-set.”
“This sounds like someone who wants to protect the company.”
“I would much rather see someone be cautious than reckless in this kind of situation.”
Then I asked the folks who skewered the candidate what they didn’t like:
“We’re a company that always has to find solutions, no matter what; and this person just gives up the second something becomes what he or she considers too hard.”
“I don’t like the ‘inundated with requests’ part. It makes me think that just about every request this person gets is going to be outside his or her comfort zone.”
“I was really turned off by the admission of ‘I just turned the situation over to someone else.’ Also, there was zero mention of keeping the customer in the loop.”
“I can handle the bringing in somebody else with more expertise part, but what bothers me is that I see no initiative to learn those new skills so that next time this person does know what to do and doesn’t have to rely on outside help.”
The results of this exercise highlight two big problems that organizations face when they Hire For Attitude without using Answer Guidelines. First off, you can usually find something you like, and something you dislike, in virtually every person you interview. (Of course, given that the consequences of hiring a bad attitude are worse than not hiring a good attitude, I’m more concerned about the former.) So without having some foundation to orient us and to tell us what good and bad answers sound like, it’s awfully hard to evaluate candidates consistently and correctly.
The second big problem is the extent to which everybody involved in your hiring process does (or does not) understand your Brown Shorts. It may seem absurd, but there are a lot of people in your organization, including leaders, who don’t know, or can’t articulate, what makes your culture special. Similarly, they can’t clearly tell you what separates your high and low performers.
This problem is not exclusively related to Brown Shorts. We conducted a study on whether employees understand their company’s strategy. Using data from Leadership IQ’s employee engagement survey, we assessed more than 70,000 employees on the extent to which they felt they could clearly articulate their organization’s goals for the year. Only 34 percent felt they could clearly articulate those goals. And it gets worse because next we took part of that 34 percent and asked them to go ahead and actually articulate the goals. According to the supervisors who graded their answers, only about half of them
really knew the goals. This left us with only 17 percent of employees who could correctly articulate their organization’s goals. And while it’s possible that some people actually could articulate the goals but rated themselves low because they didn’t feel confident, my experience tells me that’s not a large percentage of employees.
How are you supposed to achieve a strategy when nobody on your team knows what that strategy is? It reminds me of the old joke: the bad news is we’re lost, but the good news is that we’re making great time.
Strategies, Brown Shorts, Mission Statements—they’re all susceptible to wrong interpretation. I recently gave a speech at a hospital’s leadership retreat about how to translate your strategy into the front lines. After I finished, the CEO was so pumped up by what I’d said that he pulled $1,000 cash out of his pocket and offered it to the first manager who could correctly write down the company Mission Statement. Guess how many were able to do it?
Out of the 150 leaders in the room, not one answered correctly.
Nobody got the $1,000, and the really sad part is that the company’s Mission Statement is only 12 words long. As a reference, the Pledge of Allegiance contains 31 words. The real kicker, though, is that their Mission Statement is printed on the back of the name badge worn by every employee. But these folks were at an off-site retreat so they weren’t wearing their badges.
Too often we take our Brown Shorts for granted. “Well of
course we know who we are,” we say, along with “and I can tell you exactly what differentiates our high and low performers.”
But it’s always useful to see to what extent everybody agrees about that. If you went around the table at your next executive team meeting and asked each person to list the top three characteristics that differentiate high and low performers at your company, would each person have the same answer?
What if you conducted this exercise at your next manager meeting? Or at an employee town hall?
The hiring managers at the chemical company where I
conducted this exercise certainly didn’t use the same high and low performer characteristics to rate their candidate. But that inconsistency wasn’t because they weren’t living and breathing their Brown Shorts every single day. Rather, it was because they hadn’t yet learned to distill their Brown Shorts so they could explicitly say “These are the five characteristics that predict success or failure and that will let us measure every candidate accordingly.”
Two of the numerous Brown Shorts we identified during that chemical company’s Discovery stood out (and are especially appropriate for driving this all home).
You take ownership of problems—even if you’re not the one who will ultimately fix it, you shepherd the process until it’s resolved.
You’re a self-directed learner—you take full
responsibility for growing and developing your skills, and while you may not learn everything, you’re in a constant state of growth.
Even if you lack any other knowledge about the organization, you now know that these two characteristics drive the success of its best people. You can easily assess the answer to a question such as “Tell me about a time when you didn’t know how to do something that a boss or customer was asking you to do.” And you can easily recognize that the sample answer I passed around that day clearly revealed that the candidate in question was a poor cultural fit. There just wasn’t much
ownership, self-directed learning, or desire for personal growth in that answer.
Hiring for Attitude requires both your Brown Shorts and your Brown Shorts Interview Questions. But in order to make it all work, you also need your Brown Shorts Answer Guidelines.
That way, when you (and every member of your hiring team) are in the middle of a live interview, you’ll know exactly what you should be listening for and how you should react when you hear it.
Let’s get started creating your Answer Guidelines.