Interview Training

1. Unconscious bias

What is unconscious bias: unconscious bias (also known as implicit bias) is a bias that happens automatically, is outside our control, and is triggered by our brain making quick judgments and assessments of people and situations, influenced by our background, cultural environment and personal experiences.

Unconscious biases cause us to make a snap judgment of a candidate within the first 10 seconds of meeting them. That doesn’t just refer to interviewing. It can also occur when you’re dealing with a customer or a peer. We use these unconscious biases to inform ourselves of decisions we’re making. When conducting an interview, for example, we can make a snap judgment and spend the rest of the interview reinforcing those biases that we have. Imagine you are interviewing somebody and you see that they went to the same school as you did or they like the same sports team that you do. You may unconsciously feel a natural affinity towards that person because they look like you or remind you of you. In that scenario, you may spend the rest of the interview looking for signs to reaffirm that this candidate is the right fit for the job. In reality, it could be an affinity bias. As part of this training, we will discuss some ways to mitigate that bias and ensure that we’re giving everybody a fair shot at getting a job, regardless of whether or not they like the same sports team as you.

So in interviewing, the question becomes: how do we prevent them from impacting our hiring decisions? In other words, how do we rely on data and facts to make decisions, rather than our gut?

1. The first way that we can prevent unconscious bias is to take the Harvard implicit bias test. We strongly encourage all our teammates to take this test if you’re interested in learning more about your personal biases. It’s okay that we all have these, but awareness of them should work towards reducing them to the best of our ability.
2. The second way to prevent unconscious bias is to consider likeability. We tend to want to hire people who remind us of ourselves. Make sure you’re asking yourself, does the person I’m interviewing have the innate will, skill, and values to succeed in the role? We should not hire based on whether they grew up in the same place, have the same hobbies, etc. We really should not be leveraging those things to make a decision one way or another.
3. Finally, stick to an interview script. Our hiring managers work hard at the beginning of every process to create an interview plan. An interview plan is a structured interview process that every candidate goes through regardless of background and it ensures that they are all asked the exact same questions to level the playing field. Asking those structured interview questions helps to reduce bias from affecting our decision making.

2. Structured interviewing

Research has shown, time and time again, that establishing a consistent set of interview questions and a structured process not only helps us hire people that are going to be more successful in the role, but it actually speeds up the process in which we can make a hiring decision. Building a structured interview plan aligns us on what we’re looking for early on, allowing us to make a decision and push candidates through the process faster.

All of our questions are written in a behavioral-based, open-ended format, and all candidates are asked the exact same questions. The best way to create an open ended question is to just put, “tell me” in front of every single question you ask. For example, “tell me a time when, tell me how, or tell me why.” It is important to ensure they’re behavioral based questions that are specific about past experiences. This helps us predict future behavior. At Sourcegraph, we should try to avoid any hypothetical questions.

All teammates at Sourcegraph leverage a recording tool called Brighthire to standardize our process and reduce bias. This video explains the benefits of BrightHire. We believe that candidates should be assessed based on their merits, not our memories. BrightHire allows us to recall key details, revise initial impressions, and replace implicit bias with explicit evidence. BrightHire rides along in your phone and Zoom interviews with an interactive notepad. All you need is the Chrome extension and Zoom and you’re all set. It allows us to pull all of those structured questions that our hiring manager created right into your Zoom meeting.

Please see BrightHire frequently asked questions here.

3. Leading a legal interview

Another reason we have pre-determined, structured interview questions is because it helps us avoid asking illegal questions. As an interviewer, all questions provided to you in your Greenhouse interview scorecard are written in a legal fashion. However, we understand that sometimes conversations can get off track or there are situations where you need to ask follow up questions to get to the root of an answer, so we want you to be aware of the legal rules. Please review federal EEO laws, your state and country’s employment laws, and your internal counsel when necessary.

What questions are illegal?

Example questions such as “Do you have children?,” “What does your partner do for a living?” or even “what are your hobbies” or “what do you like to do in your spare time” are all topics that could unintentionally create an illegal situation.

So where do we draw the line? A common interview question to ask is “tell me about yourself and walk me through your background.” If a candidate openly shares personal information that relates to the “topics to avoid” list below, the best thing to say is, “That’s interesting, thank you for sharing.” Then, redirect the conversation back to topics that directly relate to a candidate’s ability to do the job. You can ask about candidates and their background, so long as it deals with their ability to perform the duties of the role.

FAQ:

• Can we ask, “what is your current compensation?” No, and this is something that we absolutely want to avoid. There are laws in most places around the world now that make it illegal for employers to ask candidates what they’re currently making or what they’ve made in the past. This is in an effort to reduce the gender pay gap. Rather than asking what candidates make, we should ensure that we have ranges defined when we post a job. When we launch our search, we should create a level and identify a range so that we establish salary expectations with all candidates at the beginning of the process. Rather than ask what they’ve earned, the recruiters should ensure that they fall within our range. Nobody other than the recruiters should ask about compensation because the Talent Team is equipped and trained to ask this in a legal way. If the recruiters pass somebody to the interview process, it can be assumed that they have already covered compensation.
• We cannot ask candidates where they are from. It doesn’t have anything to do with their performance on the job. If we ask that and ultimately do not hire them, they could claim that they were discriminated against.

What questions ARE legal?

To ensure that your question is phrased in a legal way, always ask yourself “does my question directly relate to the candidate’s ability to perform the duties required and succeed in the role?” If you can confidently say “yes,” then feel free to ask!

Please review federal EEO laws, your state’s employment laws, and your internal counsel when necessary.

The process

These are some pillar steps that every single candidate goes through, regardless of which position they’re interviewing for.

• The recruiter screen is where we talk about compensation and conduct an overall qualification assessment to make sure that candidates are going to be a strong fit based on our alignment with the hiring manager.
• We then transition to a hiring manager screen. This is normally 30 minutes. If we’re doing a deep dive, sometimes it’s 60 minutes.
• We then transition candidates to an assessment for some roles, but not all. The talent team will work with the hiring manager to determine if there’s an assessment stage and will depend on them to build it out.
• Next is the in-depth interview stage. This is not standardized for all roles. One of the things we’re rolling out in a kind of beta testing with some hiring managers is what’s known as a resume deep dive or a topgrade. Some studies have shown that this reduces a mishire rate of over 50% down to 10%. I’ll train people who are interested and then once we’ve tested it enough internally we can determine if it makes sense to have that be mandatory across all interviews at Sourcegraph.
• You will see that our Values Interview is mandatory and every single candidate, regardless of level, is going to go through it. Historically, we have assessed candidates to ensure that they’re aligned with our values, but it’s been ad hoc and in different ways depending on the hiring manager and the role. So we are standardizing the interview moving forward. Every candidate will be assessed for values in the exact same way, with the exact same questions.

Logistics: running an interview

When you get scheduled for any interview, you are going to receive a calendar invitation.

• Please accept the calendar invite to indicate that you can attend.
• If you cannot make the interview, please decline the calendar invite and message whoever sent you the invitation to let them know.
• Very important: if you decline the same day as the interview, please slack the #hiring channel as well and we’ll find a replacement or reschedule immediately.

How to prepare for your interview:

• Click on the Greenhouse scorecard in the notes section of your calendar invite. Here, you can review:
  • The candidate’s resume
  • The interview prep section, including the job description
  • The scorecard, containing the structured interview questions that you will ask during the interview

How to kick off your interview:

• Launch BrightHire from Google Calendar by clicking the “Interview with BrightHire” button. It will pull up both the Zoom recording as well as your Chrome extension which will guide you through your structured questions.

This is the ideal structure for all interviews:

• Step 1: Develop rapport through introductions and set the interview agenda
• Step 2: Ask behavioral-based questions
• Step 3: Leave time to answer questions, build rapport, and sell the candidate on Sourcegraph

Please make sure to start your interviews with an agenda and end with rapport building and selling. The middle is the time for screening and asking those structured questions. The agenda goes a long way with candidates so that they know what’s coming and they can prepare. At the end, remind the candidates that interviewing is a two way street and we want them to feel like they have all their questions answered. This will require you to pace yourself, watch the clock, and make sure you leave enough time at the end. When the structured questions go to the very end and you catch yourself only allowing two minutes for questions, it’s unfair to the candidate. They feel like they’re not getting the questions they need answered in order to make the best decision for themselves.

Beginning of interview - review agenda & build rapport:

As we all know, interviewing can be stressful. The priority is to develop rapport, get the candidate at ease, and make them feel comfortable. We should ask them how they’re doing, make sure they have a glass of water, and answer any initial questions they have. We always want to introduce ourselves and set an agenda. You could start by saying, “Here is what I hope to accomplish during the next 30 minutes. For 20 minutes, I will ask you some behavioral-based questions. Then, I will leave 5 minutes at the end to answer any questions that you have.”

Middle of interview - ask structured questions:

Interviewers should never be going into an interview not knowing the questions that they’re supposed to be asking. All interviewers should check their scorecard ahead of time to make sure they have access to their questions.

End of interview - answer questions & rapport build:

At the end of the interview, be prepared to honestly answer any questions the candidate has and then go back to rapport building and selling. Regardless of whether or not you intend to recommend that candidate for hire, you should still establish rapport and leave them on a high note feeling good. If you know that this is the best candidate you’ve talked to and you are super excited to move them forward, go into selling mode. You could say, “I’m so glad to have met you and I’m happy to be a resource as you go through the process and have any questions. It’s a great place to work.” As a team, we will develop a guide of all of the selling points and how we can consistently sell Sourcegraph. We will create bullets that you can leverage to sell our company, depending on the scenario.

The only time that selling can go awry is if we give any indication that they’re going to get an offer, and then they don’t get an offer. We need to be careful that when we’re selling, we’re not saying things like, “when you join us” or “I’m so excited to work with you.” Instead, we should say, “here is why Sourcegraph is so great” or “here is why I’m so excited to be here and these are some reasons why I decided to join Sourcegraph.”

All questions that we ask should start with “Tell me…” to ensure they are open-ended and prevent a “yes” or “no” answer. We use the interview funnel for each question, as follows: we start with an open ended question, narrow down to a specific area and listen for how the candidate goes about it, discover the candidate’s motivations, listen for the candidate’s achievements, then summarize the answer back to the candidate and seek agreement (see chart).

Best practices:

• We may need to dig deeper and drill down to determine whether candidates embody a specific value, but interviewers can still phrase questions as “tell me”
• Remind candidates that they can pause if they need to think of an example
• If you notice that a candidate is getting nervous, it’s okay to say, “take a breath, take a pause, it is just as important for you to ask me questions and it is for me to ask you”

If you ask a candidate to tell you a problem that they identified and then took the initiative to solve, and the candidate says, “the problem was that things were falling through the cracks” and then stops—obviously we’re not going to be able to walk away from that answer thinking that they nailed it or failed it. Interviewers are going to need to dig deeper. Think about using the “tell me about” phrasing as a way to drill down to get to the root of the answer. To follow up on that answer, you could say, “tell me, how did you identify that things were falling through the cracks? Tell me, why was that a problem? Tell me, what did you do to fix that problem? Tell me, what initiative did you take?”

Whatever it takes to drill down to actually get an answer that allows you to confidently determine that they either embody a particular value or they do not. Those follow up questions are not scripted and it’s where interviewers will need to be mindful of the legal training.

It is your responsibility to make the candidate feel comfortable and remind them that this is not a marathon. They should take a pause and think about their answer. We’ve all been a candidate, too, and sometimes you just want to answer everything and not take your time to think about it. If you notice that a candidate is getting nervous or starting to slip up on their words, it is perfectly fine to say, “I want you to take a breath and take a pause. This is just a conversation, and it is equally as important for you to ask me questions and get to know me. So please take a second and feel free to regroup.” Often candidates are able to decompress and calm down and they end up doing great.

Providing feedback

• Please provide feedback as soon as possible, ideally no later than 24 hours after the interview, so that we can move quickly and ensure a positive candidate experience. You can use Brighthire to review the interview and copy / paste notes at a later time, but please try your best to get in the habit of submitting feedback directly after the interview since recall reduces by 44% after one hour. Feedback is best when you use the attributes to help score responses.
• Most importantly, if another teammate interviewed at the same time as you or directly before you, please don’t ask for their assessment of the candidate until you have officially filled out your scorecard. Asking for their opinion is a surefire way to introduce bias into your decision making. In an effort to reduce bias, we intentionally prevent teammates from viewing past scorecards in Greenhouse prior to their interview (with the exception of Hiring Managers and Executives). Interviewers should feel conviction in their process and assessment of the candidate. If we notice that some candidate has gotten a couple yeses and a no, we will bring the team together to talk about it and determine why that happened. We can leverage BrightHire again here to see if there was a miscommunication or if their story was inconsistent. BrightHire will allow us to dig deeper if the panel did not agree on a decision.
• Please leverage the definitions in this interview feedback overall recommendation guide when rating candidates in Greenhouse. It’s natural to want to make hiring decisions based on comparison points. As you talk to more people, you can start to understand where certain candidates fall in comparison. That being said, we should avoid the “shopping effect”—where we feel like we need to talk to five or 10 candidates in order to make a hiring decision. Sometimes it’s the first candidate we talk to that’s the best fit, and by the time we’ve talked to all of the other folks, we’ve lost that person.

There are plenty of companies that pull interviewers into a room and stack rank candidates. We want to avoid this to the best of our abilities. Rather than comparing candidates together in a room, ask yourself a binary question, “can this candidate do the job, yes or no?” Then consider where this candidate would fall in terms of being an example to others. It’s difficult because our brains are wired to compare, which introduces bias because we start to trend towards likability and/or recency bias.

Ultimately, it’s ideal if you can interview two or three people and you are a yes on all of them against the clear guidelines and questions that we’ve provided. It’s ultimately up to the hiring manager to make the final call as to who they’re going to be moving forward with. Our goal is to put in structure to prevent that bias and establish definitions so we’re all speaking the same language.

Lastly, be mindful about what you leave in your interview form. There should be nothing illegal, like “diversity hire.” Candidates have the right to request access to this information. All written feedback should follow the same set of rules and guidelines that were addressed in our interview question do’s and don’ts. A good rule of thumb, again, is to think, “does what I’m about to write directly relate to their ability to do the job?”

Candidate Experience

We’ve all been interviewed before and every experience is different. Whether or not you get the job, you’ll always remember how you were treated by the company. 83% of candidates say that an unimpressive interview can make them change their mind about a potential employer. Since job seekers are empowered to be selective when considering opportunities, the experience they have during the recruitment process is crucial.

It’s important to ensure that any candidate who interviews with Sourcegraph has a top notch experience. Even if the candidate doesn’t get the job, they will more than likely share their experience with others. They may leave a review on a job site like Glassdoor or perhaps refer their friends and family to Sourcegraph. This means that you need to respect the candidate’s time, clearly explain the position, build a connection, provide everything they need to make an informed decision, and ensure they feel excited and confident about the role and Sourcegraph.

Candidates are understandably nervous and the interview process can be really stressful. Your attitude should convey that everything you do is to help them succeed and not fail. Creating a five star experience requires you to make sure your candidate’s needs are met at every step of the hiring process. Conveying a consistent message about the value of your company and leading with kindness in every interaction is critical to Sourcegraph’s hiring brand.

We want to make sure that every single candidate who interviews with us, regardless of hiring outcome, feels good. Be sure to provide them with an experience where they was to go tell their friends, their family, their networks and say, “despite the fact that I didn’t get hired, Sourcegraph seems like an incredible company, I felt super respected throughout the process, and it was the best interview experience that I’ve ever had, bar none.”

We use an anonymous candidate survey through Greenhouse to gather data and feedback on our candidate experience and recruiting process. Click here to see more information about the Greenhouse candidate survey.

Every candidate is going to get surveyed with these questions, which are automatically sent 48 hours after we have either hired or rejected them. We are able to pull reports to see where we’re hitting the mark and where we’re missing the mark, which, of course we’re going to take immediate action to rectify and put processes in place to make it better.

Any questions? Please do not hesitate to reach out to us in the #hiring Slack channel.

Need practice? Please reach out to @hiring and we can either mock interview or lead an interview and you can shadow.