In this chapter, we reviewed the related work on employment interviews and video resumes. We first presented the taxonomy and communicative functions of nonverbal behavior, and dis- cussed the state-of-the-art methods to automatically extract nonverbal cues, which will serve as a basis for obtaining an accurate representation of the displayed behavior of protagonists in employment interviews. We then discussed the use of verbal behavior for the inference of social variables, as well as its possible representations. Employment interviews have been studied for decades by psychology researchers, and several relationships between nonverbal behavior and hirability impressions have been established. This important body of related work is useful as it provides us with hints on what nonverbal cues are associated with hiring decisions, therefore it serves as a catalog for the behavioral cues to be extracted. In social com- puting, relatively few studies have investigated employment interviews, despite their ubiquity in the personnel selection process. To our knowledge, our work constitutes the first aiming at inferring hirability impressions automatically. The related work in social computing focusing 18
2.7. Conclusion
on the inference of social constructs in face-to-face interactions is however sufficiently similar to our research problem to provide good insight in terms of general framework, cue extraction, and inference methods. In the remaining of this thesis, we present our approach to analyze employment interviews and video resumes.
3
Collection of employment interviews
One of the objectives of this thesis is to investigate the formation of hirability impressions in employment interviews. To address this broad research problem, we collected the SONVB em- ployment interview dataset, a corpus comprising 62 recordings of real employment interviews, where participants were applying for a paid marketing job. This data collection was motivated by the lack of available datasets including a job interview scenario. In this chapter, we present the experimental design and the technical setup used for this data collection. This dataset was collected within the framework of the SONVB (Sensing Organizational NonVerbal Behavior) project1, a collaboration between Prof. Daniel Gatica-Perez (Idiap Research Institute), Prof. Marianne Schmid-Mast (University of Neuchâtel), Prof. Tanzeem Choudhury (Cornell Univer- sity), and their respective research teams. The data collection was done in collaboration with Denise Frauendorfer (PhD student at University of Neuchâtel). A condensed version of this chapter was originally published in [97].
3.1 Scenario
Participants were recruited by advertising a part-time research assistant job opening in a social psychology lab. The position was a marketing job, where the hired applicants were expected to convince people on the street to participate to psychology experiments, and the job was paid 200CHF (208USD) for four hours of effective work. No specific requirement was set other than fluency in French, but applicants were expected to have strong communication, persuasion, conscientiousness, and stress resistance skills. The job offer was advertised on classified ads web platforms of two Swiss universities, and fliers were disseminated in three Swiss universities. Due to the large participation of students (90% Bachelor and Master students, 4.8% PhD students, 3.2% employed), the average age was 24 years (st d=5.68 years). The gender was somewhat unbalanced: 45 females (72.5%) and 17 males (27.5%).
A consent form was completed by participants upon arrival at the sensing lab, installed by Idiap Research Institute at the University of Neuchâtel. The informed consent form included
Chapter 3. Collection of employment interviews
In this corpus, we used a structured behavioral design, meaning that each interview followed the same structure and that some questions were related to applicant past experiences in specific situations. The sequence of questions is listed below:
1. Short self-presentation.
2. Motivation for applying to the job.
3. Importance of scientific research (which is the field of the job). 4. Past experience where communication skills were required. 5. Past experience where persuasion skills were required. 6. Past experience of conscientious/serious work. 7. Past experience where stress was correctly managed. 8. Strong/weak points about self.
Questions 4-7 are behavioral and were used to assess four hirability measures (communication, persuasion, conscience, stress resistance). Specifically, they were coded based on the quality of the applicant answers to these questions. One additional hirability measure (hiring decision) was coded on the whole interview sequence.
Figure 3.1 – Interview structure and hirability annotations.
the following points: (1) the interview was audio- and video-recorded; (2) data could only be used for research purposes; (3) data distribution outside the project team was not permitted for data privacy reasons; (4) videos, snapshots, or audio snippets could be used as demos or included in publications upon specific request; (5) participants could withdraw their consent at any time. Once signed, applicants were given a copy of the consent form. All applicants agreed to give their consent.
Applicants were then asked to complete a series of psychometric questionnaires. The question- naires used in this data collection are presented in Section 3.3. The interview was designed as a structured behavioral interview,structuredmeaning that the interview strictly followed the same sequence of questions, ensuring that comparisons could be made between candidates, andbehavioralimplying that some questions were related to applicant past experiences in specific situations, eliciting a wide variety of behavioral responses. The psychology literature suggests that structured behavioral interviews are among the most valid tools for selecting applicants [69]. The interview structure is detailed in Figure 3.1. It included four behavioral questions, related to the specific skills required for the job, namely communication, persua- sion, conscientiousness, and stress resistance, as well as standard job interview questions (self-presentation, motivation to apply for the job, strong/weak points about self ). The in- terviews were dyadic: the interviewer and the applicant were seated at both ends of a table (see Figure 3.2). All interviews were conducted by the same person, the doctoral student in organizational psychology at University of Neuchâtel. The average interview duration was∼11 minutes; in total, the dataset comprises 670 minutes of recording.