Methodology

Two questionnaires were designed for each phase of the study. One each for the trainers and trainees.

The Phase I questionnaires were based on a previous study carried out on paediatric recruitment.32

For the interviewers, five of the questions inquired about their social-demographics. They

were then asked to consider 20 statements about the interview process and to score them on a Likert scale of 1 (strongly disagree) to 6 (strongly agree). These statements were designed to assess the interviewers‟ perceptions of the validity, reliability, fairness and content of the MSI.

For the interviewees, six questions were asked about their demographics, two questions on

whether the candidates had any previous experiences of MSIs and how well informed they felt about the format prior to the interview. Eight questions, scored on a Likert scale of 1 (strongly disagree) to 6 (strongly agree), then asked the candidates how they found the format of the interviews, whether the questions were easy to understand, and how fair the candidates found this format when compared to a panel interview.

Responses were initially divided into two categories: agree or disagree, based on the Likert scale.Likert scale 1 to 3 indicated that the respondents disagreed with the statement, whereas Likert scale 4 to 6 indicated that the respondents agreed with the statement being suggested for each specific question. The six point Likert scale responses were then treated as a linear scale to obtain mean scores and 95% confidence intervals.

Phase II (a) and Phase II (b), were held nine months after the commencement of the Orthodontics Specialist Training. All the Trainers and Trainees were invited to participate in a questionnaire-based study to assess StR progression through training and to assess their perceptions of National Recruitment. The questionnaires for Phases IIa and IIb were based

51 | P a g e

on the responses made in a previous survey undertaken by COG of the British Orthodontic Society (BOS) in Autumn 2011 and one carried out by the TGG of BOS when National Recruitment was introduced. The free text comments were used to determine key areas of concern which were then used to formulate the themes and individual questions contributing to the questionnaires.

For Phase III, all Orthodontics StR intakes from 2011 and 2012 and their trainers were invited to participate in a questionnaire based study designed to evaluate their perception towards WBAs. The questionnaires were based on a previous study carried out in psychiatry to assess WBAs.24,25 The questionnaires were organised into six thematic frameworks.

Section A - questions were asked about demographic data and general information of trainers and trainees.

Section B - questions explored trainers‟ and trainees‟ opinions on why WBAs were introduced.

Subsequently, trainers and trainees were asked to indicate the extent to which they agreed with the statements posed in the following sections:

Section C - attitudes and perceptions of WBAs,

Section D - WBAs as an assessment tool and

Section E - overall perception about WBAs.

Trainers and trainees were asked to score their perceptions based on a five-point Likert scale for the last three sections, where 1 was strongly disagree, 2 - disagree, 3 - neither agree nor disagree, 4 - agree, and 5 - strongly agree.

Section F explored trainers‟ perception of themselves as an assessor or trainees‟ perception of their assessor. Trainers and trainees were asked to rate any given subject on a five-point scale, where 1 - poor, 2 - poor, 3 - acceptable, 4 - good, and 5 - very good.

52 | P a g e

For the descriptive statistical analysis of Phase II and Phase III, responses were divided into two categories; agree or disagree based on the Likert scale. Likert scale 1 and 2 indicated that the respondents disagreed with the statement, whereas Likert scale 4 and 5 indicated that the respondents agreed with the statement. Likert scale of 3 indicated that the respondents neither agree nor disagree with the statement.

A pilot study was conducted before sending out the Phase I, II and III questionnaires in order assess and then to improve the design of the questionnaires. The pilot study was carried out on consultants, orthodontics StRs and post-CSST trainees who attended the Merseyside Deanery Audit meeting in January 2013. The questionnaires were subsequently modified for the present study.

All the questionnaires were checked by the biostatistician (GB) before they were sent out. GB analyses the questionnaires to determine the Chronbach‟s Alpha to determine the questionnaire‟s internal consistency. He then deleted questions that did not contribute to this and to ensure the Cronbach‟s alpha was at least 0.7 which is the minimum level of acceptable reliability. This meant that all the questions in the specific sections were related to the themes that were set.

Invitations to participate in the surveys were distributed by email to potential participants (all current trainers and trainees in each Deanery) in August 2013. The secretaries of the Consultant Orthodontist Group (COG) and Training Grade Group (TGG) of the British Orthodontic Society (BOS) distributed the e-mails for the trainers and trainees respectively. The email sent by the secretary, of the relevant groups, contained a link to the online questionnaire via Survey MonkeyTM and an information sheet that described the study in detail.

Data collection was initially intended to be over the course of four weeks. However, the response rate was low and remained at less than 30% despite further reminder emails sent during the last two weeks of data collection. Therefore, the data collection period was

53 | P a g e

extended for another 4 weeks with weekly reminders being sent to the trainers and trainees encouraging them to respond to the questionnaire(s).