Programme Effectiveness Interviews

Interviews were conducted with faculty members who have taught students at the undergraduate level who have completed the PYP, as well as students who have not done so. The interview started by reminding the interviewees of their anonymity, as well as their right to withdraw from the interview or request the withdrawal of their interview data from the study at any point. The subject and aim of the study as a whole and the interviews specifically were explained. The primary question guiding the interviews was: ‘Comparing graduates of the preparatory year that you have taught

with students who did not attend the preparatory year, is there a noticeable difference between the two groups?’

Next were questions about specific skills and knowledge to identify in detail the differences between the two groups. As such, interviewees were selected to represent both male and female genders.

The interviewees were asked the primary question as an open question at first and then were asked to elaborate on their responses without explicit guidance from the interviewer. This was important to avoid the interviewer guiding their answers and thereby potentially missing unexpected aspects.

Once the interviewee has responded to the primary question freely, he or she will be asked to elaborate on any observed differences in the following respects:

‘In light of the answer to this question, please explain the observed differences in the following areas: English language skills, Computer use, Cooperative learning/teamwork ability/constructive dialogue, Decision-making/problem- solving/response to constructive criticism, Conscious interaction with the university environment and the local environment, Work ethic and commitment to attendance, Identifying their abilities, potential and independence, Being accustomed to structured learning and planning and Ability to quickly understand learning topics’

Next were the following questions: ‘Is either group easier to deal with? Are the differences because of what the students studied in the preparatory year, or are there other factors at play?’ The interview concluded with the following question: ‘Is there any additional information you deem appropriate?’

3.12 Summary

The present chapter outlined the methodological approach to data collection and analysis used to address the RQs. The chapter was split into four other parts. Each addressed one of the four RQs because the methodological treatment of each research question, as well as the data collection instruments used, differed significantly. These four sections each included subsections describing the data collection instrument and study populations, including any exclusion criteria, the methodological strategy to be taken, and descriptions of items in the case of the surveys and their underlying constructs. The ethical considerations were discussed in a subsection of the introduction, as the ethical considerations applied across the different parts of the study

because the data collection instruments, which were surveys and semi-structured interviews, differed in content rather than form.

The Mixed methods concurrent triangulation strategy was used in this study to collect quantitative and qualitative data at the same time, which was then analysed separately to compare or combine the results of each.

Based on the literature review and study questions, the items used in each survey concerned outcome and process indicators, as opposed to input and output indicators. This approach was taken with consideration for the assertions in the literature that these indicators have been neglected in higher education, despite their importance for effectiveness at the course, faculty, institution and national levels (Chalmers 2008a, 2010). The ethical considerations involved in the present study were relatively straightforward. That is, there were no minor children or vulnerable groups involved, so all participants were able to understand the description of the survey and give informed consent.

The student survey explored the predictive values of admissions criteria, with comparisons between track and gender groups of the study population. This concerned RQ1 (What is the predictive value of the admission criteria for the preparatory year programme at the University of Taibah, for each track and each gender within each track?) explicitly. The ultimate objective was to suggest relative weightings for the different admissions criteria analysed by track and by gender groups within each track. As pointed out in detail, this dissection into subgroups was important; both as the skills required for success in different tracks, as well as the curricular contents differ, and co-mingling the data would ignore such differences and result in a predictive model that fits poorly for some subgroups. Also, as discussed in the literature review, gender was of particular importance in the case of Saudi Arabia because males and females are taught separately; therefore, any inherent differences between genders may be magnified. This marked a difference from the popular approaches to such research. A correlation study using Pearson’s correlation coefficient was performed to explore the correlations between GPA and HSG, Capabilities Test and Achievement Test for each track (health sciences, natural sciences, and social studies) and each gender.

This student questionnaire and associated semi-structured interviews (n=8) concerned the effectiveness of the preparatory year programme from the perspective of students. This provided an explicit answer to RQ2 (How effective is the Preparatory Year Programme from the perspective of students for each track and each gender within each track?). First, the data collection tools were explained, along with the study population and participant selection. The items concerning each of the underlying constructs of academic content effectiveness, trainers’ effectiveness, assessment methods, academic advising, university environment and achievement of objectives for the PYP were described and justified for inclusion. Eight students were chosen to participate in the interviews. The means score and factor score for each item or factor was calculated. Then, ANOVA analysis was applied to reveal statistically significant (p<.05) differences in factor scores across the six track/gender combinations of females in each of the tracks, and males in each of the branch).

The trainer questionnaire and interviews (n=4) concerned the effectiveness of the preparatory year programme from the perspective of trainers. This addressed RQ3 (How effective is the Preparatory Year Programme from the perspective of trainers in each academic subject?). First, the data collection tools were explained, along with the study population and participant selection. Similarly, to the student questionnaire, items concerning each of the underlying constructs of academic content effectiveness, academic advising, university environment, and achievement of objectives for the PYP were described and justified for inclusion. Four trainers were chosen to participate in the interviews. As with the methodology for RQ2, the means score and factor score for each item and factor were calculated. Then, ANOVA analysis was applied to reveal statistically significant (p<.05) differences in factor scores across the six track/gender combinations of females in each of the branch, and males in each of the branch). Finally, the questionnaire concerning the effectiveness of the preparatory year from the perspective of undergraduate faculty members examined the effectiveness of the preparatory year programme from the perspective of faculty members asked to compare the performance of students who have completed the PYP and those who have not. This answered RQ4 (How effective is the preparatory year programme in terms of improving undergraduate performance of students who went through the programme compared to students who did not?). This part of the study differed

significantly from the student and trainer-based data, as the distinction between process and outcome indicators was somewhat blurred, as discussed previously.

Chapter 4 Quantitative Results