Constructs and Items

As discussed in the literature review, the items included in the survey and, thence, the interviews were loosely based on the work on quality systems and indicators of learning and teaching by Chalmers and others, which has gaining traction as an international standard for evaluating higher education at the national, institutional, departmental, and programme level (e.g., Chalmers 2010; Chalmers, Lee and Walker 2008).

The thirteen categories of process indicators identified in the national survey of Australian practice (Chalmers and Thomson 2008) included mission, vision, and objectives. As in the student questionnaire, these were covered in the trainer questionnaire regarding pedagogical outcomes, as well as outcomes in line with the principles of the Saudi higher education directives discussed in the Introduction and literature review of the present paper. They also included “Teaching and Learning Indicators”, which were covered in the survey sections on academic contents and in the “Theme II” section of the survey on the relationship between academic subjects and goals of the programme from the trainers’ perspective. The categories also included “Organisational Unit Review”, including Disciplines, Divisions, Faculties, Schools and Centres, which, as mentioned in the context of the student survey, remained central to the present study, as the data from respondents were separated according to the different tracks and branches of the PYP. Another category,

“Curriculum Review” including units, unit sets, programmes, was covered in the academic contents section of the surveys.

The literature review chapter contained a more detailed discussion of these trends in assessment. Generally, the use of process indicators was increasingly seen as essential to identify areas of weakness and strength in education, including higher education, which was evidenced by the relatively strong performance of countries where such practices have been adopted and the prevalence of recommendations to that effect in countries such as Ireland, Australia, and other OECD countries. This was the main motive behind the focus on such indicators in the present study and the non-inclusion of many commonly used, typically resource-based, input and output indicators. The following items were used to assess trainers’ assessment of academic contents: (1) Contents of preparatory year courses suit the needs of students, (2) It is easy to identify the scientific objectives of the contents, (3) Contents are consistent and coherent, (4) Content Units are suitable in terms of length, (5) Activities in contents help to achieve learning goals, (6) Contents encourage students to interact with the trainer in light of educational material, (7) Contents contain a series of exercises and problems that develop students’ thinking skills, (8) Contents are relevant to intended purposes, (9) The organization and arrangement of the content suitable to students’ needs, (10) Contents take continuity into account, (11) Language of contents is clear and error-free, (12) Textbooks are appropriate and well organized.

As with the student questionnaire, numerous items used here were based on the QAA, which identified a number of intellect and transferable skills in their guidelines. As noted, they regarded them as essential to develop at all levels of higher education. They defined intellect skills as analysis, synthesis, evaluation and problem-solving, as reflected in items 2, 3, and 7. Items 1, 8 and 11 were also indicative of the development of students’ intellect skills. It is implicit that contents suit the needs of students so that such skills are fostered, being the general tools required for success at the undergraduate level and beyond. Item 7 spoke explicitly to communication and teamwork skills; Item 8 concerned the skills of synthesis and analysis. Items 4, 9, 10 and 12 concerned inherently important practical aspects of delivering course content,

The role of the trainer was very limited in most assessment methods that were used, including setting questions for the different tests, making corrections and the exam schedule. Therefore, it was important to know the views of the trainers about the evaluation process in the programme. Trainers were asked to evaluate the assessment methods, and the following seven items were used: (1) Exam question types are appropriate for students, (2) Test questions cover content of the subjects, (3) Different methods of evaluation were used, (4) Students get grades in a timely manner, (5) Testing atmosphere is comfortable, (6) Time of the tests is appropriate for students, and (7) Students know the dates of the tests and the location of halls at an appropriate time.

The following items were used to assess trainer's opinion of academic advising: (1) Academic advice is effective and useful, (2) Academic advisors are continuously available to respond to trainers’ questions, (3) Academic advisors communicate with trainers individually and collectively, (4) Academic advisors visit the trainers inside the classroom, (5) Trainers find great support from the academic advisor.

This was a relatively novel section, concerning interaction between the trainers and the academic advisors. Such data had not been collected in similar studies on preparatory year programmes, and not explicitly included in the common higher education data collection instruments discussed in this chapter and the literature review. However, an orchestrated approach to individual students that capitalized on the mutually complimentary roles of trainer and academic advisor was of clear benefit, as reflected in various other constructs in the literature. For example, Chalmers, in her detailed meta-review of HE performance indicators, included a “Student Engagement” indicator, explicitly citing “student interactions with faculty members” and “supportive campus environment” as objectives to be measured as part of this indicator. As mentioned previously, in the version in Gonyea et al. (2003), “Experiences with faculty” is given a single dedicated section (p. 154). In addition, in the US and Canadian National Survey of Student Engagement (Chen et al. 2009), student discussions with faculty members and academic advisors were each given dedicated constructs or items. Given the specialized, that is, programme-specific nature of this section, only five items were used, as in the corollary section of the

student survey. Again, similar to the CSEQ, the items focussed on the quality and breadth of interaction.

The following items were used to assess trainers’ opinions of the university environment: (1) Classrooms are comfortable and clean and of appropriate size, (2) The university environment has high health standards, such as lighting and ventilation, (3) Devices within classrooms operate efficiently in most cases, (4) Laboratories are properly equipped, (5) Computer labs are sufficient and appropriate, (6) The library provides adequate sources and references, (7) The library provides appropriate places for viewing and reading, (8) The library provides adequate and sufficient computers for reading and research, (9) Stadiums and halls are available for practicing sports and recreational activities, (10) It is easy to use the electronic registration system of the university, (11) In case of any problem using the electronic registration system, it is easy to communicate with technical support and (12) Food and beverages facilities provide appropriate services and meet the needs of users.

As with the student questionnaire, several items were included in this section based on responses in the pilot survey and interviews, such as items 9, 10 and 11. Similar items or constructs were used in other literature, including the CSEQ, NSSE (Chen et al., 2009) and the work of the OECD by Chalmers and others. They were selected for inclusion in this case based on the feedback and comments during the pilot study. Moreover, again, the practical elements considered in the 12 items used in this section of the present student questionnaire had to cover the students’ experiences with the library (items, 6, 7 and 8) and lab environments and equipment (items 4, 5 and 6), each of which received a dedicated section in the CSEQ (Gonyea et al. 2003).

The following items were used to assess the extent of achieving the objectives of the PYP: (1) The preparatory year programme contributes to deepening the Islamic and national identity through the curriculum and student activities, (2) The programme provides a well-developed course with high quality standards, (3) The programme directs students to the appropriate college given their abilities and skills, (4) The programme promotes the outstanding academic performance of the students, and (5) The programme provides a high-quality learning environment to improve the outcome

The objectives of the PYP considered in the present study were discussed in detail in the introduction and literature review sections, and the items in this section of the questionnaire reflect these more or less explicitly. However, they also have close equivalents among a broad range of process and outcome indicators, including the CSEQ Gonyea et al. 2003), NSSE (Chen et al., 2009), Chalmers (2008b) and Chalmers and Thomson (2008). However, as with the identical section in the student questionnaire, in the case of the present study, the target outcomes of the programme were determined by the stated objectives and the fact that the programme was designed for preparation for higher education studies. Thus, item 1 was included, despite its lack of universalizability in HE or preparatory year evaluations.

Finally, for investigating the relationship between academic subjects and goals of the preparatory year from the perspective of trainers, the respondents were asked to indicate the extent of the link between the courses they teach and the goals of the PYP from their points of view. The trainers were asked to indicate their classes taught from a table identical to that in Theme II of the student questionnaire. Note again that the five response categories used in this part were “Very Strongly Related” (1), “Strongly Related” (2), “Moderately Related” (3), “Slightly Related” (4), and “Very Slightly Related” (5). As discussed concerning the corollary section in the student questionnaire, it was also decided in this case that point five on the scale would be “Very slightly related” rather than “Not related” in order to give symmetry to the scale as much as possible, as recommended by Norman (2010); moreover, it was felt to be unlikely that any trainer would consider any of these broad items completely unrelated to the subjects they taught, since they, as with the students, were instructed to select only the items that were or should be related.

3.11 The Effectiveness of the Preparatory Year from the Perspective of