Academic Performance Implications

As detailed in Sections 7.3.1 to 7.3.3, it can be noted that the use of a pedagogical- PDE, such as SimplifIDE, has a number of beneficial effects on academic performance of novice programmers learning to program in an introductory programming course. These beneficial effects support the explicitly stated or implicit belief of textual pedagogical-PDEs6 developers that the alteration of the PDE affects the academic performance of novice programmers.

A number of implications regarding the observed effects in the current study are that: • low ability and high ability subjects do not derive the same benefit from a

PDE;

• programming skill transfer from computer based to non-computer based (pen- and-paper) environments occurs; and

• not all kinds of programming knowledge are equally supported.

6

(Mueller & Hosking 2003; Storey et al. 2003; Reis & Cartwright 2004; Chatley & Timbul 2005; Chen & Marx 2005; Rigby & Thompson 2005)

Academic performance improvement for low ability subjects is slight, while that for high ability subjects is significantly higher. If one assumes that high ability subjects have better problem solving skills than low ability subjects, then this finding supports the statement made by Chen and Marx (2005) that the difficulties experienced by novice programmers are as a result of poor problem solving skills, rather than the sophistication of the PDE. Conversely, it could be argued that the effects of a PDE supporting programming mechanics on subjects with good problem solving skills are substantial in terms of academic performance.

The transfer of programming skill from a computer based to a non-computer based environment is affected by the PDE used. It was found that program construction knowledge, in particular, was transferred to the pen-and-paper environment by both low and high ability subjects, although to varying degrees. There appears to be no significant transfer of program comprehension knowledge between the computer based and non-computer based environments.

A transfer in program comprehension knowledge would have been evidenced by significant differences in the academic performance variables that tested program comprehension explicitly, namely the multiple choice questions in the pen-and-paper assessments (ThMC and ExMC). The lack of apparent program comprehension knowledge transfer between computer based and non-computer based environments could be due to PDEs supporting programming mechanics not providing enough support for program comprehension tasks.

7.4 Perceptions

The perceptions of novice programmers can have an effect on their self-efficacy and hence on their performance, all things being equal (Zimmerman 1995; Jenkins 2001a, b). This section examines the results of the weekly questionnaires (Appendix F) that were administered to subjects regarding the practical learning activities and the practical assessment. Sections 7.4.1 to 7.4.3 examine the perceptions of the full complement, low ability and high ability subjects respectively. Section 7.4.4 discusses implications of the findings of the effects that the PDE has on the perceptions of novice programmers.

7.4.1 Full Complement

The perceptions of the full complement are considered in relation to the administered practical learning activities and practical assessment.

Practical Learning Activities

There is a significant difference in the perceived ease of use of the PDE for the full complement in the initial exposure to the PDE. Generally the treatment group found the designated PDE easier to use than the control group, but this was only significant in the very first practical learning activity. One can therefore conclude that overall a textual pedagogical-PDE, as discussed in the current study, is not perceived to be significantly more difficult to use than a commercial-PDE, except perhaps initially. This corresponds with the results of a survey conducted amongst a number of tertiary institutions in which it was found that the PDE used was not a large area of concern for students in an introductory programming course (Lahtinen et al. 2005).

(a) Paper-based Form (b) Subprogram Wizard

The full complement of subjects in the treatment group felt that they learnt more and achieved more in latter practical learning activities than the control group. Practical

learning activities subsequent to the introduction of procedures and functions (week 10) are perceived in this way. A possible explanation for this phenomenon is that:

• the paper-based subprogram form used during lecture sessions (Figure 7.1(b)) is very similar to the electronic wizard (Figure 7.1(a)); and

• the generation of subprogram skeleton code allowed subjects to concentrate on writing the subprogram body.

The paper-based subprogram forms (Figure 7.1(a)) were used frequently in lecture sessions to discuss how subprograms should be constructed, how one would decide whether the subprogram should be a function or a procedure, which parameters should be value or reference parameters and how a subprogram should be documented. The electronic version thereof was the Subprogram Wizard (Figure 7.1(b)) and has a very similar format to that of the paper-based form. Subjects would therefore be able to leverage existing knowledge and have it reinforced within the PDE. In addition to this, the wizard also generated the source code skeleton of the subprogram in the correct place within the program. Subjects were therefore not required to write as much, potentially error containing, code as the control group (Section 4.3.5). The perceptions of subjects regarding the practical assessment were not as varied as those of the practical learning activities.

Practical Assessment Perceptions

There were no significant differences in perceptions regarding the practical assessment for the full complement. In general, less of the subjects in the treatment group thought that they did not have enough time, that the questions were difficult, that they did badly in the assessment or that the PDE was difficult to use compared to the control group, but this was not significant. Therefore, the perceptions of subjects using the pedagogical-PDE were not significantly better than those using the professional-PDE under test conditions.

7.4.2 Low Ability Subjects

The perceptions of the low ability subjects are considered in relation to the administered practical learning activities and practical assessment.

Practical Learning Activities

Low ability subject perceptions followed the same general pattern as that of the full complement. Low ability subjects in the treatment group felt that they learnt more and achieved more in latter practical learning activities than those in the control group. Practical Assessment

Resembling the full complement in this regard, there were no significant differences in perceptions of low ability subjects regarding the practical assessment.

7.4.3 High Ability Subjects

The perceptions of the high ability subjects are considered in relation to the administered practical learning activities and practical assessment.

Practical Learning Activities

The pattern of high ability subject perceptions regarding the practical learning activities is similar to that of the full complement. High ability subjects in the treatment group felt that they achieved more in the latter practical learning activities.

The same conclusions regarding the perceptions of the high ability subjects regarding the practical learning activities can be drawn as for the full complement.

Practical Assessment

No significant differences were found for high ability subjects regarding perceptions of the practical assessment. It is interesting to note, however, that even though there is not a significant difference between the treatment group’s and control group’s perceptions of the practical assessment, there is a significant difference in their academic performance (Section 7.3). Since it would appear that there is no difference in self-efficacy (Jenkins 2001b) between the treatment and control group7, one can therefore conclude that the designated PDE was the overriding factor in the academic performance for the practical assessment. Therefore for high ability subjects, the PDE used appears to play a larger role in academic performance than self-efficacy for practical assessments.