Threats to Validity

In this study we used a semi-structured interview that we specifically geared towards learning programming. We also primed participants to improve their programming skills. Our specific focus on learning may skew our results away from other factors that also affect decisions and perceptions of value. However, even when we explicitly encouraged participants to prioritize learning, participants were more likely to make their decisions based on their preferences and desire for challenge.

This study sought to identify the perceptions of value that novice programmers see in two specific instructional formats. While the specifics of these perceptions may not hold to other formats, our general conclusions about the factors that participants feel are important, like challenge, will likely hold for other instructional formats. However, outside of middle school children, these results would likely differ for other demographics, like expert programmers. We also note that participants with a stronger interest in programming may have self-selected to participate in this study.

5.6 Discussion

Below we discuss how participants’ responses and behavior lend further support for code puzzle completion problems as a motivating and independent learning resource.

Choice & Learning Prior work has found that giving learners full control over their

instructional task choices can lead to both positive and negative outcomes [92, 202]. Overall, we believe the participants in this study made reasonable choices that helped improve their programming knowledge. This suggests that for independent contexts, our approach of letting

participants opt to complete code puzzle completion problems alongside working towards their own goals, will likely benefit learners.

While participants probably benefited from their choices, their selection of instructional tasks and formats was suboptimal for maximum learning gains. Participants often made choices that were motivated by enjoyment rather than learning. When participants did make decisions about improving their programming skills, they often chose tutorials, as they perceived tutorials as more supportive of learning. Yet after completing a code puzzle, participants frequently cited them as more useful for improving their programming skills. The mixed responses from participants suggest that novices lack the expertise needed to efficiently self-regulate their learning choices [204].

Fortunately, self-regulation is itself an important skill for independent learners to acquire [42]. Through the process of contrasting and leveraging tutorials and puzzles throughout the study, participants may have indirectly improved their self-regulating ability for future decisions. During the evaluation, a majority of participants made choices based on trying to discover which tasks and instructional formats were appropriate for their skill level. This suggests that participants were attempting to self-regulate for tasks that align with their current expertise. Over time, learners’ ability to self-regulate between tutorials and puzzles will likely improve, thus further supporting that novices have the capacity to make effective use of code puzzle completion problems in independent contexts.

Motivation & Challenge In our first study comparing tutorials and code puzzles (Chap-

ter 3) we found no differences in motivation between the two instructional formats. From this evaluation, we found that participants are frequently motivated by things that they enjoy. Not only did participants enjoy the animations of the instructional tasks, but they had fun completing the puzzles. Further, participants opted to complete more code puzzles than

tutorials throughout this evaluation which lends further evidence to support that code puzzle completion problems are motivating to independent learners.

Beyond simple enjoyment, participants often stated that it was challenge that motivated them. In fact, almost every participant made a decision during the study where they were seeking challenge. Reassuringly, participants found the code puzzles to be more challenging than the tutorials.

Lastly, the extra motivation provided by challenge in the code puzzles may explain the difference in task completion time between tutorials and puzzles in the study. In our first study, we found that puzzles took significantly less time to complete than tutorials. We suspect that the opposite result in this study is due to participants purposefully choosing puzzles which would challenge them. Challenging puzzles likely took longer to complete, thus possibility reducing learning efficiency. However, it is not clear how self-imposed challenge affects learners’ cognitive load. It is possible that self-imposed challenge in code puzzle completion problems may increase germane cognitive load thereby increasing programming knowledge while simultaneously motivating learners.

5.7 Conclusion

In this chapter, we presented a qualitative study into validating Hypothesis III by investigating which instructional formats users preferred and why they chose to use them. From this evaluation we learned that independent learners’ care about more than just an enjoyable activity. We discovered that enjoyment, challenge, and perceived value all play important roles in a learner’s decision to choose between tutorials and code puzzles. The results of this study suggest that code puzzle completion problems are viewed as a useful resources by independent learners that also complement and support their goals. But, a key feature of

this support is that it is balanced against other informal learning resources, like tutorials. In the next chapter, I discuss what role code puzzle completion problems may have in the future and how they might further support independent learners.