Measuring Learner Engagement

A number of tools and methods for assessing or measuring learner engagement have been adopted or developed (see, for example, Chapman, 2003; Fredricks & McColskey, 2012). The strengths and weaknesses of the main methods (self-report measures, experience sampling techniques, teacher ratings, interviews and observations) were reviewed by Fredricks and McColskey (2012). For example, one advantage to self-report instruments mentioned in Fredricks and McColskey (2012) is that they provide insight into students’ subjective perceptions that cannot be accessed via measures of behavioral engagement such as

attendance or homework completion rates. Citing Appleton, Christenson, Kim and Reschly (2006), the authors point out the methods such as observations and teaching rating scales would be highly inferential when looking at emotional or cognitive engagement. And although they are easy to administer and quite often the most practical means of measuring

engagement, self-report instruments include shortcomings in terms of accurate self-

representation and a tendency toward broadly-worded items (Fredricks & McColskey, 2012). Experience Sampling is described by Fredricks and McColskey (2012) as growing out of research by Csikszentmihalyi (1990) and his theory of “flow” states. In brief, this involves students being signaled several times during the day via pager or electronic device and then

completing a short questionnaire regarding their in-the-moment psychological states (Egbert, 2003; Shernoff, Csikszentmihalyi, Schneider, & Shernoff, 2003). Although this method suffers some of the same shortcomings as other self-report methods, it has the advantage of capturing impressions or perceptions as they occur and not upon reflection (Fredricks & McColskey, 2012).

Teacher checklists or ratings scales of student engagement were mentioned by Fredricks and McColskey (2012) as especially useful for younger children for whom self report instruments would be impractical. Hopkins (2008) sees the reliance on rating scales and checklists as stemming from the concern (mainly in North America) for more scientific

approaches to teaching and evaluation, while Chapman (2003) and Parsons and Taylor (2011) stress these types of instruments as helping to triangulate self-report data.

The major disadvantages of observations are that they are labor intensive, and they usually involve only a small number of students and contexts. This raises concerns about the generalizability to other settings. Finally, the quality of descriptive observations depends heavily on the skills of the observer and on his or her ability to capture and make sense of what was observed (Turner & Meyer, 2000).

One self-report survey instrument that has been developed and used for large-scale studies in the U.S. is the High School Survey of Student Engagement (HSSSE). In the words of Yazzie-Mintz and McCormick (2012):

There are 35 major items on the survey; including sub-questions, there are over 100 items to which students respond. Survey questions cover a wide range of aspects of engagement, including how students spend their time, the importance they place on particular activities, the rigor and challenge of classes, reasons for going to school, tendency toward boredom in school, potential for dropping out, and types of teaching they find engaging. The last question on the survey is

an open-response question, to which students can provide longer-form thoughts on their school experience. (p. 750)

Built on the foundation of the National Survey of Student Engagement (NSSE) for postsecondary students, the HSSSE survey originally measured similar constructs of engagement (Kuh, 2003). More recently, HSSSE has been a research and professional development project directed by the Center for Evaluation and Education Policy (CEEP) at Indiana University in Bloomington. From 2006 through 2010, more than 350,000 students in over 40 states took the survey.

Although student self-report instruments have dominated the landscape of learner engagement studies, Wimpenny and Savin-Baden (2013) note recent calls for a shift away from purely quantitative to qualitative or mixed-methods investigations. DeWaelsche (2015) used a combination of student surveys and focus group discussions with 42 university EFL students in Korea to explore student participation in student-centered, critical thinking activities in their English classes. They conclude that Korean students can overcome

sociocultural impediments (e.g., a reluctance to speak up in class or challenge each other) and engage in conversations and critical dialog with their peers if they have sufficient target- language skills. Target-language ability as it relates to learner engagement is not a central focus of my study but these findings suggest that I include it as a contributing factor in my conceptual model of learner engagement.

One purely qualitative study of disengagement (alienation) is reported in Case (2007). Using semi-structured interviews with third-year chemical engineering students in South Africa, Case (2007) identified six possible relationships for university students: (1) to one’s studies, (2) to the broader university life, (3) to home, (4) to career, (5) to one’s classmates, and (6) to the teacher. Their study found that most students identified with a small group of

students and that racially homogeneous groups of two to five members was the predominant pattern of school affinities, although students in this study expressed receptive attitudes to forming relationships with different racial groups. Case (2007) defines alienation as “the absence of a relationship that students might desire or expect to experience” (p. 119), and found that negative classroom experiences (drudgery, discipline, denial, lack of passion, lack of enjoyment) can lead to complete disengagement. These findings are key to my study in that I am also looking at how well the basic psychological need of relatedness is being met in university EFL classes in Japan.

Stepping back from the research, we can see the predominance of studies that develop and/or test one or more self-report instruments, with results subjected to some type of

inferential statistical analysis (e.g. structural equation modeling or factor analysis). Results from such studies have helped broaden and deepen our understanding of issues surrounding engagement as well as provided us with an expanding list of theoretical models. As with broader motivation research, however, in SLA we are still in need of more context-specific, qualitative studies of how engagement is manifested for learners, both individually and as a group. This, again, is one of the unique contributions that I hope to make through this experiential look at engagement for three groups of learners in my teaching context.

One other consideration in designing my study–in addition to finding measures that would help me answer the research questions–was to identify where on the grain size

continuum (mentioned above) I would focus, and the strenghts and weaknesses of my chosen measures for that conceptualization of engagement. In brief, I have chosen to look specifically at classroom engagement (meso level), but understand there are links to both school

engagement (macro) and task engagement (micro). These issues are discussed further in the next chapter (Research Design, section 4.3).

3.3.6 Engagement Research in L2 Learning Contexts

As mentioned earlier, SLA research has been dominated by the L2 motivation construct. Only recently do we see more direct references to engagement in SLA contexts. It should also be noted that there have been a range of conceptualizations and approaches to investigating learner engagement (Svalberg, 2009). Specifically related to corrective feedback, Ellis (2010) views learner engagement in terms of “how learners respond to the feedback they receive” (p. 342). In line with the core dimensions of engagement mentioned above, Ellis (2010) viewed cognitive, behavioral and affective perspectives as applicable to the study of both oral and written corrective feedback.

Philp and Duchesne (2016) see engagement as consisting of behavioral, cognitive and social dimensions, with emotion or affect related to motivation. They concur with Pekrun and Linnenbrink-Garcia (2012) that emotions are “intrinsic to rather than a facet of engagement” (p. 52 - italics from original), and influence the other dimensions of engagement in either activating or deactivating ways. Using this same framework, Lambert, Philp and Nakamura (2016) had thirty-two Japanese learners complete parallel versions of narrative tasks–one teacher generated and one learner generated–and then measured learner engagement in L2 use in terms of behavioral, cognitive, and social components. Results of their study indicated that tasks operating on learner-generated as opposed to teacher-generated content had positive effects on all aspects of engagement in L2 use during task performance. They further

concluded that learners were also more affectively engaged in the performance of the tasks in the learner-generated content condition than they were in those in the teacher-generated content condition.

In her review of how learner engagement has been conceived of and investigated, Svalberg (2017) found “considerable differences” in how the term is construed. The author was interested in how her concept of Engagement with Language (EWL)–discussed in

Svalberg (2009)–relates to other views on engagement. She concluded that EWL was part of task engagement, which was in turn part of contextual engagement (described as school engagement by Skinner, Kinderman & Furrer, 2009), and that both affective and social factors impact a learner’s willingness to engage. In relation to grain size (discussed above), she was interested in the micro level.

Philp and Duchesne (2016) analyzed interactions among L2 learners involved in a variety of classroom language learning tasks in an attempt to more explicitly define what is meant by engagement in these contexts. They conclude that similarly to work in educational psychology, engagement in these contexts is a multidimensional construct, with behavioral, cognitive, social and emotional dimensions operating interdependently. Questions they proposed for future research were:

- What are the processes by which engagement and language learning are linked?

- How do these processes vary in different contexts?

- How are the dimensions of cognitive, social, emotional, and behavioral engagement evident in various learning situations? How do the dimensions interact to influence learning? (p. 68)

This limited review of studies highlights first the need to clarify what is meant when talking about learner engagement in my context and other SLA settings. Second, there is substantial room for more detailed analysis of engagement as it is manifested in specific language learning episodes (micro) as well as a wider analysis of how engagement is

influenced by external factors such as workload, family or peers (macro). My approach will be to start at the meso level, focusing on influences of the classroom learning milieu.

3.4 THEORETICAL FRAMEWORKS FOR EXPLORING LEARNER