Project Activities

The project is aimed to be developed using an iterative design process guided through the communication and interaction with the Development Teachers through the first two years of the grant. These teachers will play an integral role in develop-

ing LIVE-CHART and HAND-RAISE to best augment their teaching strategies and helping to promote positive help-seeking and responding behaviors amongst their students. The vignette of Ms. Kelly interacting with her students in a hypotheti- cal setting describes our initial design and use cases for the tool; as we described, however, the behaviors for which a teacher is notified as well as the actions made available to teachers will be selected and developed in a data-driven manner gained through interactions and feedback from the Development Teachers.

While we describe the intended timeline for the project and participants in Sec- tion C.2.a, we will provide greater detail as to the specific activities planned for the 20 Development Teachers over five three-month stages spanning just beyond the first year of the grant; the three-month timespan illustrates the intended short-term feedback loop intended for the project to promote faster development cycles that are able to effectively incorporate the information gained from the Development Teachers.

Stage 1. Jan, Feb, Mar: The goal of this initial stage is to begin to learn the types of behaviors that are most important to teachers as well as potential actions that can be taken as a result of observed behavior. The development of the LIVE-CHART tool, and subsequently HAND-RAISE, relies on teachers being able to effectively take action to help students become more engaged in the classroom with these tools helping teachers to recognize where such action is needed; this starts, however, with teachers helping to identify cases that are most actionable as well as potential actions that are likely to positively impact student confidence (e.g. being able to praise a student, such as Getting it Grace in the vignette, for doing well, particularly after struggling) as well as engagement (e.g. ensuring students, such as Gaming Ganji and Bored Billy, are on-task and practicing positive learning strategies).

the action-level clickstream data of their students and identifying what they would address in that data and how they would take action. ASSISTments already provides action-level reports to teachers, but we will provide an augmented version of such a report to the development teachers that includes additional detectors of student behavior (i.e. student gaming), and affective state (i.e. concentration, confusion, frustration, and boredom) to provide teachers with a breadth of information. The protocol that these teachers will be asked to follow is to be guided by a set of informal prompts to help facilitate helpful feedback. Such prompts will include asking the teacher “What, if anything, would you say to the student if he/she were present after looking at the data?” and “Is there any instance where you would praise the student for their work? Where?” as well as other such questions that will evolve as we gain more information from teachers.

Stage 2. Apr, May, Jun: During the second stage, the Development Teachers will continue to look at their students’ clickstream data and providing feedback on a nightly basis to continually help inform the types of behaviors that 1) commonly emerge, 2) are commonly identified as important, and 3) can be effectively addressed through clear actions. The type of data displayed to teachers during this collection process will be informed by their responses (particularly in response to the informal prompts). Information such as whether or not a student looks bored, for example, may not be as useful as other detectors of student behavior (or perhaps the reverse), in which case we can learn how to prioritize and select the types of student data on which to focus. It is important that LIVE-CHART, as intended as a real-time notification tool, is developed to be very selective of the types of notifications sent to teachers; it is important to not overwhelm the teacher with information about all students (20 simultaneous notifications occurring each second is likely neither useful nor practical for teachers), but also we do not want teachers to be constantly looking

at their device and ignoring what is happening outside the tool in the classroom; the tool should help provide information to teachers when it is most useful without consuming their complete attention.

At this stage in the development of the tool, the interface by which we collect data from teachers will also be updated to more closely resemble what will become the student-level display of LIVE-CHART as illustrated in Figure 3. As it is through such a display that teachers will be able to view and interpret recent student per- formance through LIVE-CHART in the classroom, we will also ask for feedback regarding the layout, type, and visualization of data to improve on how such data is represented and displayed to teachers.

Stage 3. Jul, Aug, Sep: As most, if not all, Development Teachers will likely not be using ASSISTments with students during summer months, they will be asked to look at past student data to continue to regularly provide feedback during de- velopment. By this stage, however, it is also the goal to provide the Development Teachers with an initial version of LIVE-CHART. This initial version will be de- signed to play back, in real time or at slightly faster speed, student data from a class period from the previous academic year. In this way, the prototype will simulate a real time classroom by playing back pre-recorded student log data and displaying notifications to teachers as if they were present in the class.

Use of this prototype will help to gain feedback on the types and frequency of student notifications through the system, the user interface, and will also be the first chance that the teachers will be able to provide feedback on classroom-level data. By looking at clickstream data of individual students, as was the case in the first two stages, it is likely easy for teachers to find something that is worth addressing and taking action within each student’s sequence of actions. By allowing the teacher to select which students to address from a classroom display (and limiting the displayed

actions up to that instant of the simulated class period), we can learn not only which types of notifications are important to teachers, but also when such notifications are important; the temporal information is likely just as important to consider when deciding what to present to teachers (e.g. it is likely unhelpful to notify a teacher of a student behavior multiple times in a short time span, but perhaps there are instances where this would be important).

Stage 4. Oct, Nov, Dec: The final stage of the first year of development is aimed at improving the prototype version of LIVE-CHART to allow for real-time functionality in real classrooms. Following the development cycle of stage 3 and subsequently the feedback gained from the Development Teachers during that time, it is the goal to provide such teachers with a version of the tool that can be used in their real classrooms during the first half of the academic year. The Development Teachers will be asked to use the prototype in their classrooms at least once per week and continue to look at pre-recorded class periods as had been done in stage 3 on nights where the tool had not been used. We will ask the teachers to, on the night following usage of the tool in their classrooms, follow the same procedure of looking at and providing feedback for the tool using pre-recorded class periods, but specifically replaying the class period where the tool had been used on the previous day; this will allow the teacher to provide feedback on the usage of the tool in the classroom, as it is unlikely that the teacher will have sufficient time during the class period to do so.

Stage 5. Jan, Feb, Mar (year 2): It is the goal of development to produce an initial version of the HAND-RAISE tool and begin implementing its functionality within the ASSISTments tutor and LIVE-CHART by the end of this stage. As the HAND-RAISE functionality is a focal point of the intervention described in this project, while facilitated through the real time functionality of LIVE-CHART,

it is important to allow teachers time to test the functionality and utility of the tool in real classroom settings. The initial version will allow students to select an option to raise their hands and articulate a question that is then sent to the teacher’s LIVE-CHART display. Allowing the teacher to be able to address such student questions is vital to the implementation of the intervention. Subsequent development on additional functionality, such as allowing teachers to direct questions to other students, is also planned to be implemented by the end of this stage. The goal is to allow the Development Teachers the opportunity to use all aspects of the tool and provide feedback on the usage (in addition to other design elements) before evaluating the tool with the Pilot Teachers during the subsequent academic year.

Timeline for Participants

The timeline for the Development Teachers and Pilot teachers is illustrated by Fig- ure 5 over the 3 year period of the grant. The timeline focuses early on the iterative development of the system using feedback from the Development Teachers as de- tailed in the previous section, while working toward the final pilot version of the intervention to be deployed to the Pilot Teachers in Fall of 2020. The Pilot Teach- ers will participate in several in-class live training/demonstration sessions that will occur at the beginning and end of the final two full academic years as will be de- tailed further in Section C.3. During the last academic year (2020-2021), the Pilot teachers will use LIVE-CHART and HAND-RAISE in their classrooms, allowing for final analyses and evaluation of the intervention during the final months of the grant period.