Student Perception of
Educational Best Practice
Utilization in
Introductory Computer
Science Classes
UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL HONORS
THESIS
JILLIAN TROFTGRUBEN
Table of Contents
Abstract...3
Introduction...3
Quizzes...5
Live Coding...8
Relatable Assignments and Examples...10
Peer Learning...13
Interactive Activities...15
Change in Activity...17
Methodology...19
Results...24
Discussion...27
Appendix...31
Participation...31
Best Practice Logs...31
Pivot Tables...32
Introduction to Programming Section 1...32
Introduction to Programming Section 2...33
Introduction to Programming Recitation...34
Foundations of Programming...35
Abstract
Lecture-based approaches to teaching introductory computer science classes have been
proven to not be as effective as many other pedagogical approaches. Active learning, however,
has been shown to be very effective but often is difficult to scale and takes professors a
substantial amount of time to prepare for each class. This study focuses on five active learning
techniques that can all be integrated into any lecture-style classroom in order to improve
student engagement, increase inclusivity and community, and improve long-term learning. The
techniques include quizzes, active coding, peer learning, relatable examples, and interactive
activities. Other changes in activity during lecture that are not explicitly one of the five
mentioned above are also tracked. This study monitors two introductory programming classes
taught at the University of North Carolina at Chapel Hill to determine how commonly these
active learning techniques are currently being included in lecture-style classrooms. To monitor
classroom activity, the students used a web tool that assists in determining which active
learning techniques the students perceive to be most helpful and in providing professors with
better insight on how to improve their lectures. The results were ultimately inconclusive.
Introduction
Studies have proven that traditional didactic lectures are an ineffective teaching style
[1]. While many governing bodies have pushed for active learning techniques to be
implemented, professors still primarily rely on traditional didactic lectures. Active learning is a
engaging learning environment. Active learning often focuses on the application of concepts
and allows for deeper learning. In a recent study of over 500 STEM faculty at 25 universities in
North America teaching over 2000 classes, fifty-five percent of instructors report relying on
lecturing for 80% or more of class time [1]. Professors often point to a lack of preparation time
as one of the main reasons why they do not integrate other teaching methods [2]. Additionally,
professors often cannot cover as much material when utilizing active learning.
Lecturing still can be an effective teaching method when it is properly utilized to
organize and deliver content [2]. It is often most effective, though, when it is combined with
active learning techniques. This study primarily focuses on five ways active learning can be
implemented into lectures. The five methods include giving quizzes as formative assessments,
live coding, peer learning, providing relatable examples or assignments, and interactive
activities during lectures. These techniques can overlap and be used in conjunction with one
another. If, for example, a professor gives a group quiz to solve a cooking problem using a
concept recently learned during lecture, this utilizes peer learning, relatable examples, and
quizzes as active learning techniques. This study also tracks when changes occur in the
classroom to refocus student’s attention. The goal of this study is to show the impact of active
learning techniques on students’ evaluation of introductory computer science courses at the
University of North Carolina at Chapel Hill. These techniques will be evaluated based on student
reviews after each class. The students rate the course based on their overall impression,
understanding of the material, sense of community, engagement, and interest. When
professors utilize more active learning techniques, student ratings for the course generally
Quizzes
Quizzes provide consistent feedback to both students and professors. There are many
types of quizzes including written, online, clickers, graded, ungraded, anonymous, announced,
and pop quizzes. Each of these is proven to benefit learning, thus all will be included in the quiz
category.
First, quizzes allow professors to identify common errors in students’ work and provide
immediate clarification and follow up about more difficult topics. Tutoring is commonly known
as the most effective way to teach as it provides immediate feedback and correction for
students [3]. In a large classroom setting, however, it is infeasible for professors to provide
individual feedback for students during each lecture. By requiring quizzes and iterating based
on the responses, professors can identify what teaching style works best for the majority of the
students and can help to clarify common errors that occur during the quizzes. Quizzes also
allow professors to provide positive feedback to students who understand the material and to
alert students who are falling behind of the errors they are making or which topics they do not
fully understand. Students can additionally seek out individual help from professors and
teacher’s assistants based on questions they missed to provide more of a tutoring-style learning
environment.
In addition to providing professors with feedback about student performance, quizzes
also promote better attendance in classes. Attendance has been shown to significantly impact
both exam grades and assignment grades. In one study of introductory computer science
courses, attendance accounted for 6% variance in exam scores and a 20% variance in
instructors about assignments, understanding lecture material beyond the book, and more
opportunities for collaboration [4]. Quizzes provide a strong incentive for students to attend
class. In one study about extra credit quizzes, class attendance increased by 10% when quizzes
were given [5]. In another study about weekly quizzes, class attendance increased from 70% to
80% [6]. By providing any form of quizzes, students are motivated to attend and, thus, learning
is greatly benefited.
Quizzes also greatly help students’ study habits. By requiring quizzes, students must also
study throughout the course rather than cramming for the exams. This leads to better
long-term retention of the material. Learning over time is commonly referred to as the “spacing
effect,” which requires students to repeatedly study a little bit over a large period of time. This
phenomenon has been researched for more than 100 years and many studies point to the same
conclusion: that cramming works for short term learning but does not work for long-term
retention [7]. In one study at Santa Clara University in California, researchers looked at the
effects of cramming for a Principles of Marketing course. They found that students who cram
typically have lower GPA’s. Also, while cramming did not have a significant impact on
immediate exam scores, when tested later, students who crammed for the exam scored
significantly lower than students who studied over time [7]. In another study, students were
split into two groups and were required to study flash cards. One group studied one set of flash
cards over time while the other studied smaller piles of flash cards fewer times. In the end,
spacing was more effective for 90% of the participants even though 72% of students thought
topics that they will be required to use throughout their coding career. Therefore, since
long-term learning should be prioritized, quizzes should as well.
In addition to requiring students to study over a longer period of time, quizzes also
require students to actively retrieve information and apply concepts, producing better
retention of the material. In one study, two groups of students were given identical time to
learn the same material. In one group, however, the students practiced retrieving the
information multiple times, while the other group practiced retrieving the information just
once. The group that practiced retrieving information more frequently performed significantly
better on the final exam even with the absence of feedback [9]. When continuous feedback is
given, student’s ability to retrieve correct information increases in the future. Thus, in a
classroom environment, where the total length of the semester is constant, quizzes requiring
continuous retrieval of information will be beneficial for overall performance.
Finally, quizzes have often been shown to reduce test anxiety on final exams. While this
greatly varies based on the nature of the quizzes, quizzes are often worth far less than a final
exam and thus create a less-threatening learning environment for students prior to the exam
[10]. Quizzes help combat test anxiety because they help students feel more prepared and
understand the types of questions they will be asked in an exam [11]. Having several small
quizzes instead of just one large exam can dilute the pressure and negative impact of a poor
performance if a student has a bad day [12]. In one study, for example, students were given
pre-lecture quizzes before three exams. While the quizzes did not make their anxiety decrease
anxiety varies greatly for each student; however, quizzes can help relieve some of the pressure
students feel during exams and make them feel more prepared going in to the final exam.
Live Coding
A computer science education builds skills such as computational thinking and problem
solving [13]. Live coding is one way to help students hone these abilities when they are first
learning how to program. Live coding occurs when professors write code, typically from scratch,
in front of students during lecture. This is beneficial as it allows students to see the full process
of coding from how to approach a problem to how to debug code. It also teaches students
about good programming practices such as coding style. While it does have drawbacks, such as
that it can slow down lectures, it is very beneficial for teaching long-term coding best practices.
One benefit of live coding is that it allows students to understand how to approach a
programming problem. For many students, they have never approached a subject like
computer science. While students often have strong problem-solving skills and feel like they can
solve the problem, they cannot formalize a solution using a computing language [14]. As a
result, students often feel anxious and discouraged when starting computer science [15]. To
combat their feeling of being unable to formalize a solution, many students now are reverting
to a “cut and paste” approach for coding, meaning they pattern match rather than think
through the problem when they begin. This style of learning leads to many misconceptions
about what coding is and what will be expected in future coding classes [14]. By using live
coding, professors show students their thought process on how to approach complex computer
science programs. They not only break down the problem, but they also show students how
concepts into working code that can be applied to many problems [16]. This helps students
apply these concepts in the future to a wide variety of programming assignments without
copying and pasting solutions.
Live coding also helps students understand the processing of debugging. When students
first begin to code, they often feel discouraged because their code does not work on the first
try. They believe their first attempt should look like the pre-written code samples given in
lecture; however, this is far from the truth with any coding assignment. Students begin to doubt
their coding skills and begin to feel inferior [16]. Live coding helps to combat this misconception
by showing that even professors must debug their code. It is very rare that, even if professors
have written the code prior to lecture, they will perfectly type the solution. Professors,
therefore must debug the code in front of students. They will often use techniques such as
adding print statements to find little errors, using the debugger to find bigger errors, and
deciphering error statements to help pinpoint where the problem may be. Even if the code is
perfect, professors can also intentionally include errors in the code to have students participate
in the debugging process. In a study at the Colorado School of Mines, a group of students were
presented with live coding during lecture and another group was presented with static code.
When the professor introduced errors in the lecture code, students who were used to live
coding were able to grasp the information faster and point out the errors. These students also
performed better on post-course survey which tested the student’s ability to differentiate
assignment vs equivalence in a code snippet [17].
Another benefit of live coding is that many students prefer it over static code examples,
cited that live coding was more engaging and “increased interest in programming” [16]. In
another study at the University of South Florida, 60% of students in one class and 30% in
another class strongly agreed that live solutions were more useful than seeing the final
solution, and no student in either class found the just seeing the final solution to be more
helpful [14]. Finally, in the Colorado School of Mines study, several factors led to the conclusion
that live coding was beneficial. First, the grade distribution of students in both the live coding
group and the controlled group were statistically similar with the exception of the final project,
where the live coding group scored statistically better. Second, as stated above, the students in
the live coding group scored better on the post-course exit question which tested reference vs
assignment statements. Finally, 90% of students who experienced live coding found the coding
examples to be more educational than presentation slides, whereas only 67% of the static code
control group found coding examples more educational [17]. Live coding has benefits that are
also difficult to quantify such as code style. When professors are coding, they often point out
language-specific style best practices such as correct capitalization, naming conventions, and
variable placement. Students also gain insight on how to explain their thought process for
coding interviews and are exposed more to white boarding code examples. Overall, live coding
is a very effective pedagogical approach to teaching coding in introductory computer science
classes and is often preferred by students to static coding examples given in lectures.
Relatable Assignments and Examples
Relatable examples and assignments connect real world experiences and applications in
computer science. This is also commonly referred to as context-based learning because it
the concepts are important. While context-based learning usually only refers to direct
application of concepts to real-world examples, analogies will also be included in this section, as
they still provide a framework for students to apply concepts. This technique is very broad and
can be utilized in many ways including using pop culture references during lecture, assigning
case studies about how the concepts are applied in the corporate world, and creating
assignments that allow students to build a product or feature they are already familiar with.
This pedagogical technique is also subjective, as what relates to one student may not relate to
another. However, the more that professors can relate to their students, the more students will
engage with the material.
Using relatable examples can improve student’s attitude towards learning. In a sample
of nine studies that focus on context-based learning in science, seven of the studies show that
context based learning improves student’s attitude towards science. In three of these studies,
they added statistical analysis and, in all three, context-based was statistically significant in
improving student’s perception of science with a significance level of .05 [18]. Relatable
examples improve attitudes because they increase the students’ motivation to learn the
material. According to the Vanderbilt University Teaching Guide, students who are intrinsically
motivated have a “fascination with the subject, a sense of its relevance to life and the world, a
sense of accomplishment in mastering it, and a sense of calling to it,” while extrinsically
motivated students are driven by “parental expectations, expectations of other trusted role
models, earning potential of a course of study, and grades” [19]. Relatable examples specifically
Students who are intrinsically motivated about the subject want to learn the material, so they
often have a better attitude.
Additionally, relatable examples and assignments increase class engagement. According
to James Middleton, if students see the activity as interesting, they will be engaged. If students
are not interested, then they evaluate the activity based on whether or not the activity is
stimulating and provides personal control. This means the activity must be challenging and
spark curiosity while not being too difficult [20]. By using relatable examples, professors can
increase student’s interest to maintain engagement. In one study, Dutch Computer Science
teachers were interviewed about their experience with context-based learning, and all teachers
mentioned an increase in engagement [21]. Increased engagement has also been cited in many
other studies that implemented context-based learning in science classrooms [22]. While
relatable examples have sometimes shown to have no effect on engagement, it rarely has a
negative effect. It has, therefore, been concluded that relatable examples and assignments can
increases engagement by sparking curiosity in subjects that students may not be initially
interested in.
Increasing student’s intrinsic motivation to learn a subject through relatable examples
can also sometimes improve student’s understanding of the material. While there is not
conclusive evidence that understanding is improved, it has been concluded that understanding
Is not adversely affected by context-based learning. In a collection of twelve context-based
learning studies conducted in science classes, four studies indicate a better understanding of
the material while only one noted a decrease in understanding [22]. If given multiple examples,
context-based learning actually allowed students to perceive the world differently [21]. This is
because students may see a concept being used in a way they are unfamiliar with and, thus, can
comprehend a wider use for a concept or solution.
Peer Learning
Peer learning occurs when students work together to learn the lecture material. There
are many ways in which professors can incorporate peer learning in their course such as
student presentations, small student-led discussion groups, and pair programming. Peer
learning is scalable, as the professor does not need to be involved with every group, and it
provides a smaller atmosphere for students in a large classroom setting.
Peer learning helps promote gender balance in the classroom by making female
students feel more comfortable asking questions. In a survey of 3128 students at 2 different
universities in 17 courses at all different levels, it was found that in almost all courses, women
were less likely to ask questions in class and often preferred getting help from TAs and peers
rather than professors. Male students, on the other hand, felt equally comfortable asking
questions to the professors as to their peers. When women do not ask questions during class,
they are often left with doubts which affect their deeper understanding of the material [23].
Peer learning allows women to ask for help from their classmates and get immediate answers
during lectures. This increases their confidence in future lectures as they feel more comfortable
with the foundational material.
Additionally, Peer learning improves grades, student retention, and depth of student
understanding of the material. Professors who implement peer learning often have lower
of splitting students into groups and having them answer questions together. This study took
place over ten years analyzing four courses. It found that peer instruction decreased failure rate
by an average of 67% [24]. In another study that also utilized peer instruction, students scored
an average of 5.7% higher on the final exam than students who had only had traditional lecture
practices [25]. By improving grades, students are less likely to fail out, and thus retention rates
improve [24]. When students work together to solve problems, they learn from each other.
They not only ask each other questions, but also discuss possible solutions, verbalize any
confusion, and clarify concepts. When students explain concepts, they realize where the gaps
are in their knowledge. If, for example, one student asks a question and no one can explain the
answer, this indicates a gap in the group’s knowledge that professors can fill in. According to
BYU Center for Teaching and Learning, explaining concepts to peers can also strengthen
memory because it creates a social connection made to the material [26]. When questions are
asked in the future about a topic a student had to explain, the student will remember the
situation and be able to easily recall the information. When working in groups, students also
hear multiple explanations, which can deepen learning. There are several advantages to having
information represented in multiple ways including that students can piece together the
different explanations to find one that works best for them and that hearing multiple
explanations can help students avoid misinterpretation [27]. Hearing multiple explanations or
multiple ways to solve a problem also allows students to find more efficient solutions in the
future [28]. One down side to peer learning, especially peer programming, is that students who
work together will often have the same solution and may rely on one another to solve the
often leads to a deeper understanding of the code as they typically will work together to find a
better solution. It also mimics what occurs in the work place as most code written in industry
relies on multiple team members solving the problem together.
Interactive Activities
Adding interactive activities also greatly improves students learning. Interactive
activities occur when students participate during lecture. This could include a group discussion,
a game, or any other way the students and the professor can interact during the class. The goal
of these activities is to allow students to apply what they just learned during lecture and to ask
clarifying questions to better understand the course material.
Interactive lectures lead to improved faculty-student engagement. Faculty-student
engagement is a major factor that determines a big part of a student’s educational experience.
One study showed that an increase in faculty-student engagement causes students to feel more
fulfilled by their educational experience. This study also showed that faculty-student
engagement had a “positive impact on both cognitive and affective student development” [29].
Another study showed that there is a correlation between positive student outcomes such as
improved GPA, institution commitment, self-confidence, and leadership skills when professors
and students interact more [29]. While student-faculty relations are important, many students
feel intimidated by professors and, thus, they do not feel comfortable approaching professors.
In a study of several student focus groups, students noted that they are more willing to
approach professors who utilize interactive teaching styles. Students also feel more
comfortable attending office hours and speaking up in class [30]. While other factors such as
comfortable seeking help outside of the classroom, interactive lectures are one way professors
can connect with students to improve student-faculty engagement.
Students also have a better perception of classes that implement interactive lectures. In
one study, professors implemented different one-minute interactive activities such as
brainstorming sessions, open discussions, and problem sets during lecture. The study found
that students felt the interactive lectures were more effective than traditional lectures.
Students also stated that interactive lectures were more engaging and reported having fewer
distractions in class. Finally, students were more confident with the lecture material [2].
Another study that integrated a similar engagement tactic showed that interactive lectures led
to more students being interested in the subject matter compared to traditional lecture styles
[31]. This is most likely due to the fact that students feel more connected to the material than
in traditional lecture styles. By immediately applying techniques, students gain a better
understanding of the concepts so they feel more confident.
Finally, interactive lectures promote diversity and inclusion in the classroom. While
many studies show that encouraging students to actively participate in lectures benefits all
students, these activities especially benefit underrepresented students including women and
members of minority groups. In one study, students took an initial assessment that showed a
significant performance gap between men and women. Using daily practice problems, overall
student performance increased and, by the end of the semester, eliminated the initial
performance gap [32]. Active learning through interactive activities also helps equalize students
with different educational backgrounds. In a second-year physiology course at the University of
knowledge. In previous teachings of the course, students with limited background knowledge
performed very poorly, but when active learning was included, these students performed
equally as well as students with extensive science backgrounds [31]. By actively engaging with
students and applying the concepts in lecture, many students understand the importance and
application of various techniques and concepts. Students can actively seek help earlier in the
course when they do not understand a concept. Students feel more comfortable and confident
utilizing the material and more students benefit from attending lectures.
Change in Activity
Change in activity is the final category included in this study. While it does not directly
relate to any single pedagogical approach, it will act as the catch-all for any changes during
lecture not reflected by the other five categories.
Student attention spans have been shown to decrease anywhere between ten to thirty
minutes after the start of a traditional lecture. While this number is highly variable based on the
student, it is evident that student’s ability to pay attention is not maintained through a 50 to
90-minute lecture [24]. In order to maintain student’s attention, the lecture must have an
activity change [34]. This could be anything from incorporating one of the previous techniques
discussed to giving students a one-minute stretch break.
Every time we switch tasks, dopamine is released in the brain. This causes humans to
naturally want to do more than one thing at a time. This is especially evident in university
classrooms when students have both laptops and smartphones out. In recent studies of
18-20-year-olds, 44% of people access at least one of their technical devices once every ten minutes.
working on only one task, the sensory portion of the brain and the mechanical portion of the
brain work together to complete the task. While the brain naturally wants to switch tasks, when
more than one task is introduced the connection between the two parts of our brain decreases.
When the brain focuses on one task for too long, it essentially goes on ‘autopilot’ meaning our
thoughts can wonder even though the mechanical part of our brains is still completing that
task. This is evident in lectures when students may be copying notes from the lecture but are
day dreaming about something else. When students’ brains go on autopilot they can still be
writing notes, but they will not be able to recall certain parts of lecture as the sensory portion
of their brain is not focused on the lecture material. Students, however, cannot complete two
tasks using the same part of their brain. This means that students cannot understand what they
are hearing and day dream at the same time. Many students may even be unaware that they
are missing information when they inadvertently lose focus. However, when they change
movement, students become aware of the fact that they lost focus and can bring their attention
back to the lecture [35]. Thus, if professors change the lecture activities, requiring students to
physically move, students must check if they are engaged and bring focus back on the class.
Changes in activity during lectures also prevents students from getting bored and diverting
attention to their phones.
Many professors are reluctant to include changes in activity or breaks during lecture
because they often cannot cover as much material as if they lectured straight through the class
period. However, when professors cover more material and do not change during lecture,
students are not able to retain all the information [36]. This causes professors to have to repeat
students can only focus for a certain amount of time, professors should be less focused on
covering an extensive amount of material and be more focused on student engagement with
the material.
Methodology
This study tracks the use and evaluations of the aforementioned active learning
techniques in two introductory computer science courses at the University of North Carolina at
Chapel Hill. The first course is an introduction to Computer Science course that utilizes Python
as the primary language. This course is split into two sections and taught by the same professor
for both sections. This course also has a weekly recitation taught by a teacher’s assistant. The
second course is a foundations of programming course that does require prerequisite
knowledge of computer science. This course is taught in Java, and while it does have a
recitation, this was not tracked for the purposes of this study.
To track the best practices, students in the courses utilize a web application that allow
them to log every time a best practice is utilized and to rate the class at the end of each session.
The web application was chosen since students already utilize their laptops during these
courses. The web application also allows students the opportunity to use the platform on any
device if they do not have access to their computer. The application has a central login page
that validates the user and redirects them to the assigned view. There are three views: one for
the students, one for the professors, and one for the site admin. The views are assigned based
on the permissions of the user. When users first open the application, they see the instructions
page. The instruction page defines what each of the best practices are and gives a few examples
main menu bar with two buttons. One button allows users to sign up while the other button
allows users to sign in. The signup page can only be used to make new student accounts to
ensure that students do not obtain professor permission. Professor accounts are set up through
either the admin page or directly through the Firebase backend. Admin accounts can only be
created using the firebase backend.
When students log in, they are directed to a page that allows them to select the course
they are reviewing. They can only see courses they are enrolled in from the signup page. To add
or remove courses, students must visit their profile page. Once the student selects a course,
they can either go to the best practice log page by selecting the class home button or they can
choose to review that course. To begin with, students will typically start in the class home page.
This page should be opened throughout the lecture. The class home page uses a tile view that
lists all five of the best practices plus the option for an activity change. When professors apply a
best practice during lecture, the students should immediately select the associated tile. The
time, user id, and best practice are then logged. Once the lecture is over, students review what
they thought of the lecture. To navigate to the review page, students should select the review
button in the top menu. Each class session is reviewed based on five key criteria using a 1-5 star
rating system (5 is the best 1 is the worst). A three-star rating signifies an average lecture. The
five criteria are overall impression, understanding of the material, sense of community,
engagement, and interest. Overall impression asks the students to rate how well they liked the
lecture. This helps to better understand whether or not students overall like lectures that use
more of the active learning techniques. If students consistently rank overall impression high
students do like the active learning techniques. The second criterion used is understanding of
the material, which asks students if everything in the lecture was clear. One of the main
benefits of utilizing active learning is that students can better comprehend lecture material, so
this criterion allows students to validate that claim. The third criterion is sense of community. It
has been shown that one reason underrepresented students drop out frequently is because
they lack a community and a support system. This question ensures that students feel included
in lecture. It can help indicate to professors when students are feeling ostracized from the class.
The fourth ranking category asks how engaging the lecture was. Lectures should be engaging to
keep student’s attention so they do not play on their phones. If professors are effectively using
active learning techniques, students should feel very engaged throughout the lecture. The fifth
and final criterion is about interest. When students are interested in the subject matter, they
are more motivated to learn and retain the information long-term. Active learning techniques
help to build interest for a topic and motivate students. Once the students have finished their
review, they can log out.
For professors, the first page they see once they login is the raffle page. In this study,
one professor raffled stickers to increase app participation. At the beginning of each lecture,
the professor uses the raffle page to randomly select one student who participated in the last
class. To select a winner, the professor selects which course they are teaching and the date of
the last class. The winning student’s university id then is shown on the screen. Every time the
professor changes selection criteria or re-loads the page, a new user is chosen. This ensures
that if one student is absent, another student can be selected as the raffle winner. It was
enough time to respond. The second page professors can see is the best practices utilization
page. This page directly mimics the class home page from the student view except for that,
rather than being able to click on the tiles, the tiles immediately show how frequently each best
practice is used. It should be noted that every time a student selects a best practice as being
used, one additional count is added to the frequency of that best practice. Professors can filter
this page by date range and course. Professors can also see the collective rating for each lecture
in the reviews tab. All reviews and logs are completely anonymized as professors can only see
the overall star ratings. Professors can use these tools to modify their lectures. If professors see
that a best practice is heavily utilized in a lecture and students rate the lecture well, professors
can continue to incorporate this best practice in future lectures.
The final view is the admin view. This view is to only used to review the results of the
study. For the professor pages, professors only have access to the rankings for their specific
course. The admin pages have all the same functionality as the professor pages, but the admins
can see data for all the courses. Admins can also export the information into a CSV file for
further discovery.
The application utilizes Firebase for the backend and React for the front end. Firebase is
used to store user authentication, class logs and reviews in the database, and host the
application. Firebase was chosen because it keeps user information secure and maintains all
parts of the application in a single location. React was used so professors could see information
update in real time and allow for a more dynamic user experience. It also makes the application
look more streamlined across views so the same main components can be seen in all three
To introduce the application to courses, students first received a walk-through
introduction of the application. This walk-through explained the meaning of each best practice
and the rating criteria. It also showed students exactly how the application should be used
during lecture and gave a point of contact for any questions students have during the process.
The application was launched to the Introduction to Computer Science course during recitation
on January 16th. It was then introduced to the Foundations of Programming course during
lecture on January 22nd. The application was utilized in classes until March 7th. The start date
was chosen to give students enough time to finalize their schedules, and the end date was
chosen as it coincided with spring break. It was completely optional for students to utilize the
application and student responses remained anonymous for the duration of the study.
Once all the data was collected, duplicate logs were reviewed. Each student had a
unique number in Firebase that was used to keep the identity of the student anonymous. A log
was determined to be duplicate if it was submitted by the same student with the same active
learning technique in under two minutes. The application’s design allows students to easily pick
multiple active learning techniques at once and only uses banners to notify students that their
response went through. For this reason, some students could forget they logged a technique
and reselect the activity or could accidently double click the technique. Once all the duplicate
logs were removed, additional logs that were not recorded during class time were removed. For
students taking the introduction course with the recitation, they would sometimes submit
reviews and logs for the recitation when they meant to submit for the class. The logs were also
meant to be recorded during class time so any logs submitted five minutes after class were
data, if any days had no class reviews, they were removed as to not skew the results when
looking at the correlation between active learning techniques and the course reviews. Pivot
tables were then used for each class to compare how each of the five review criteria changed
depending on if a given active learning technique was present.
Results
In all courses combined, there were 307 users who created accounts for the application,
including professors and students. In the Introduction to Computer Science course, 123
students logged at least one best practice between the two sections and the recitation. In the
Foundations of Programming course, 70 students logged at least one best practice. For the
course reviews, 162 students filled out a review at least once with 113 of the students in the
Introduction to Computer Science course and 49 students in the Foundations of Programming
course.
Professors in both courses did an excellent job of utilizing best practices. The only times
that there were no recorded best practices were generally times when there were also no
recorded class review responses. In the four class periods in which no best practices were
recorded, there were very few course review responses recorded, so it is still possible that
active learning techniques did occur. For the purposes of the study, any number of
documentations by students of a best practice being used in a given class was considered
sufficient evidence for assuming the practice was used. For the introductory course, the first
section contained 12 total classes that were recorded. There were only three classes that did
not record a quiz, live coding, a relatable example, or an interactive activity. All other class
in activities, there was only one additional class where they did not occur. In the second section
of the introductory course, there were 10 classes recorded. All 10 sessions contained live
coding and relatable examples. Only two sessions did not contain quizzes, and only one class
did not contain an interactive activity. However, changes only occurred in four classes, and peer
activities were only in one class. In the recitation for this course, there were seven meetings
recorded. Each active learning practice again was in most meetings, although there were fewer
quizzes occurrences and more changes during recitation. The last course included in the study,
the more advanced Foundations of Computer Science course, had 12 meetings recorded. This
class contained relatively fewer best practices. There were only two quizzes given, and live
coding and interactive activities were only used in half of the classes. Peer learning was used in
seven of the courses, and relatable examples were used in ten of the twelve, which was the
most of any technique in this course. The distribution of the techniques was also very
uncorrelated for this class, as classes that contained one technique were not more likely to
contain any of the others. Between all of the classes, live coding was the most recorded
technique, while changes in activities was the least. This was expected for changes, as it should
mostly have been used when other practices were not applicable.
Throughout the study, there was not a clear correlation between utilization of active
learning techniques and student reviews of the course across all classes. In some classes, when
more techniques were recorded, reviews were nearly all higher, while in other classes, the
exact opposite was true. In the first section of the introductory course, every active learning
technique actually reduced the rating of the class in each category. This was especially
techniques consistently lowered this review from above a 4 to below a 3. However, these
results are very skewed by student response patterns. The three courses that contained no
active learning techniques each contained at most three responses, and they had the highest
rankings. This small sample thus biased the results for this class. This result was definitely still
surprising, as the opposite was expected to be true, but it is unlikely that this result is
representative.
In the second section of the same course, nearly the exact opposite result occurred. The
only active learning technique that did not consistently improve ratings was peer learning. Peer
learning, though, was only absent in one session, so this session’s ratings determined this
technique’s effect. Additionally, live coding and relatable examples were utilized in every class,
so it is unclear what their effects would be on the ratings. Yet, in the recitation, ratings again
were negatively correlated with active learning techniques. The only rating that increased when
active learning techniques were utilized was the understanding rating with quizzes. This rating
increased from an average of 3.07 to 3.40. However, every rating for all other active learning
techniques fell. The discrepancies between the two sections of this course and the recitation
clearly show a student-dependent interpretation of the efficacy of active learning techniques
for the introductory course. Students who documented most regularly also tended to give the
best reviews, which additionally biased the interpretation given discrepancies in the number of
student responses. Differences in the raw number of active learning techniques in the two
sections of this class also raise questions about the validity of the numbers for active learning
techniques. For example, quizzes were given on the same day to both sections, but they were
Finally, in the Foundations of Computer Science course, active learning again typically
increased reviews. In this case, results were likely more explanatory than in the other class, as
the variability of usage of each technique allowed for more differences between classes and
thus clearer evaluation. Quizzes, live coding, peer learning and interactive activities all
increased ratings for each category. For relatable examples, usage increased engagement, but it
decreased each of the other ratings categories slightly. The most any category decreased on
average for this technique, though, was only 0.4 out of 5. The only other active learning
technique that decreased ratings was change in activities, which negatively affected ratings for
overall impression and sense of community. It did increase ratings for understanding, though.
Engagement and interest remained about the same. In total, the number of responses in this
class were relatively low after the first class, but the number of responses was consistent
between one and six for each session. In the other course, however, the number of responses
ranged from one to 37, as students were incentivized to respond in some classes. Thus, again
the consistency of the foundations course may lead to more accurate results than for the
introduction course. In both courses, the most reviews occurred on the first day when the
application was announced.
Discussion
Ultimately, the results of this study were somewhat inconclusive. It is clear that
professors at the University of North Carolina at Chapel Hill do value the use of active learning
techniques, as they are including them in almost every lecture. However, the efficacy of these
techniques was clouded in this study by inconsistent responses and data representativeness
learning techniques makes it difficult to draw strong conclusions that would go beyond the
effects on a certain class. However, it is likely that student learning did improve as a result of
these techniques, as the other course that had more consistent results had a strong positive
correlation between reviews and best practice usage for each ranking area. Student reporting
on the use of active learning techniques was also subjective, as some students would report a
given technique while others did not for each class session. It was therefore difficult to
understand what occurred for each class. If repeated, the results should be cross referenced
with the professors’ lecture notes and syllabi. Another issue that lead to inconsistent test
results is the amount of time and thought students gave to reviews. Often times they would
simply mark all fives since this was quicker rather than taking the time to think about what a
fair assessment may be. Additionally, there is some selection bias in this study of the courses
chosen, as the professors volunteered to be a part of the study due to a likely prior preference
toward the use of active learning in their lectures.
The design of the application also plays a role in the data collected. For the best practice
student page, the goal was to make it easy for students to select best practices used while not
taking away from the lecture and their notes. It uses a banner to notify the student that a
button was clicked, but does not discern which button was clicked. This meant that multiple
instances of the same best practice were logged, making it more difficult to distinguish how
frequently each practice was used. Students would also log the same practice a few minutes
apart so it was unclear if they forgot they had already logged the active learning technique or if
when the course was not going on. This meant they would sometimes be logging for the wrong
course. In the future, it would be beneficial to only allow responses during the course time.
If this study were to be repeated, there are several steps that should be taken to
improve the validity of the results and the implications of the conclusions. First and foremost,
the study should either require student participation or provide better incentives to have more
consistent data on which to reflect. Professors could use the app for attendance or another
required activity. The study should also be run over the course of a few years rather than a
semester. This will help to even out students’ bias and give professors time to adjust their
courses to include more active learning techniques. Participation from multiple courses would
also give a more holistic view of the computer science department. Only two professors were
included in this study so results may have been different if more courses were included. To see
the effect on learning, it would be beneficial to include student grades on quizzes and exams.
This study only focused on the professors, so grades were not included. Additionally, student
reviews of the course did not always seem genuine. There were many times when student
ratings were all the same number and seemed rushed. It would help if professors gave a few
minutes at the end of class before the course was over to allow students to reflect and give
accurate reviews. Finally, professor should self-report active learning techniques to cross
reference with student interpretation. This would help better identify when students accidently
log a technique or forget to open the application at the start of class.
Overall, the results of this study were inconclusive and further study is needed to truly
understand active learning. The study began under the impression that active learning
study, professors were shown to use active learning techniques often, although they may have
been predisposed to do so. Strong conclusions cannot be drawn about their effectiveness,
though. This tool is still a beneficial way for students to give feedback. Rather than using it to
look at which best practices were utilized during lectures, professors could have students log
when they really enjoyed an activity or have students rate the class on a weekly basis to see
Appendix
Participation
Number of students who participated in the study
Course Total course
enrollment
Submitted at least 1 log
Submitted at least 1 review Introduction to Programming Section 1 100 58 60
Introduction to Programming Section 2 69 34 30 Introduction to Programming Recitation 169 90 78
Introduction to Programming 169 123 109
Foundations of Programming 409 66 49
Total 578 189 158
Best Practice Logs
Number of logs for each best practice in Introduction to Programming Section 1
Quiz 106
Live Coding 151
Peer Learning 34
Relatable Examples or Assignments 78
Interactive Activity 64
Change 24
Number of logs for each best practice in Introduction to Programming Section 2
Quiz 74
Live Coding 132
Peer Learning 3
Relatable Examples or Assignments 88
Interactive Activity 20
Change 7
Number of logs for each best practice in Introduction to Programming Recitation
Quiz 34
Live Coding 145
Peer Learning 36
Relatable Examples or Assignments 84
Interactive Activity 62
Change 20
Number of logs for each best practice in Foundations of Programming
Quiz 32
Pivot Tables
Introduction to Programming Section 1
Quizzes Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 4.111111111 3.666666667 4.333333333 3.833333333 3.777777778 1 3.009305138 2.864245198 2.851730302 3.052441132 2.967325063 Grand Total 3.284756632 3.064850565 3.22213106 3.247664182 3.169938242
Live Coding Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 4.111111111 3.666666667 4.333333333 3.833333333 3.777777778 1 3.009305138 2.864245198 2.851730302 3.052441132 2.967325063 Grand Total 3.284756632 3.064850565 3.22213106 3.247664182 3.169938242
Peer Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.833333333 3.4375 4.0625 3.875 3.583333333
1 3.010468281 2.878525847 2.801946589 2.933996273 2.963240696 Grand Total 3.284756632 3.064850565 3.22213106 3.247664182 3.169938242
Relate Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 4.111111111 3.666666667 4.333333333 3.833333333 3.777777778 1 3.009305138 2.864245198 2.851730302 3.052441132 2.967325063 Grand Total 3.284756632 3.064850565 3.22213106 3.247664182 3.169938242
Grand Total 3.284756632 3.064850565 3.22213106 3.247664182 3.169938242 Change Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 3.833333333 3.4375 4.0625 3.875 3.583333333 1 3.010468281 2.878525847 2.801946589 2.933996273 2.963240696 Grand Total 3.284756632 3.064850565 3.22213106 3.247664182 3.169938242
Introduction to Programming Section 2
Quizzes Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 2.9875 2.475 2.575 2.8625 3.0875
1 3.393648019 3.095716783 2.826959499 3.132029429 3.07317162 Grand Total 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296
Live Coding Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 1 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296 Grand Total 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296
Peer Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 3.291576017 2.982303807 2.737852888 3.086803937 3.084485884
1 3.5 2.875 3.125 3 3
Grand Total 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296
Relatable Example Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 1 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296 Grand Total 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296
1 3.337030562 3.049222999 2.832549858 3.137308987 3.145091945 Grand Total 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296
Change Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 3.142929293 2.773232323 2.632954545 2.989520202 2.896590909 1 3.566652098 3.269085082 2.991987179 3.211028555 3.345206876 Grand Total 3.312418415 2.971573427 2.776567599 3.078123543 3.076037296
Introduction to Programming Recitation
Quizzes Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.135416667 3.072916667 3.010416667 3 3.03125
1 2.736694678 3.402614379 2.642608154 2.448210395 2.413352007 Grand Total 2.964535814 3.214215686 2.852784447 2.763518741 2.766436575
Live Coding Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.75 3.75 3.5 3.75 3.5
1 2.65035014 2.999901961 2.593898226 2.368926237 2.473011204 Grand Total 2.964535814 3.214215686 2.852784447 2.763518741 2.766436575
Peer Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.75 3.75 3.5 3.75 3.5
1 2.65035014 2.999901961 2.593898226 2.368926237 2.473011204 Grand Total 2.964535814 3.214215686 2.852784447 2.763518741 2.766436575
Relatable Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.75 3.75 3.5 3.75 3.5
Interact Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.75 3.75 3.5 3.75 3.5
1 2.65035014 2.999901961 2.593898226 2.368926237 2.473011204 Grand Total 2.964535814 3.214215686 2.852784447 2.763518741 2.766436575
Change Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting
0 3.833333333 3.833333333 3.833333333 3.75 4
1 2.617016807 2.966568627 2.460564893 2.368926237 2.273011204 Grand Total 2.964535814 3.214215686 2.852784447 2.763518741 2.766436575
Foundations of Programming
Quizzes Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 3.685714286 3.542857143 2.7 2.866666667 3.128571429 1 4.228723404 4.117021277 3.478723404 3.845744681 3.819148936 Grand Total 3.776215805 3.638551165 2.829787234 3.029846336 3.24366768
Live Coding Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 3.3 3.133333333 2.133333333 2.833333333 2.733333333 1 4.116369952 3.999421045 3.327254306 3.170212766 3.608192213 Grand Total 3.776215805 3.638551165 2.829787234 3.029846336 3.24366768
Peer Row Labels Average of Impression Average of Understanding Average of Community Average of Engaging Average of Interesting 0 3.555555556 3.444444444 2.388888889 2.5 2.888888889 1 3.996876055 3.832657886 3.270685579 3.559692671 3.598446471 Grand Total 3.776215805 3.638551165 2.829787234 3.029846336 3.24366768
1 3.731458967 3.599594732 2.729078014 3.069148936 3.225734549 Grand Total 3.776215805 3.638551165 2.829787234 3.029846336 3.24366768
Interactive
Row Labels
Average of Impression
Average of Understanding
Average of Community
Average of Engaging
Average of Interesting 0 3.25 3.111111111 2.194444444 2.861111111 2.861111111 1 4.302431611 4.165991219 3.465130024 3.19858156 3.626224249 Grand Total 3.776215805 3.638551165 2.829787234 3.029846336 3.24366768
Change
Row Labels
Average of Impression
Average of Understanding
Average of Community
Average of Engaging
Average of Interesting 0 3.833333333 3.592592593 2.925925926 3.055555556 3.259259259 1 3.604863222 3.776426883 2.541371158 2.952718676 3.196892942 Grand Total 3.776215805 3.638551165 2.829787234 3.029846336 3.24366768
References
[1] M. Stains et al., “Anatomy of STEM teaching in North American universities,” vol. 359, no. 6383, pp. 1468–1470, 2018 [Online]. Available:
[2] C. J. Miller, J. McNear, and M. J. Metz, “A comparison of traditional and engaging lecture methods in a large, professional-level course,” vol. 37, no. 4, pp. 347–355, Dec. 2013 [Online]. Available: http://advan.physiology.org/content/37/4/347
[3] D. Wiliam, “What is assessment for learning?,” vol. 37, no. 1. pp. 3–14, 2011 [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0191491X11000149
[4] W. C. Hsu and S. W. Plunkett, “Attendance and Grades in Learning Programming Classes,” presented at the Proceedings of the Australasian Computer Science Week Multiconference, Canberra, Australia, 2016, pp. 4:1-4:6 [Online]. Available:
http://doi.acm.org/10.1145/2843043.2843061
[5] D. A. Wilder, W. A. Flood, and W. Stromsnes, “The use of random extra credit quizzes to increase student attendance,” vol. 28, no. 2, pp. 117–120, 2001 [Online]. Available: https://auth-lib-unc-edu.libproxy.lib.unc.edu/ezproxy_auth.php?
url=http://search.ebscohost.com.libproxy.lib.unc.edu/login.aspx? direct=true&db=eft&AN=507777931&site=ehost-live&scope=site
[6] M. Botek, Quizzes as a Tool for Student’s Attendance and Participation. 2013.
[7] S. H. McIntyre and J. M. Munson, “Exploring Cramming: Student Behaviors, Beliefs, and Learning Retention in the Principles of Marketing Course,” vol. 30, no. 3, pp. 226–243, May 2008 [Online]. Available:
https://journals-sagepub-com.libproxy.lib.unc.edu/doi/abs/10.1177/0273475308321819. [Accessed: 30-Mar-2019]
[8] N. Kornell, “Optimising learning using flashcards: Spacing is more effective than cramming,” vol. 23, no. 9, pp. 1297–1317, 2009 [Online]. Available: https://doi.org/ 10.1002/acp.1537
[9] H. L. Roediger and A. C. Butler, “The critical role of retrieval practice in long-term retention,” vol. 15, no. 1, pp. 20–27, Jan. 2011 [Online]. Available:
http://www.sciencedirect.com.libproxy.lib.unc.edu/science/article/pii/ S1364661310002081. [Accessed: 30-Mar-2019]
[10] M. J. Brown and J. Tallon, “The Effects of Pre-lecture Quizzes on Test Anxiety and Performance in a Statistics Course,” vol. 135, no. 3. pp. 346–350, 2015 [Online]. Available:
https://www.ingentaconnect.com/content/prin/ed/2015/00000135/00000003/ art00011
[12] B. W. Kennedy, “The value of quizzes,” vol. 44, p. 409, 2015 [Online]. Available: https://doi.org/10.1038/laban.857
[13] I. O. Pappas, M. N. Giannakos, and L. Jaccheri, “Investigating Factors Influencing Students’ Intention to Dropout Computer Science Studies,” presented at the Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education, Arequipa, Peru, 2016, pp. 198–203 [Online]. Available: http://doi.acm.org/10.1145/2899415.2899455
[14] A. Gaspar and S. Langevin, “Restoring Coding with Intention in Introductory Programming Courses,” presented at the Proceedings of the 8th ACM SIGITE Conference on Information Technology Education, Destin, Florida, USA, 2007, pp. 91–98 [Online]. Available: http://doi.acm.org/10.1145/1324302.1324323
[15] M. Bower and K. Falkner, “Computational Thinking, the Notional Machine, Pre-service Teachers, and Research Opportunities,” presented at the ACE 2015, pp. 37– 46.
[16] A. G. S. Raj, J. M. Patel, R. Halverson, and E. R. Halverson, “Role of Live-coding in Learning Introductory Programming,” presented at the Proceedings of the 18th Koli Calling International Conference on Computing Education Research, Koli, Finland, 2018, pp. 13:1-13:8 [Online]. Available:
http://doi.acm.org/10.1145/3279720.3279725
[17] M. J. Rubin, “The Effectiveness of Live-coding to Teach Introductory Programming,” presented at the Proceeding of the 44th ACM Technical Symposium on Computer Science Education, Denver, Colorado, USA, 2013, pp. 651–656 [Online]. Available: http://doi.acm.org/10.1145/2445196.2445388
[18] J. Bennett, Bringing Science to Life. Rotterdam: SensePublishers, 2016, pp. 21–39 [Online]. Available: https://doi.org/10.1007/978-94-6300-684-2_2
[19] “Motivating Students.” [Online]. Available: https://cft.vanderbilt.edu/guides-sub-pages/motivating-students/
[20] J. A. Middleton, “A Study of Intrinsic Motivation in the Mathematics Classroom: A Personal Constructs Approach,” vol. 26, no. 3, pp. 254–279, 1995 [Online].
Available: http://www.jstor.org/stable/749130
[22] D. King, Teaching and Learning in Context-Based Science Classes. Rotterdam: SensePublishers, 2016, pp. 71–85 [Online]. Available: https://doi.org/10.1007/978-94-6300-684-2_5
[23] C. Alvarado, Y. Cao, and M. Minnes, “Gender Differences in Students’ Behaviors in CS Classes Throughout the CS Major,” presented at the Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, Seattle, Washington, USA, 2017, pp. 27–32 [Online]. Available:
http://doi.acm.org/10.1145/3017680.3017771
[24] L. Porter, C. Bailey Lee, and B. Simon, “Halving Fail Rates Using Peer Instruction: A Study of Four Computer Science Courses,” presented at the Proceeding of the 44th ACM Technical Symposium on Computer Science Education, Denver, Colorado, USA, 2013, pp. 177–182 [Online]. Available:
http://doi.acm.org/10.1145/2445196.2445250
[25] B. Simon, J. Parris, and J. Spacco, “How We Teach Impacts Student Learning: Peer Instruction vs. Lecture in CS0,” presented at the Proceeding of the 44th ACM Technical Symposium on Computer Science Education, Denver, Colorado, USA, 2013, pp. 41–46 [Online]. Available: http://doi.acm.org/10.1145/2445196.2445215
[26] M. B. Isakson, “Why Try to Explain What You are Learning?,” 2019. [Online]. Available: https://ctl.byu.edu/why-try-explain-what-you-are-learning
[27] C. S. Grobe and A. Renkl, “Effects of multiple solution methods in mathematics learning,” vol. 16, no. 2. pp. 122–138, 2006 [Online]. Available:
http://www.sciencedirect.com/science/article/pii/S0959475206000156
[28] C. McDonald, Why Is Teaching Programming Difficult? Cham: Springer International Publishing, 2018, pp. 75–93 [Online]. Available: https://doi.org/10.1007/978-3-319-98590-9_6
[29] M. Schwartz, “Increasing Faculty-Student Engagement ” [Online]. Available: https://www.ryerson.ca/content/dam/lt/resources/handouts/FacultyStudentEngag ement.pdf
[30] S. R. Cotten and B. Wilson, “Student–faculty Interactions: Dynamics and Determinants,” vol. 51, no. 4, pp. 487–519, 2006 [Online]. Available: https://doi.org/10.1007/s10734-004-1705-4
[32] S. Grissom, S. Fitzgerald, R. ’ee McCauley, and L. Murphy, “Exposed! CS Faculty Caught Lecturing in Public: A Survey of Instructional Practices,” presented at the Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, Seattle, Washington, USA, 2017, pp. 261–266 [Online]. Available: http:// doi.acm.org/10.1145/3017680.3017702
[33] M. S. Young, S. Robinson, and P. Alberts, “Students pay attention!: Combating the vigilance decrement to improve learning during lectures,” vol. 10, no. 1, pp. 41–55, 2009 [Online]. Available: https://doi.org/10.1177/1469787408100194
[34] N. Scheele, A. Wessels, W. Effelsberg, M. Hofer, and S. Fries, “Experiences with Interactive Lectures: Considerations from the Perspective of Educational Psychology and Computer Science,” presented at the Proceedings of th 2005 Conference on Computer Support for Collaborative Learning: Learning 2005: The Next 10 Years!, Taipei, Taiwan, 2005, pp. 547–556 [Online]. Available:
http://dl.acm.org/citation.cfm?id=1149293.1149366
[35] L. Coulter-Smith, Changing Minds: Multitasking During Lectures. Cham: Springer International Publishing, 2018, pp. 3–16 [Online]. Available:
https://doi.org/10.1007/978-3-319-98590-9_1
