Mining Collaborative Patterns in Tutorial
Dialogues
1SIDNEY D’MELLO
Institute for Intelligent Systems, University of Memphis,
Memphis, TN, 38152, USA [email protected]
ANDREW OLNEY
Institute for Intelligent Systems, University of Memphis,
Memphis, TN, 38152, USA [email protected]
NATALIE PERSON
Department of Psychology, Rhodes College,
Memphis, TN, 38112, USA [email protected]
________________________________________________________________________
We present a method to automatically detect collaborative patterns of student and tutor dialogue moves. The method identifies significant two-step excitatory transitions between dialogue moves, integrates the transitions into a directed graph representation, and generates and tests data-driven hypotheses from the directed graph. The method was applied to a large corpus of student-tutor dialogue moves from expert tutoring sessions. An examination of the subset of the corpus consisting of tutor lectures revealed collaborative patterns consistent with information-transmission, information-elicitation, off topic-conversation, and student initiated questions. Sequences of dialogue moves within each of these patterns were also identified. Comparisons of the method to other approaches and applications towards the computational modeling of expert human tutors are discussed.
Key Words: Tutorial dialogue, pattern mining, expert tutors, lectures, likelihood metric, information transmission, information elicitation, human tutoring, educational data mining
________________________________________________________________________
1. INTRODUCTION
It is widely acknowledged that the classroom is not the optimal place for learning at
deeper levels of comprehension. Hence, it comes as no surprise that students turn to
one-on-one human tutoring when they are having difficulty in courses that require them to
demonstrate causal reasoning, diagnose and solve problems, make subtle comparisons,
generate inferences and explanations, and show application and transfer of acquired
knowledge. Investing in one-on-one tutoring does have a big payoff, as evident from the
substantial empirical evidence showing that human tutoring is extremely effective when
compared to typical classroom environments [Bloom 1984; Cohen et al. 1982; Corbett
1This research was supported by the by the Institute of Education Sciences, U.S. Department of Education,
2001]. Cohen et al. [1982] performed a meta-analysis on a large sample of studies that
compared human-to-human tutoring with classroom controls. Even ―novice‖ human
tutors achieved significant learning gains; with an effect size of .4 sigma (approximately
half a letter grade) compared to classroom and comparable control conditions.
Intelligent Tutoring Systems (ITSs) that promote active knowledge construction
similar to novice human tutors are also quite effective. The ITSs that have been
successfully implemented and tested (such as the Andes physics tutor, Cognitive Tutor,
and AutoTutor) have produced learning gains of approximately 1.0 standard deviation
units (sigma), or approximately one letter grade [Corbett et al. 1999; Graesser et al. 2004;
VanLehn et al. 2007]. This is an impressive feat because the 1.0 sigma effect size
produced by ITSs is superior to the 0.4 sigma effect size obtained from novice human
tutors [Cohen, Kulik and Kulik 1982], although it is lower than the 2.0 sigma effect
obtained by some expert human tutors in mathematics [Bloom 1984].
The effectiveness of one-on-one tutoring in human and computer tutors raises the
question of what makes tutoring so powerful? Chi et al. [2001; 2008] formulate three different hypotheses, which are known as the tutor-centered, student-centered, and
interaction hypotheses. The tutor-centered hypothesis contends that it is the pedagogical
and motivational strategies of the tutor that underlie the effectiveness of one-on-one
tutoring. Alternatively, the student-centered hypothesis predicts that tutoring is effective
because it gives students more opportunities to actively construct knowledge, rather than
anything the tutor does in particular. The interaction hypothesis is essentially the
blending of the tutor and student-centered hypotheses, by focusing on the coordinated
effort of both tutor and student. These different hypotheses require different research
methodologies and techniques of data analysis as is elaborated below.
The tutor-centered hypothesis has been of primary focus over the past several decades
of research, yielding some important insights about the pedagogical strategies employed
by tutors. For example, we have learned how tutors adapt to students needs by (a)
modeling and monitoring student knowledge [Chi et al. 2001; Derry and Potts 1998], (b)
employing tutoring tactics and strategies that contribute to learning gains [Du Boulay and
Luckin 2001; Fox 1991; Fox 1993; Lajoie et al. 2001; Palinscar and Brown 1984; Person
et al. 1994], (c) planning at local and global levels of discourse [Littman et al. 1990;
McArthur et al. 1990; Putnam 1987], and (d) providing emotional support for students in
social, affective, and motivational ways [del Soldato and du Boulay 1995; Issroff and del
are coded and the most frequent are assumed to be associated with enhanced learning.
More sophisticated models using regression models attempt to identify combinations of
tutor moves that are associated with learning gains in students [Chi et al. 2008; Di
Eugenio et al. 2009; Ohlsson, Di Eugenio, Chow, Fossati, Lu and Kershaw 2007].
The student-centered hypothesis contends that students are active participants in the
construction of their own knowledge, rather than being mere information receptacles.
This hypothesis has found substantial support in the tutoring literature [Chi 1996; Chi,
Siler, Jeong, Yamauchi and Hausmann 2001; Core et al. 2003; Jordan and Siler 2002;
Shah et al. 2002]. As with the tutor-centered methodology, it is fairly common for student
behavior to be coded and regressed or correlated with learning outcome measures [Chi,
Roy and Hausmann 2008]. This is perhaps not surprising: since structurally both the
tutor-centered and student-centered hypothesis considers the behavior of each participant
in isolation, there is usually no consideration of the larger structure or context
surrounding the interaction.
In contrast, the interaction hypothesis predicts that the effectiveness of tutoring draws
from both tutor and student behavior and their coordination with each other. Accordingly the traditional methodology of regressing or correlating coded behaviors with learning
gains is no longer sufficient. Instead, investigating the interaction hypothesis requires
regressing or correlating patterns of coded behavior with learning gains. The interaction
hypothesis has substantial overlap with the collaborative learning literature where it has
been shown that group learning outperforms individual learning [Wiley and Bailey 2006;
Wiley and Jensen 2006]. Simply put, both stress the interaction between participants.
Equally related are collaborative forms of instruction that are dialogue-centric,(e.g.,
reciprocal teaching) [Palinscar and Brown 1984]. Indeed, some researchers are beginning
to explore mixtures of collaborative learning and tutoring [Chi, Roy and Hausmann 2008;
Graesser et al. 2008; Kumar et al. 2007].
Investigations into the nature of one-on-one tutoring will inevitably encounter issues pertaining to ―grain size‖ or the level of analysis required to answer certain theoretical
questions. Barring a few exceptions [Cromley and Azevedo 2005; Shah, Evens, Michael
and Rovick 2002], the analysis of tutorial dialogue usually takes place at a very
fine-grained level, or the speech act level. Examples include investigations into how tutors
detect errors and misconceptions [Brown and Burton 1978; Chi et al. 1981; Corbett and
Anderson 1992; Feltovich et al. 2001; Fox 1991; Fox 1993; Sleeman et al. 1989], how
tutors provide feedback [Fox 1991; Fox 1993; McKendree 1990; Shute 2008], and how
tutor questions to elicit information from the student [Graesser and Person 1994; Graesser
et al. 1995; Lajoie, Faremo and Wiseman 2001].
Much of the analyses focus on the distribution of dialogue moves [Ohlsson, Di
Eugenio, Chow, Fossati, Lu and Kershaw 2007] and occasionally extend to two-step
student-tutor response cycles, (e.g., bigrams and transition probabilities) [Forbes-Riley et
al. 2005; Litman and Forbes-riley 2006; Lu et al. 2007]. Comparatively little is known
about how larger sequences of dialogue moves unite to achieve particular pedagogical
goals. Simply put, what are the dialogue patterns beyond two step student-tutor response
cycles? Identifying these dialogue move sequences along with their contextual
underpinnings will certainly expand the understanding of tutorial dialogue at a deeper
level that an analysis of the frequency of individual dialogue moves (or speech acts).
Another limitation of some of the previous work on human tutoring is that the vast
majority of studies monitored novice or unaccomplished tutors. These tutors were
untrained in tutoring skills and had moderate domain knowledge; they were peer tutors,
cross-age tutors, or paraprofessionals, but rarely were accomplished professionals or
expert tutors. Since, there is some evidence that expert tutors are more effective in
promoting learning gains than their unaccomplished counterparts [Bloom 1984; Cohen,
Kulik and Kulik 1982], an analysis of the discourse structure of expert tutoring dialogues
warrants further consideration.
This paper provides a methodology to examine the structure of tutorial dialogues
between students and expert human tutors by mining frequently occurring sequences of
dialogue moves generated during 50 naturalistic tutorial sessions. We show how two-step
transitions between moves can be aggregated into a graphical structure which permits
visual examination as well as the detection of more complex patterns. The method is
applied to a large corpus (over 45,000 speech acts) of tutoring dialogues between students
and the expert tutors in order to detect the latent structure and frequent patterns inherent
in tutorial lectures.
2. BRIEF OVERVIEW OF RELATED WORK
Although the impetus of investigations into tutorial dialogues has been at the dialogue
move level [Ohlsson, Di Eugenio, Chow, Fossati, Lu and Kershaw 2007], some recent
research has attempted to mine more complex patterns. This brief overview discusses
some of the recent data mining approaches for detecting patterns in tutorial dialogues.
The review focuses on Hidden Markov Models (HMMs) [Rabiner 1989], as a number of
Although sequential association rule mining techniques have not been extensively used in
the context of educational data mining, these are also described as they are potentially
useful pattern detection methods.
2.1 Hidden Markov Models (HMMs)
Some of the recent efforts to investigate the latent structure of tutorial dialogues utilizes
Hidden Markov Models (HMMs) [Beal et al. 2007; Boyer et al. 2009; Boyer et al. 2009;
Jeong et al. 2008; Soller and Stevens 2007]. HMMs are powerful tools for modeling
systems with sequential observable outcomes when the states producing the outcomes
cannot be directly observed (i.e., they are hidden) [Rabiner 1989]. HMMs can be used to model tutorial dialogue by assuming that the hidden states are ―tutorial contexts‖ or ―tutorial modes‖ [Cade et al. 2008], and the observed states are the dialogue moves. Examples of tutorial modes include ―lecturing on a topic‖ and ―modeling a problem‖,
while examples of dialogue moves are hints, prompts, and positive feedback. In this
context, an HMM will specify (a) the distribution of start states [start matrix], (b) the
distribution of dialogue moves given that the HMM is in a particular mode [emission
matrix], and (c) the probability of transitions between modes [transition matrix].
HMMs can be trained to automatically detect tutorial modes [Boyer, Phillips, Ha,
Wallis, Vouk and Lester 2009; Boyer, Young, Wallis, Phillips, Vouk and Lester 2009],
which is an advantage over manual annotation of modes [Cade, Copeland, Person and
D'Mello 2008]. However, the large number of parameters that need to be estimated
during the training process raises some problems. For example, the corpus analyzed in
this paper consists of 8 modes and 43 speech acts (described in the Tutorial Corpus
section). An HMM to model this data set would require 412 parameters (8 for the start
matrix, 8 × 43 for the emission matrix, and 8 × 8 for the transition matrix). This is, of
course, a conservative lower bound estimate, because we assumed that the number of
hidden states was known prior to the parameter estimation. In many cases, when there is
no underlying theory for guidance, determining the appropriate number of hidden states
is a challenging problem in itself.
Another problem is that the parameter estimation procedures that are used to train
HMMs, such as the popular Baum-Welch algorithm [Jurafsky and Martin 2008; Rabiner
1989], are quite sensitive to initial parameter estimates and can converge onto local
instead of global optima. This is a critical problem when models are seeded with random
parameter estimates. Although the parameterization problem can be alleviated by seeding
complex models with a large number of parameters, such as the models discussed in this
paper.
2.2 Sequential Association Rule Mining
Detecting structure in tutorial dialogues is essentially a sequential data mining problem.
Several algorithms that detect frequent patterns in sequential data have been proposed.
These include algorithms based on the Apriori framework such as GSP [Srikant and
Agrawal 1996], SPADE [Zaki et al. 1997], PSP [Masseglia et al. 1998], and a more
recent set of algorithms based on the pattern-growth framework such as Span,
PrefixSpan, CloSpan, and IncSpan [Han et al. 2000; Mortazavi-Asl et al. 2004].
Classification based on associations [Liu et al. 1998] has recently been used in a tutoring
context by Lu and colleagues to automatically extract feedback rules from a corpus of
human-human tutoring [Lu et al. 2008].
In general, sequential association mining algorithms and exploratory sequential data
analysis techniques attempt to detect frequent sequences of events as well as frequently
occurring nested events [Sanderson and Fisher 1994]. To this extent, they can contribute
towards an analysis of tutorial dialogues. However, one of the limitations of these
algorithms is that apart from identifying frequent patterns, most do not offer a mechanism
to integrate different patterns in order to obtain a broader understanding of the
phenomenon being analyzed. For example, log-linear models can be used to determine
whether a sequence is significant, but they are not well-suited for extracting larger
structure than two steps [Olson et al. 1994]. Lag sequential analysis is perhaps more
appropriate for determining larger structures since it can be used to determine across
multiple spans, or lags, whether a succeeding category occurs significantly differently
from chance [Bakeman and Gottman 1997].
The common sequence mining approaches have some limitations that introduce some
challenges with respect to their applicability for the analysis of tutorial dialogues. For
example, one limitation of log-linear models is that they are sensitive to low frequency
cell counts in the contingency tables used to construct these models. One proposed
solution to alleviate this problem involves eliminating low-frequency data elements prior
to the analyses [Olson, Herbsleb and Rueter 1994]. Although this is a viable solution in
some domains, it is not suitable for analyses of tutorial dialogues, because infrequent
dialogue moves can be very meaningful, and in some cases they play a more important
role than high frequency moves [Ohlsson, Di Eugenio, Chow, Fossati, Lu and Kershaw
however, it has a different set of limitations that are discussed in length in a previous
publication [Olson, Herbsleb and Rueter 1994]. Another limitation that is specific to
sequential association rule mining techniques that are based on the Apriori framework
[Srikant and Agrawal 1996], is that these approaches require the specification of arbitrary
parameters to prune the search space (i.e., support thresholds).
Finally, PRONET is a methodology for constructing a network representation from
proximity data [Cooke et al. 1996]. The basis of the PRONET approach is the Pathfinder
algorithm [Schvaneveldt 1990; Schvaneveldt et al. 1989] which takes a proximity matrix
as input and yield a directed or undirected graph representation. The resulting graph has
the property that each link is a minimum weight path connecting two entries in the
proximity matrix. The graph representation is parameterized by , which is the
Minkowski distance between nodes, and , which is the maximum number of links that
can connect two nodes. In essence, Pathfinder is not a sequential datamining algorithm,
but rather a technique of scaling data into a network representation.
2.3 Differences between Current Approach and Existing Sequential Analysis
Methods
As evident from the aforementioned discussion, HMMs, sequential association mining
approaches (e.g., Span, PrefixSpan, CloSpan, IncSpan, GSP, PSP), and exploratory
sequential data analysis techniques (e.g., log-linear models) are all viable methods to
extract patterns in tutorial dialogues. These methods are accompanied with an associated
set of disadvantages that limit their applicability towards the analysis of tutorial
dialogues. The present paper proposes an alternate method to address some of these
limitations. Briefly, our method involves detecting frequent two-step sequences of
dialogue moves and integrating these sequences into a directed graph. Different aspects
of the tutorial dialogues can be investigated by focusing on different characteristics of the
graph, as will be elaborated later.
Our approach capitalizes on the benefits of the various pattern mining methods, but
differs from these methods in a number of ways. First, it incorporates fewer parameters
than HMMs and does not require complex parameter estimation methods, thereby
avoiding the local versus global optima conundrum. Second, unlike methods derived
from the Apriori framework, it utilizes null hypothesis significance testing instead of
arbitrary threshold specification for search space pruning. Third, it is not sensitive to low
and finally, it integrates frequent patterns into a larger structure that is conducive to a
broader analysis of the phenomenon being modeled (i.e., tutorial dialogues in this paper).
We illustrate the method by analyzing the latent structure in student-tutor dialogue
moves from an existing corpus. We begin with a brief overview of the corpus and the
coding scheme used to annotate the student and tutor speech acts.
3. TUTORIAL DIALOGUE CORPUS
The corpus consisted of 50 tutoring sessions between students and expert tutors on the
topics of algebra, geometry, physics, chemistry, and biology. The students were all
having difficulty in a science and/or math course and were either recommended for
tutoring by school personnel or voluntarily sought professional tutoring help.
The expert tutors were recommended by academic support personnel from public and
private schools in a large urban school district. All of the tutors have long-standing
relationships with the academic support offices that recommend them to parents and
students. The criteria for being an expert tutor were (a) have a minimum of five years of
one-to-one tutoring experience, (b) have a secondary teaching license, (c) have a degree
in the subject that they tutor, (d) have an outstanding reputation as a private tutor, and (e)
have an effective track record (i.e., students who work with these tutors show marked
improvement in the subject areas for which they receive tutoring).
Fifty one-hour tutoring sessions were videotaped and transcribed. To capture the
complexity of what transpires during a tutoring interaction, two coding schemes were
developed to classify every tutor and student dialogue move in the 50 hours of tutoring
[Person et al. 2007; Person et al. 2007]. A total of 47,296 dialogue moves were coded. A
dialogue move was either a speech act (e.g., a tutor hint), an action (e.g., student reads
aloud), or a qualitative contribution made by a student (e.g., partially-correct or vague
answer). Multiple dialogue moves could occur within a single conversational turn. For
example, the conversational turn ―[No, no, let’s, let’s], [let’s let’s not get confused on
that]. [When we talk about mitosis, we’re talking about how your skin would produce new skin cells]‖ is comprised of negative feedback, a general motivational statement, and
direct instruction.
The Tutor Coding Scheme consisted of 27 categories inspired by previous tutoring
research on pedagogical and motivational strategies and dialogue moves [Cromley and
Azevedo 2005; Graesser, Person and Magliano 1995; Lepper and Woolverton 2002]. The
moves consisted of various forms of information delivery (e.g., direct instruction,
prompts, pumps, forced choices), elaborated and non-elaborated feedback (i.e., positive,
negative, neutral), some motivational moves (e.g., general motivation statement,
solidarity statement), humor, and off-topic conversation (see Table I).
A 16 category coding scheme was also developed to classify all student dialogue
moves (see Table I). Some of the student move categories captured the qualitative nature
of a student dialogue move (e.g., correct answer, partially-correct answer, error-ridden
answer), whereas others were used to classify types of questions, conversational
acknowledgments, and student actions (e.g., reading aloud or solving a problem).
Four trained judges coded the 50 transcripts on the dialogue move schemes. Cohen’s
kappas were computed to determine the reliability of their judgments. The kappa scores were .92 for the tutors’ moves and .88 for the students’ moves. A kappa of .75 or greater
is considered to be excellent [Robson 1993].
In addition to coding dialogue moves, we also coded ―tutorial modes,‖ or briefly ―mode‖. A mode can be considered to be the overarching context or teaching phase
during which learning occurs, such as a lecture, a problem modeling phase, a problem
scaffolding phase, etc. Each of the dialogue moves in the current corpus was assigned to
one of eight modes: introduction, lecture, highlighting, modeling, scaffolding, fading,
off-topic, and conclusion. Kappa scores for the eight modes ranged from 0.8 to 1.0 with a
Table I. Coding Schemes for Dialogue Moves
Dialogue Move Examples
Tutor Moves
Comprehension Gauging Question ―Do you understand‖, ―You remember?‖ Attribution Acknowledgment ―You’re just making a mental block.‖
Conversational Ok ―Ok.‖ ―Alright.‖
Counter Example ―And I don’t mean chunks like in pieces of potato.‖
Direct Instruction ―Well the lysosomes contain digestive enzymes.‖
Example ―Most picked FedEx as an analogy for Golgi body.‖
Forced Choice ―Would that be random, uniformed, or clumped?‖
Motivational Statement ―See, you get it now.‖ ―Wow, you’re so good.‖
Hint ―It’s a family tree…it’s just a regular family tree.‖
Humor ―You never see a rose bush jump on a person.‖
Negative Feedback ―No.‖ ―Uh uh.‖
Negative Feedback Elaborated ―Before that, you’re jumping too far ahead.‖
Neutral Feedback ―Well, that’s not quite it.‖
Neutral Feedback Elaborated ―Uh, ATP is gonna be one of the players.‖
Paraphrase [S] ―It ………...‖ [T] ―It’s inside the wall.‖
Off topic ―I’m cooking dinner—I’m really excited.‖
Pose New Problem ―Difference between prokaryotes and eukaryotes.‖
Pose Simplified Problem ―What is the outside layer?‖
Positive Feedback ―Correct.‖ ―Right.‖ ―Exactly.‖
Positive Feedback Elaborated ―It’s still random, exactly.‖
Preview ―Chloroplast, we haven’t talked about that yet.‖
Prompt ―So 200 times one is what?‖
Provide Correct Answer ―Code next to it.‖
Pump ―Which is?‖ ―And?‖ ―Then?‖
Repetition S: ―Is it commensalism?‖ T: ―Commensalism.‖
Solidarity Statement ―Now think with me.‖ ―Alright, here we go.‖
Summary ―Alright, so we’ve talked about…‖
Student Moves
Common Ground Question ―Aren’t they more lined up, like more in order?‖ Conversational Acknowledgment ―Ok.‖ ―No sir.‖ ―Yes ma’am.‖
Correct Answer ―In meiosis it starts out the same with 1 diploid.‖
Error Ridden Answer ―Prokaryotes are human, eukaryotes are bacteria.‖
Gripe ―Ugh.‖ ―<groans>‖
Knowledge Deficit Question ―What do you mean by it doesn’t have a skeleton?‖
Metacomment ―I don’t know.‖ ―Yes, I understand.‖
Misconception ―I always used to get diploid and haploid mixed up.‖
No Answer ―Umm.‖ ―Mmm.‖
Off topic (Tutor) ―Did you all have a quiz today.‖
Partial Answer ―It has to do with the cells.‖
Read Aloud ―Question 7: Plot growth pattern.‖
Social Coordination Action ―No, I didn’t hear about that.‖
Student Works Silently -
Think Aloud ―500 equals 50 and 50 divided by 500 gives 10.‖
4. MINING STRUCTURES IN LECTURES
We use lectures as an illustrative example of the pattern mining approach. Our use of
lectures is motivated by a number of factors. First, lectures and scaffolding are the two
most frequently occurring modes during expert tutoring sessions. They collectively
comprise 77% of the corpus, with approximately 30% of the turns being lecture
deliveries. Second, the complexity of lectures lies somewhere between scaffolding and
the other modes, thereby making lectures an ideal example to illustrate the method. Third,
our analysis of the structure of lectures yielded some interesting patterns that are
somewhat different from some of the ways lectures have been previously conceptualized
[Chi 1996; Lu, Di Eugenio, Kershaw, Ohlsson and Corrigan-Halpern 2007; Wheatley
1991].
The method involves three stages or processes. These include (1) detecting significant
excitatory transitions between dialogue moves, (2) constructing a representation that
integrates the significant excitatory transitions, (3) generating and testing data-driven
hypotheses from the representation. The input data consists of time series of student-tutor
dialogue moves that have been annotated by a human or a computer (human annotation in
this paper). Each student-tutor dyad has its own time series; hence, the input consists of
such time series.
4.1. Transition Detection
4.1.1 Likelihood Metric. The first step involves detecting significant two-step transitions between dialogue moves from the time series. If there are dialogue
moves in the corpus, then the number of transitions considered is . These
transitions are computed for each individual time series and significant transitions are
identified via null hypothesis significance testing. It should be noted that this form of
brute force sample space searching is only applied to detect significant two-step
transitions. The search space is substantially pruned for higher level transitions as will be
illustrated in Section 4.3.
We use the likelihood metric [D'Mello et al. 2007] to compute the likelihood of a
transition between any two moves. The metric can be represented as ,
where is the move at time (the current move), is the next move at .
We motivate the likelihood metric by contrasting it with the conditional probability of
a transition to given that the immediate move is (see Equation 1). While
conditional probabilities are suitable for exploring the strength of association between
feedback (NF), is very frequent, the conditional probability (NF | ) will be high
because (NF | ) is proportional to the frequency of NF (also known as base rate).
(Equation 1)
The likelihood metric addresses the influence of base rate by penalizing associations
that are not greater than an expected amount of association. Formally, the likelihood
metric is defined as:
(Equation 2)
The reader may note significant similarity to Cohen’s kappa for agreement between
raters [Cohen 1960] and indeed the likelihood metric can be justified in a similar fashion.
The definition of Cohen’s is listed in Equation 3.
(Equation 3)
In Equation 3, Pr(A) is the agreement between two raters, and Pr(E) is the expected
agreement. So removes the agreement expected by chance and then normalizes by the
total possible agreement (1: perfect agreement) minus expected agreement again. The
likelihood metric proceeds in the same way, however rather than agreement we use
conditional probability as a measure of association (see Equation 2). The expected degree
of association is , because if and are independent, then
Therefore the numerator of Equation 2 equals the degree of
association observed minus the degree of association expected under independence. If the
observed degree of association is higher than that expected under independence, then the
numerator will be positive (a positive association). If the observed association is equal to
that expected under independence, then the numerator will be zero (no association).
Finally, if the observed association is less than that expected under independence, the
is intuitively understandable as the direction and size of the association between
and , accounting for the base rate of .
One sample t-tests can be used to ascertain whether a transition is statistically
significantly greater than (excitatory), less than (inhibitory), or equal to zero (no
relationship between immediate and next move). The effect size (in standard deviation
units) for a transition can be measured by Cohen’s , , where
and are the sample means and standard deviations for a transition,
respectively. Effect sizes of .2 sigma are small effects, .5 sigma is a medium effect, and .8
sigma or greater is a large effect [Cohen 1992].
4.1.2 Applying Likelihood Metric to Lectures. A time series that preserved the temporal ordering of the moves was constructed for each session. We extracted the move
sequences that were classified as lectures. Four out of the 50 sessions did not include a
lecture mode; hence, we examined the remaining 46 time series. On average, there were
298 moves per time series (SD = 325). Time series ranged from 16 to 1692 moves with a median of 172 moves. Overall, the 46 lectures comprised 13,712 moves.
We applied Equation 2 to each of the lecture time series. A one-sample t-test was
used to test the significance of 1806 out of the possible 1869 (43 × 43) transitions.
Transitions between the same moves were not considered as the coding scheme does not
permit such transitions. It appears that there were 970 transitions that were statistically
significant at the .05 level.
It should be noted that there is a risk of increasing Type 1 (false positive) errors when
a large number of significance tests are conducted. One option to alleviate this risk is to
focus on a small set of theoretically derived predictions instead of considering the entire
set of potential transitions. This option was not considered because the primary goal of
the method is to mine tutorial dialogues for interesting patterns and latent structures.
Focusing on a small subset of transitions does not advance this goal.
Another option is to make the significance criteria more stringent by applying a
post-hoc correction. A commonly used multiple test correction is the Bonferroni correction
[Shaffer 1995], where the significance criterion (α) is adjusted for the number of
comparisons ( ). In particular, , where is the corrected significance
threshold.
Applying the Bonferroni correction to our data would result in an extremely
conservative α threshold of 0.000028 ( Using such a low alpha value is
also not attractive as this substantially increases the risk of committing Type II (false
A closer examination of the number of significant transitions revealed that it is
unlikely that our results were obtained by a mere capitalization on chance. Monte-Carlo
simulations across 100,000 runs confirmed that the probability of obtaining 970 out of
1806 significant transitions (52%) by chance alone is approximately 0. Therefore, it is
unlikely that the overall patterns in the data were obtained by chance.
Table II presents the direction (+,-) of the transitions that were significant at the .05
level. The results indicate that a majority of the transitions were inhibitory, which is what
we expected, as the likelihood metric is quite conservative. Although inhibitory
transitions are informative in their own right, an examination of these transitions is
beyond the scope of the paper. Hence, the subsequent analyses exclusively focus on the
68 excitatory transitions.
There is an obvious concern that 68 out of 1806 significant excitatory transitions is
not sufficient to capture the complexity of expert tutors’ lectures. We addressed this
concern by constructing a session × transition matrix (46 × 1806), where each
represented the probability of observing transition in session . The average for each
column represents the mean probability of transition across the 46 sessions with
lectures. The results indicated that the 68 excitatory transitions represented 52.4% of all
possible transitions. Hence, a substantial portion of the variance is explained by a handful
Table II. Significant transitions between moves
Next Move
Current Move 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
1. Attribution Acknowledgment + - - - -
2. Acknowledgment - + - + - - - + - - - -
3. Counter Example - - - -
4. Common Ground Question - - - + + - - - + + - - + - - - -
5. Comprehension Gauging Question + - - - + - - - -
6. Correct Answer - - - + + + - + - - - -
7. Direct Instruction/Explanation + + - - + - - - - + - - - -
8. Error Ridden Answer - - - + - + - - - + - - - -
9. Example - + - - - -
10.Forced Choice - + - - - -
11.General Motivational Statement - - - -
12.Gripe - - - -
13.Hint - + - + - + - - - + - - - -
14.Humor - - - + - - - -
15.Knowledge Deficit Question - - - - + - - - -
16.Metacomment - - - + - - - -
17.Misconception - - - + - - - -
18.Negative Feedback - - + - - - -
19.Negative Feedback Elaborated - - - -
20.Neutral Feedback - + - - - -
21.Neutral Feedback Elaborated - - - -
22.New Problem - - + - + - + - - - + - - - +
23.No Answer - - - + - - - + - - - - -
24.Conversational Ok - - - - + - - - + - - - - + + - -
25.Other - - - + - - - + + - - - - -
26.Other - - - + - - - -
27.Partial Answer - - - + + - - - -
28.Paraphrase - - - -
29.Provide Correct Answer - - - -
30.Positive Feedback - + - - - -
31.Positive Feedback Elaborated + - - - -
32.Preview - - - -
33.Prompt - - + - + - - - + - - + - - - -
34.Pump - - - -
35.Read Aloud -
36.Repetition - - - - + - - - + - - - -
37.Social Coordination Action - - - + - - - -
38.Simplified Problem - - - + - + - - - + + - - - + - - + - - - +
39.Solidarity Statement - - - -
40.Summary - - - + - - - -
41.Student Works Silently - - - -
42.Think Aloud - - - -
4.2. Integrated Representation
It is difficult to detect tutorial dialogue patterns from the transition matrix presented in
Table II. What is needed is a representation that integrates the two-step excitatory
transitions. We propose a directed graph representation of the excitatory transitions in the
matrix. Directed graph representations are useful for a number of reasons. First, they
serve as powerful illustrative tools for visual pattern detection. Second, graph algorithms
(e.g., paths, trails, searches, and many others) can be used to reveal complex,
non-intuitive patterns that elude visual detection.
Figure 1 presents a directed graph representation of the excitatory transitions in Table
II. The vertices of the graph are the individual dialogue moves and the edges represent
significant excitatory transitions between the moves (see Figure 1). For simplicity, we
only included the 34 transitions with effect sizes greater than the median, which was.52
sigma. Effect sizes for these 34 transitions ranged from .52 to 1.88 sigma (medium to
large effects), with a mean effect size of .89 sigma (SD = 35). These 34 transitions comprised 44.6% of all possible transitions observed in the data.
4.2.1 Visual Inspection of Collaborative Patterns. A visual analysis of the directed graph in Figure 1 yielded four collaborative student-tutor move clusters. The first cluster
pertains to conversational moves that support direct instruction (solid links in Figure 1).
These include, the tutor transmitting information via direct instruction and explanations, comprehension gauging questions initiated by the tutor, students’ metacomments, and
conversational moves such as acknowledgements and oks. This cluster can be considered
to be an information transmission cluster as its primary pedagogical goal appears to be the delivery of information from tutor to student.
The second cluster, or the information elicitation cluster (dotted links in Figure 1), consists of moves associated with attempts by the tutor to elicit information from the
student. These moves are variations of the Initiate Respond Evaluate (IRE) sequence
[Mehan 1979]. The sequence begins by the tutor asking the student a question with
prompts, pumps, forced choices, or simplified problems. The student responds with no
answers, error-ridden answers, misconceptions, partially-correct answers, or correct
answers. The tutor evaluates the student’s response and provides positive, negative, or
neutral feedback, with or without elaborations.
The two clusters (information transmission and information elicitation) appear to be
connected with a link from the various forms of feedback to direct instruction. The
following short excerpt from an actual lecture between a biology tutor (T) and a student
representative of information elicitation, while moves 149-151 are indicative of
information transmission. The two clusters are connected by the link from positive
feedback to direct instruction (moves 148 → 149).
[145]
T So in other words, is it is equal, greater, or less than? [Forced Choice] [146]
S Equal. [Correct Answer] [147]
T Equal. [Repetition] [148]
T Very good. [Positive Feedback] [149]
T So when you are at the carrying capacity, birth rate equals death rate.
That makes sense because you reach the carrying capacity. [Direct
Instruction] [150]
T Makes sense? [Comprehension Gauging Question] [151]
S Mm hmm. [Acknowledgment]
[152]
T So the growth rate is 0 [Direct Instruction]
In addition to the information transmission and information elicitation clusters, there
is also an off-topic conversation exchange between the student and the tutor (separate graph on top right of Figure 1). These off-topic conversations appear to be triggered by
humor (e.g., ―Well, let’s get to a good one. Surely. Ooh Rhombus! Ooh hexagon!‖) on
the part of the tutor or a socially coordinated action (e.g., ―Hand me the calculator‖) by
the student.
Although students rarely asks questions in tutoring sessions [Graesser and Person
1994], they sometimes take initiative by asking common ground questions (e.g., ―You
mean the formulas?‖) or knowledge deficit questions (e.g., ―What is the difference
between anaphase and telephase?‖). Tutors respond to common ground questions with
positive feedback, presumably to encourage such questions, followed by direct
instruction. Knowledge deficiencies are corrected with direct instruction and
explanations. Thus, student questions comprise the fourth collaborative pattern observed during lectures (dashed links in Figure 1).
4.2.2 Incidence of the Collaborative Patterns. There is the question of determining how often each of the collaborative patterns appears in the tutorial lectures. One possible
way to assess the incidence of a pattern is to add the proportional occurrence of dialogue
moves within that pattern. In general, moves can be assigned to one of the four patterns
(i.e., assignment of moves to clusters is generally mutually exclusive). However, direct
patterns. Although direct instruction is mostly associated with information transmission,
it also occurs with information-elicitation and student questions. Similarly, positive
feedback is a member of the student question and information elicitation patterns. Hence,
it is not possible to analyze the occurrence of a pattern by simply adding the proportional
occurrence of moves within that pattern.
Assignment of links to patterns, however, is mutually exclusive. It is possible to
compute the incidence of a pattern by adding the proportional occurrence of links within
that pattern. The results indicated that the eight links associated with information
transmission represented 70.2% of the 34 links. The information elicitation, off-topic
conversation, and student question patterns represented 18.6, 9.0, and 2.2 percent of the
links, respectively. Hence, lectures mainly consist of information transmission with the
occasional information elicitation and off-topic conversation patterns. As always, student
Figure 1. Directed graph representation of positive excitatory transitions between moves
4.3. Hypothesis Generation and Testing
A visual inspection of the directed graph representation of the significant excitatory
transitions was useful to identify and assess the incidence of major collaborative
student-tutor patterns. However, the real merits of directed graph representations lie in their
ability to generate data-driven hypotheses that can be empirically tested. The critical
insight here is that properties of the graph provide hypotheses about complex associations
between dialogue moves.
This claim can be best explained with some examples. Graphs have paths, cycles, and
circuits. A path is a set of distinct vertices such that two consecutive vertices in a path are
connected by an edge [Tucker 1995]. For example, new problem (newp), correct answer
(cor), positive feedback (pf), and direct instruction (die) form a path of length three (see
Figure 1). Note the length of a path is computed on the basis of its edges and not vertices.
A cycle is a sequence of vertices where the starting and ending vertex are the same,
and any two consecutive vertices are connected by an edge. An edge cannot appear more
than once in a cycle, but the same vertex can be visited more than once [Tucker 1995].
Direct instruction (die) → acknowledgement (ack) → direct instruction (die) → comprehension gauging question (cgq) → acknowledgement (ack) → conversational ok (ok) → direct instruction (die) form a complex cycle in Figure 1. A circuit is like a cycle, but each vertex can appear only once, except the starting and ending vertices. Hence,
direct instruction (die) → comprehension gauging question (cgq) → acknowledgement (ack) → conversational ok (ok) → direct instruction (die) form a circuit in the directed
graph (see Figure 1).
The directed graph representations of the excitatory representations can be explored
with graph algorithms to obtain complex patterns such as paths, cycles, and circuits.
These represent hypotheses of possible patterns in the tutorial dialogues. The existence of
these patterns can be subsequently verified by scanning the time series of student-tutor
dialogue moves. This can be achieved by computing the probability of a pattern in each
of the observed time series ( ) and comparing this probability to what could be
expected by chance. Chance can be estimated by computing the probability that the
pattern exists in the randomly shuffled surrogate of the observed time series ( ).
Random shuffled surrogates are useful controls because they preserve the prior
probabilities of individual moves while breaking temporal dependencies among moves.
The transition likelihood metric from Equation 2 can be modified to quantify the
likelihood of a particular pattern , where can be a path, cycle, trail, circuit, etc, (see
(Equation 4)
This metric is similar to the kappa statistic [Cohen 1960] and can be interpreted in the
same way as the metric presented in Equation 2. Specifically, if , then the
pattern occurs at rates greater than chance; , implies rates below chance, and
, indicates occurrence at chance levels. The significance of pattern likelihood
can be assessed with one-sample t-tests. A one-tailed one-sample t-test can be used
because we are only interested in determining whether a hypothesized pattern occurs
above chance. An example application of the hypothesis generation and testing method is
described below.
4.3.1 Detecting Circuits in Lectures. A circuit detection algorithm applied to the directed graph revealed 18 potential circuits. A sequence-matching algorithm was then
used to compute the probability of each circuit occurring in each time series. One-tailed,
one-sample t-tests indicated that 11 circuits occurred at rates significantly greater than
chance. Monte-Carlo simulations across 100,000 runs confirmed that the probability of
obtaining 11 out of 18 significant circuits (61%) is very small (approximately zero).
Therefore it is unlikely that the significant circuits were obtained by capitalizing on
chance alone.
The significant circuits primarily involved links in the information-transmission
pattern, which is what could be expected as this pattern accounts for 70.2% of the links.
The simplest circuit pertains to an oscillation between direct instruction by the tutor and
acknowledgments by the student (direct instruction → acknowledgement → direct
instruction). The tutor asserts some information (e.g., ―You always have to have radius‖),
the student provides a verbal acknowledgement (e.g., ―Right‖), and the tutor continues to
assert another chunk of information (e.g., ―You’re always gonna have radius because area’s gonna equal pi r squared‖).
A conversational extension to this circuit occurs when the tutor acknowledges the student’s acknowledgment with a discourse marker (e.g., ―ok‖) followed by another
direct instruction move (direct instruction → acknowledgement → ok → direct
instruction). The conversational ok presumably indicates that the conversational floor is
now being shifted from the student to the tutor.
A more pedagogically oriented extension to this basic circuit occurs when the tutor
are saying‖) and the tutor responds with a conversational ―ok‖ and continues with more
direct instruction (direct instruction → comprehension gauging question → metacomment → ok → direct instruction).
It is important to note that although these patterns seem trivial, they serve important
pedagogical and communicative goals. Back-channel feedback and conversational
acknowledgments are how students communicate to tutors that they are listening,
attentive, engaged, and are active partners in the conversation [Heinz 2003; Ward and
Tsukahara 2000]. Tutors presumably use comprehension gauging questions to monitor
students understanding of the content [Chi, Siler, Jeong, Yamauchi and Hausmann 2001],
although there is some evidence that students cannot accurately monitor their own
understanding [Glass et al. 1999; Graesser, Person and Magliano 1995; Person, Graesser,
Magliano and Kreuz 1994]. Tutors might also interleave these questions into the direct instruction cycle to enhance students’ engagement and to cue students to the fact that they
need to attentively comprehend the lecture.
4.3.2 Connected Components and Degrees of Vertices. In addition to path, cycle, and circuit detection, some additional properties of the directed graph provide important
insights into the nature of tutorial dialogues. One important concept pertains to the
strongly connected components of a directed graph. These are a set of maximal subgraphs where a path exists from each vertex of the subgraph to each other vertex of the subgraph
[Tucker 1995]. Identifying the strongly connected components of a graph allows us to
identify tightly coupled clusters of dialogue moves.
An analysis on the graph in Figure 1 yielded two strongly connected components.
These included moves in the information-transmission cluster (direct instruction,
acknowledgement, comprehension gauging question, metacomment, and ok) and
off-topic conversation by the student and tutor. Although these components could be
analyzed with a simple visual analysis due to the relative simplicity of lectures, more
formal methods [Tarjan 1972] would be required to detect connected components for
more complex tutorial modes such as scaffolding.
Another useful feature of the directed graph representation is the characterization of
the relative importance of a given dialogue move on the basis of the degree of the vertex representing that move. The degree of a vertex is the number of edges connected to it
[Tucker 1995]. The degree of a move, represented by a vertex in the directed graph,
provides a characterization of its contextual influence beyond mere proportional
occurrence. Contextual influence refers to the number of moves a particular move is
conversational acknowledgements, which are two frequent student moves.
Acknowledgments comprise 45% of all student moves during lectures compared to the
mere 10% accounted for by correct answers. However, an examination of Figure 1
indicates that correct answers have a larger contextual influence than acknowledgements
(degreecorrect = 6; degreeacknowledgement= 4), even though acknowledgement are four times
more likely to occur than correct answers.
Additional insights can be gleaned by comparing the in and out degrees of a given
vertex. The in-degree of a vertex is the number of incoming edges while the out-degree is
the number of outgoing edges. For example, correct answers have an in-degree of 4 and
an out-degree of 2. This indicates that the tutor is making an effort to elicit correct
information from the student via forced choices, prompts, new problems, and simplified
problems (see Figure 1). Similarly, direct instruction has an in-degree of 8 and an
out-degree of 2, suggesting that to some extent all roads lead to direct instructions, which is
what could be expected during a lecture.
In summary, as our examples show, the directed graph representations can be
examined in a number of ways to detect patterns of theoretical or practical interest.
Potential analyses not described in the present paper include performing tree
decompositions with junction trees on an undirected version of the graph, finding shortest
paths between any two moves, comparing alternate paths between vertices, and
comparing graphs across domains (e.g., math versus science).
5. GENERAL DISCUSSION
In a recent paper, Ohlsson and colleagues (2007) questioned the widely used ―code-and-count‖ method for analyzing tutorial dialogues. Code-and-count methodologies
investigate tutorial dialogues by (a) designing a coding scheme to classify tutorial speech
into dialogue moves, (b) computing the occurrence of each dialogue move, and (c)
attributing learning gains to the most frequent dialogue moves. They question the
assumption that sheer frequency of a move is predictive of its casual efficacy, and
recommend focusing on effective combinations of moves.
This paper advanced this goal by presenting a method to automatically identify
collaborative patterns in tutorial dialogues. We used the method to identify patterns of
student and tutor dialogue moves during lectures. Although we did not explicitly link the
patterns to learning gains, the focus on collaborative patterns, beyond proportional
occurrence of dialogue moves, represents an important advance in the analysis of tutorial
approach, discusses some important limitations, delves deeper into the structure of
collaborative lectures, and discusses some applications of this research towards the
development of intelligent tutoring systems.
5.1 Advantages of Pattern Mining Method
Our pattern mining method consisted of identifying significant two-step excitatory
transitions between dialogue moves, integrating the transitions into a directed graph
representation, visually and statistically analyzing the directed graph for collaborative
patterns, and generating and testing data-driven hypotheses from the directed graph. This
method has a number of advantages over alternate methods such as dependent adjacency
analysis, Hidden Markov Models (HMMs), and sequential association rule mining.
Our use of the likelihood metric (Equation 2) to identify significant move transitions
has three significant advantages. First, the metric explicitly controls for base rates,
thereby eliminating spurious transitions that can be attributed to a mere capitalization on
chance. Second, the metric yields a normalized score, thereby making it possible to
compare any two transitions, including transitions with different antecedents and
consequents. This comparison is not afforded by adjacency pair analysis. The third
advantage of the likelihood metric is that it provides a standard way of computing effect
sizes [Cohen 1992]. In contrast, association rule mining algorithms require arbitrary
specification of candidate generation thresholds such as support and confidence [Srikant and Agrawal 1996].
Another set of advantages pertains to our use of directed graph representations to
integrate significant excitatory transitions. Directed graphs offer the ability to visually
identify dialogue move clusters with distinct pedagogical and motivational functions
(e.g., information-transmission versus information-elicitation). It should be noted that the
enhanced ability to visually detect patterns is not arbitrary, but is attained by virtue of
visualization algorithms from the field of graph drawing [Di Battista et al. 1994; Herman
et al. 2000]. These algorithms use optimization techniques to pictorially depict a graph in
order to feature its important properties. For example, information-transmission moves
were spatially segregated from information-elicitation moves, so these two clusters could
be detected by a visual inspection. Visual pattern detection is difficult when complexity
of the graph increases, however, a variety of graph algorithms (e.g., connected
components) can be used in these situations.
It should be noted that HMMs also afford the ability to identify interesting
of patterns (i.e., hidden states) needs to be specified apriori. It is also the case that
additional complexities arise during HMM training when a large number of parameters
need to be estimated and there is no underlying theory to guide parameter initialization.
Our use of a directed graph to generate candidate patterns has several advantages over
other methods such as naïve pattern scanning and sequential association rule mining. The
naïve approach would entail scanning the data for every possible pattern. For example,
considering all combinations for two, three, and four-length paths yields 12,341 paths of
length two (i.e., three vertices and two edges), 123,410 paths of length three, and 962,598
paths of length four (1,098,349 paths in all). In contrast, a path analysis using our pattern
mining method would only require scanning 147 potential paths. In addition to the
obvious computational advantages, testing the significance of a reduced number of
candidate patterns alleviates concerns pertaining to false positives and Type I errors.
Although sequential association rule mining techniques reduce the number of
potential patterns, they do not provide the ability to conceptualize a graph on the basis of
strongly connected components, vertex degrees, junction trees, edge covers, vertex covers
and other potentially useful constructs. Simply put, directed graph representations are
more powerful than the item sets and rules provided by association rule mining.
5.2 Limitations of Pattern Mining Method
It is important to acknowledge some of the limitations with the pattern mining method
with respect to its application for educational data mining. Perhaps the most important
limitation pertains to the substantial data preparation activities that need to be performed
before the method can be applied. In particular, the spoken tutorial dialogues need to be
transcribed and coded for dialogue moves before they can be submitted to the pattern
mining algorithm. The transcription and data coding phases are laborious and time
consuming. They may also involve additional resource intensive subphases such as
checking transcription quality and assessing the reliability of the coding protocols.
Although this limitation would also apply to related pattern mining methods, the
substantial human resources that need to be invested before our pattern mining approach
can be successfully applied warrants further consideration.
One option to alleviate this limitation is to use automatic speech recognition for
speech-to-text transcription and automatic speech act classification to code the dialogue
moves. The automatic speech recognition option is somewhat less viable because
contemporary speech recognizers are still imperfect, particularly for real world
to 45.5% based on the system, domain, and testing environment [Hagen et al. 2007; Kato
et al. 2000; Leeuwis et al. 2003; Litman et al. 2006; Pellom and Hacioglu 2003; Rogina
and Schaaf 2002; Zolnay et al. 2007].
The news is more positive for automatic speech act classification as a number of
advances have been made. For example, Olney and colleagues used finite state
transducers to accurately classify student speech acts during tutoring sessions with
AutoTutor [Olney et al. 2003]. Hoque and colleagues used a combination of
acoustic-prosodic and discourse features to classify speech acts between two conversational
participants [Hoque et al. 2007]. We are currently exploring the possibility of
automatically classifying the student and tutor dialogue moves in our expert tutoring
corpus and have made some important advances along this front [Williams and D'Mello
in press]. Although we do not expect the automatic speech act classification system to
yield perfect results, we hope that it will be sufficiently accurate to not adversely affect
the fidelity of our pattern mining approach.
The second limitation of our approach lies in the excitatory transition detection phase.
Here, two-step significant transitions are identified and preserved for subsequent
analyses. The problem is that the current algorithm only considers exact matches between
adjacent dialogue moves. For example, the transition will only be counted
once in the following sequence: . The
second transition will be ignored because of the intermediate discourse marker
between the and moves. Naturalistic tutorial dialogues are rife with such
discourse markers, acknowledgments, and back-channel feedback. Their occurrence
might interfere with the discovery of more interesting relations such as the
transition. Fortunately, this limitation can be alleviated by either removing these
conversational oriented dialogue moves from the time series prior to the analysis or by
endowing the transition detection algorithm with approximate sequence matching
capabilities [Mount 2004]. These measures were not incorporated in our analysis of
tutorial lectures because we were interested in analyzing both the pedagogical (e.g.,
positive feedback, correct answer) as well as the conversational (e.g., ok,
acknowledgment) dialogue moves of the student and tutor.
A somewhat less important, but nevertheless noteworthy limitation pertains to the
construction of the directed graph from the significant two-step excitatory transitions. In
some cases the directed graph is unusually complex and consequently less amenable to
visual inspection. The human visual system is a highly attuned pattern detector, but it
because the graph for the lecture mode was not too complex. However, when we applied
our pattern mining approach to the scaffolding mode, the derived directed graph was too
complex for visual examination.
The problem is that the complexity of the graph increases proportional to the number
of significant excitatory transitions. Hence, one simple approach is to reduce the number
of edges in the directed graph. This can be achieved by constructing the directed graph
from a predefined number of edges; presumably the edges associated with the strongest
effects. This is precisely the approach we adopted in the present paper by only including
edges with medium to large effect sizes. Although this approach solves the complexity
problem by yielding simpler graphs, one undesirable side effect is that the number of
orphan nodes increases (i.e., nodes with no parents such as the node in Figure 1).
Therefore, advanced graph visualization methods might be necessary to produce directed
graphs that can be visually inspected.
5.3 The Collaborative Lecture
Having discussed some of the advantages and limitations of our pattern mining approach
we now discuss some of the insights we have discovered with respect to expert tutor
lectures.
It is widely acknowledged that scaffolded problem solving via mixed-initiative
dialogues underlie the merits of one-on-one tutoring [Chi et al. 1994; Graesser, Person
and Magliano 1995; Lepper et al. 1997; Puntambekar and Hubscher 2005; Rogoff and
Gardner 1984; Shah, Evens, Michael and Rovick 2002]. Hence, it is somewhat
counterintuitive that expert tutors devote substantial portions (30%) of the tutoring
sessions to lectures, because didactic lectures are not expected to yield impressive
learning gains [Chi 1996]
It might be the case that the high incidence of lecturing is one of the characteristics
that distinguish expert tutors from novice tutors and intelligent tutoring systems. This is
essentially an empirical question that can be addressed by comparing expert versus
non-expert tutors. However, this analysis is compromised by the fact that relatively few
studies have investigated expert tutors. The few studies that have documented the
strategies of expert tutors have typically focused on one or two experts with no objective
criteria of what constitutes expertise in tutoring [Person, Lehman and Ozbun 2007]. In
some of the studies, the expert tutors are PhDs with extensive teaching and/or tutoring
experience [Evens et al. 1993; Glass, Kim, Evens, Michael and Rovick 1999; Jordan and
