Cognitive Assessment in Coached Learning Environments

The Alberta Journal of Educational Research Vol. XLV, No. 4, Winter 1999,392-408

Carl Frederiksen

Janet Donin

McGill University

Cognitive Assessment in Coached

Learning Environments

An approach to cognitive assessment of problem-solving in complex computer-based tutorial environments is described. The approach is based on studies of expert tutoring and students' performance in natural tutoring situations in specific domains such as engineering and statistics. A model of expert tutors' knowledge in a domain of applied statistics was developed and used as a basis for a web-based computer coach that emulates human tutoring. Cognitive assessments are obtained from records of students' actions as they learn to apply particular components of the procedural knozvledge required to solve problems in the domain with the help of the computer tutor. Learning is evaluated by studying changes in these records of performance as students practice successive problem exercises. These assessments can then be used subsequently to predict students' unassisted performance in solving post-instruction transfer problems.

Cet article décrit une approche à l'évaluation cognitive de résolution de problèmes dans un contexte complexe de tutorat assisté par ordinateurs. L'approche est fondée sur des études de tutorat par des experts et le rendement d'apprenants dans des situations naturelles de tutorat pour des domaines précis tels le génie et les statistiques. On a développé un modèle des connaissances d'experts-tuteurs en statistiques appliquées qui a servi comme base pour un tuteur informatique qui imite, sur le Web, les tuteurs humains. On obtient des évaluations cogititives à partir du record des démarches entreprises par les apprenants qui assimilent des éléments des connaissances procédurales requises dans la résolution de problèmes à l'aide du tuteur informatique. L'apprentissage est évalué en étudiant les changements dans la perfor-mance des apprenants alors qu'ils tentent la résolution de problèmes successifs. Ces évalua-tions peuvent ensuite servir dans la prédiction du rendement post-instructif des apprenants qui résoudront des problèmes de transfert sans aide.

There is a g r o w i n g interest a m o n g educators i n problem-based, collaborative, a n d a p p r e n t i c e s h i p approaches to instruction. This has been accompanied b y a n increased interest o n the part of cognitive psychologists i n h o w to m o d e l s u c h c o m p l e x i n s t r u c t i o n a l situations a n d the learning processes that occur i n them. Interest i n coached, collaborative, a n d problemoriented m o d e s of i n struction has l e d to a cognitive apprenticeship m o d e l (Collins, B r o w n , & N e w

-Carl Frederiksen is a professor in the Department of Educational and Counselling Psychology. He previously held positions at the University of California, Berkeley, the National Institute of Education, and Rockefeller University. His current research specialization is in the study of discourse processing, situated cognition and learning, and learning through interactive discourse (e.g., in tutoring, problem-based learning groups, computer tutoring environments).

Janen Donin is an associate professor in the Department of Educational and Counselling Psychology and the Department of Second Language Education. Her field of specialization is discourse processing, learning, and the acquisition of information from written and oral language (in a first or second language).

Cognitive Assessment in Coached Learning Environments

m a n , 1989). T h i s m o d e l has p r o v i d e d a rationale for reconceptualizing h o w students learn a n d h o w computers m a y be used to s u p p o r t these modes of learning. C o m p u t e r - b a s e d l e a r n i n g environments currently are b e i n g designed to s u p p l e m e n t a n d enhance natural problem-based learning situations b y p r o v i d i n g tools a n d coaching s u p p o r t for learners that are designed explicitly to s u p p o r t these k i n d s of learning (Derry & Lajoie, 1993). The challenge is to d e s i g n tools that can s u p p o r t students' i n d i v i d u a l a n d collaborative d e v e l o p -m e n t of c o -m p l e x , r e a l - w o r l d k n o w l e d g e a n d s k i l l i n authentic d o -m a i n s of k n o w l e d g e a n d contexts of p r o b l e m - s o l v i n g .

A s cognitive m o d e l s of l e a r n i n g have e x p a n d e d i n their capacity to address c o m p l e x , r e a l - w o r l d cognition, there has been a g r o w i n g recognition of the need for assessments of students' cognitive processes, learning, k n o w l e d g e , a n d s k i l l d e v e l o p m e n t that are more appropriate to these natural situations of coached, collaborative, a n d problem-based l e a r n i n g than are standard assess-ments of ability or achievement (Snow & L o h m a n , 1993). This recognition has been apparent i n calls for alternative approaches to assessment (reviewed b y B i r e n b a u m , 1995). A m o n g the characteristics that have been advocated for alternative assessment are: (a) assessment s h o u l d be integrated w i t h instruc-tion; (b) it s h o u l d be transparent, that is, it s h o u l d help students learn to m o n i t o r , self-evaluate, a n d reflect o n their o w n performance; (c) assessment tasks s h o u l d be authentic, extended over time, m e a n i n g f u l , a n d challenging; (d) students s h o u l d have access to tools, resources, a n d coaching s u p p o r t d u r i n g assessment of their performance a n d learning; (e) assessment s h o u l d be diagnostic, p r o v i d i n g i n f o r m a t i o n about students' k n o w l e d g e , cognitive processes, misconceptions, a n d errors d u r i n g performance of p r o b l e m - s o l v i n g tasks; (f) students' k n o w l e d g e a n d their ability to a p p l y (transfer) their k n o w -ledge to n e w or n o v e l p r o b l e m s s h o u l d be assessed; a n d (g) ability to col-laborate effectively w i t h others i n s o l v i n g p r o b l e m s s h o u l d be assessed. These characteristics of alternative assessment address limitations of traditional test-i n g approaches a n d p r o m o t e more authenttest-ic assessments of k n o w l e d g e , learn-i n g , cognlearn-itlearn-ive processlearn-ing, a n d p r o b l e m - s o l v learn-i n g sklearn-ill learn-i n c o m p l e x natural d o m a i n s a n d contexts of taskoriented action (e.g., i n higher education). H o w -ever, experience w i t h alternative assessment has raised concerns about issues of objectivity, reliability, a n d validity of s u c h assessments, particularly w h e n the situation is " h i g h stakes" for the students b e i n g assessed. F i n d i n g objective w a y s to achieve m o r e authentic cognitive assessments represents one of the great challenges for the field of measurement a n d evaluation.

T h e need for cognitively v a l i d assessments has been recognized b y re-searchers i n the measurement c o m m u n i t y w h o are attempting to d e v e l o p measurement m o d e l s a n d tests that can better assess cognitive processes a n d k n o w l e d g e structures (Frederiksen, M i s l e v y , & Bejar, 1993). This p r o b l e m can be seen f r o m b o t h a cognitive perspective a n d a measurement perspective. F r o m the cognitive perspective the p r o b l e m is h o w can w e use observations of students' performance i n problem-based instructional environments (such as t u t o r i n g a n d coached instruction) to p r o v i d e diagnostic i n f o r m a t i o n about their cognitive processes as they acquire k n o w l e d g e a n d effective p r o b l e m -s o l v i n g -skill-s t h r o u g h coached practice i n -s o l v i n g p r o b l e m -s i n a d o m a i n ? F r o m the measurement perspective the p r o b l e m is h o w can measurement models

C. Frederiksen and ]. Donin

a n d techniques be used to p r o v i d e cognitive assessments of skill a n d k n o w -ledge acquisition that go b e y o n d the performance of specific tasks a n d p e r m i t generalizations a n d p r e d i c t i o n of performance o n other tasks i n a d o m a i n .

W o r k has already b e g u n o n the first p r o b l e m (Frederiksen, Glaser, L e s g o l d , & Shafto, 1990; S n o w & L o h m a n , 1993), and some of this w o r k has g i v e n h i g h p r i o r i t y to the authenticity of the tasks a n d learning environments i n w h i c h diagnostic assessment is carried out (Lesgold, Lajoie, L o g a n , & Eggan, 1990). W o r k o n the second p r o b l e m has typically required the construction of tasks or items that are representative samples of tasks i n a d o m a i n , a n d the application of m e a s u r e m e n t m o d e l s s u c h as item response theory (IRT) to construct statis-tical m o d e l s that are capable of estimating a n examinee's level o n a latent ability scale (or classification into a n o m i n a l or ordinal category) that pertains to the d o m a i n of tasks b e i n g s a m p l e d . A t t e m p t s are b e i n g m a d e to construct items a n d I R T m o d e l s that can p e r m i t some degree of cognitive assessment ( M i s l e v y , 1993). F o r e x a m p l e , Embretson (1993) has constructed IRT m o d e l s for spatial rotation tasks that decompose these tasks into subskill components.

T h i s article is concerned w i t h both problems. O u r research initially focused o n the first p r o b l e m : diagnostic assessment of students' performance as they learn to solve c o m p l e x problems w i t h the help of a tutor. C u r r e n t l y it is a d d r e s s i n g the second p r o b l e m b y a p p l y i n g a d y n a m i c assessment approach that m a k e s use of performance data that are gathered as i n d i v i d u a l s practice s o l v i n g p r o b l e m s w i t h the s u p p o r t of a coach or tutor. Measurements can be d e r i v e d f r o m s u c h performance data, a n d these m a y be used to predict unas-sisted performance o n n e w transfer problems presented f o l l o w i n g instruction. H o w e v e r , because psychometric generalizations t y p i c a l l y d e p e n d o n samples of tasks (i.e., problems), the question can be posed: T o w h a t extent can w e predict performance o n a criterion p r o b l e m f r o m d y n a m i c assessment data gathered d u r i n g instruction ( C a m p i o n e & B r o w n , 1990)? If d y n a m i c assess-ments of g r o w t h i n proficiency across a set of practice problems can predict criterion performance o n n e w problems, then w e m a y be able to c l a i m some degree of generalizability w i t h o u t the need to test i n d i v i d u a l s o n large samples of p r o b l e m s . If a n i n d i v i d u a l possesses a stable a n d integrated d o m a i n k n o w -ledge structure a n d s k i l l i n a p p l y i n g it to p e r f o r m tasks effectively i n a d o m a i n , this m i g h t constitute a latent trait-like characteristic of the i n d i v i d u a l that c o u l d be assessed a n d that w o u l d reliably predict performance on tasks i n the d o -m a i n . T h i s see-ms to be the i-mplicit -m o d e l for assess-ment i n -m a n y d o -m a i n s of professional e d u c a t i o n .

A s a first step to a d y n a m i c a n d cognitive assessment of performance, w e have been s t u d y i n g students' p r o b l e m - s o l v i n g and learning processes i n natural t u t o r i n g situations i n a d o m a i n of a p p l i e d statistics. The topic is analysis of variance ( A N O V A ) . W e have f o u n d that it is possible to construct c o m -puter-based coached-learning environments that emulate m a n y aspects of the interactive s u p p o r t coaches p r o v i d e to students i n these natural tutoring situa-tions. W e have i m p l e m e n t e d s u c h a c o m p u t e r e n v i r o n m e n t i n the d o m a i n of a p p l i e d statistics (as an A N O V A Tutor) a n d are currently c o n d u c t i n g studies to assess students' l e a r n i n g a n d p r o b l e m - s o l v i n g w h i l e u s i n g the tutor. The A N O V A T u t o r incorporates a m o d e l of the expert p r o b l e m - s o l v i n g k n o w l e d g e that w a s used by an experienced tutor to coach students i n h o w to solve

Cognitive Assessment in Coached Learning Environments

p r o b l e m s i n the d o m a i n of analysis of variance. W h e n used i n conjunction w i t h practice data sets a n d appropriate statistical analysis software, the A N O V A T u t o r p r o v i d e s a w e l l d e f i n e d , interactive, problembased learning e n v i r o n -m e n t i n w h i c h students' learning a n d p r o b l e -m - s o l v i n g -m a y be assessed u n d e r realistic conditions of coached practice a n d instruction.

Development of a Computer Tutor in Statistics

In a p r e v i o u s s t u d y of students' l e a r n i n g i n a face-to-face t u t o r i n g situation i n engineering, analyses were m a d e of changes i n i n d i v i d u a l students' a n d the instructor's performance d u r i n g coached practice i n s o l v i n g a sequence of p r o b l e m s that increased i n c o m p l e x i t y (Frederiksen, R o y , & Bédard, 1999). The changes i n the tutor's explanations a n d scaffolding of students' p r o b l e m s o l v -i n g that were observed as a student g a -i n e d k n o w l e d g e a n d s k -i l l p r o v -i d e d a n almost classic example of cognitive apprenticeship. O n the first p r o b l e m the tutor demonstrated h o w to solve a n example of a n engineering p r o b l e m (in the d o m a i n of m e c h a n i c a l engineering) a n d e x p l a i n e d h e r solution procedures to each student as she s o l v e d the p r o b l e m . A n a l y s i s of the tutor's discourse and p r o b l e m - s o l v i n g actions enabled us to d e v e l o p a hierarchical m o d e l of the structure of the p r o b l e m - s o l v i n g procedures that the tutor m o d e l e d f o r the students. It also enabled u s to m o d e l h o w the tutor explained the component procedures t h r o u g h her dialogue w i t h the students. O n subsequent problems, the tutor s w i t c h e d to a coaching role i n w h i c h she asked each of the students to try to solve a n e w practice p r o b l e m b y themselves. A s a student attempted to solve the p r o b l e m , she p r o v i d e d guidance a n d help w h e n she felt it w a s n e e d e d . A n a l y s i s of the students' p r o b l e m - s o l v i n g protocols a n d discourse interaction w i t h the tutor i n these coaching sessions revealed a learning process i n w h i c h the students attempted to a p p l y the c o m p o n e n t procedures to n e w p r o b l e m examples ( w i t h guidance f r o m the tutor) a n d obtained help f r o m the tutor w h e n it w a s needed to complete the p r o b l e m . B y the third p r o b l e m a l l the students r e q u i r e d less help f r o m the tutor.

T h e results of this s t u d y of tutoring i n engineering i n s p i r e d the approach w e are t a k i n g to cognitive assessment i n the present research. This a p p r o a c h combines d y n a m i c assessment of l e a r n i n g (i.e., p r o v i d i n g coaching s u p p o r t to students w i t h c h a l l e n g i n g p r o b l e m s a n d r e c o r d i n g students' g r o w i n g ability to solve p r o b l e m s w i t h o u t assistance) w i t h diagnostic assessment of students'

prob-lem-solving processes (based o n a n expert m o d e l of conceptual k n o w l e d g e a n d

p r o b l e m s o l v i n g procedures i n a d o m a i n ) . O u r a p p r o a c h is to d e v e l o p a c o m -puter coach that emulates aspects of h u m a n tutoring a n d assessment tech-niques for this coaching e n v i r o n m e n t that are parallel to those used to evaluate students' l e a r n i n g i n the h u m a n t u t o r i n g situation. T h e engineering tutoring s t u d y demonstrated h o w a n expert m o d e l of c o m p l e x p r o c e d u r a l d o m a i n k n o w l e d g e c a n be constructed u s i n g k n o w l e d g e - m o d e l i n g a n d discourse-anal-ysis tools f r o m cognitive science to analyze the discourse a n d p r o b l e m - s o l v i n g actions of the tutor as she demonstrated a n d e x p l a i n e d the p r o b l e m - s o l v i n g p r o c e d u r e . The next step was to a p p l y these methods to analyze tutoring i n the n e w d o m a i n ( A N O V A as taught to doctoral students i n educational a n d c o u n -seling p s y c h o l o g y ) a n d d e v e l o p a c o m p u t e r coach based o n a n analysis of expert t u t o r i n g i n the d o m a i n .

C. Frederiksen and /. Donin

T o construct the c o m p u t e r coach w e used a p r o g r a m w e d e v e l o p e d called T u t o r B u i l d e r to construct a database of tutoring knowledge f r o m o u r analysis of the tutor's demonstrations a n d explanations of h o w to solve data-analysis p r o b l e m s i n statistics. T h i s database consists of a hierarchical data structure c o n t a i n i n g a large n u m b e r of H T M L files that contain i n f o r m a t i o n about c o m -ponent procedures (similar to that p r o v i d e d b y the h u m a n tutor). W h i l e stu-dents r u n a statistics p r o g r a m (e.g., S Y S T A T ) to analyze a practice data set on their computers, they can r u n the A N O V A T u t o r p r o g r a m concurrently o n a remote server u s i n g a w e b browser. The students can use the browser to v i e w and interact w i t h a hierarchical g u i d e to the organization of p r o b l e m - s o l v i n g actions. T h e y can also v i e w m u l t i m e d i a messages e x p l a i n i n g particular steps i n s o l v i n g data analysis problems. In this w a y students can use the tutor to obtain instruction a n d c o a c h i n g s u p p o r t as they practice s o l v i n g data analysis p r o b l e m s o n their c o m p u t e r . The c o m p u t e r tutor a n d statistics software togeth-er p r o v i d e a w e l l - d e f i n e d , coached, problem-based learning e n v i r o n m e n t i n statistics.

Description of the Statistics Tutoring Situations

W e have been s t u d y i n g several types of tutoring situations i n statistics. These include: (a) face-to-face t u t o r i n g of i n d i v i d u a l students (in w h i c h the tutor and student shared the use of statistical software a n d the tutor used the software and p r e p a r e d data files to demonstrate a n d explain h o w to use A N O V A to solve data analysis problems); a n d (b) n e t w o r k e d one-on-one tutoring (in w h i c h the tutor i n t r o d u c e d a student to A N O V A i n the same m a n n e r as i n the face-to-face c o n d i t i o n , but c o m m u n i c a t i n g b y means of videoconferencing software). In b o t h of these situations doctoral students' i n educational a n d c o u n s e l i n g p s y c h o l o g y shared the use of statistical software ( S Y S T A T ) as they learned to use A N O V A to solve a series of problems consisting of data sets to be a n a l y z e d . A stack of " b l a c k b o a r d " representations (e.g., graphics, equations, tabular, a n d other displays) were p r o v i d e d as resources for the students as they were tutored b y a n experienced faculty m e m b e r i n A N O V A theory a n d m e t h -ods, a n d i n h o w to use S Y S T A T as a tool for data analysis.

By s t u d y i n g t u t o r i n g i n situations i n w h i c h c o m m u n i c a t i o n w i t h the tutor was either face-to-face or by means of videoconferencing, w e were attempting to b r i d g e the gap between authentic cognitive apprenticeship situations (e.g., face-to-face tutoring) a n d c o m p u t e r environments that simulate the conditions of natural expert tutoring. O u r results indicated that n e t w o r k e d tutoring is s i m i l a r to face-to-face tutoring a n d that the same models of expert k n o w l e d g e and t u t o r i n g s k i l l a p p l y across all these situations. The next step is to see if a c o m p u t e r coach can be i n t r o d u c e d to emulate coaching a n d explanation func-tions of a h u m a n tutor and to analyze students' learning ( i n d i v i d u a l a n d collaborative) i n these environments. C o m p u t e r coaches designed i n this w a y can offer m a n y of the benefits of h u m a n tutoring. T h e y can be used over a n e t w o r k . M o r e o v e r , they c o u l d be w e l l suited for use i n university courses as a c o m p l e m e n t to the activities of instructors. In the present context w e are interested i n the potential use of these environments for cognitive assessment purposes.

(1) ANOVA Frame

(11) Research Problem

(111) One-way (1111) Subjects (1112) Method (1113) Ind. var. (1114) Dep. varv

(1115) Hyp.

(161) State Undings (162) Interpret findings (163) State limits on

findings (1621) Interpret ANOVA tests (1622) Interpret desc. results

(13211) Construct tactor A pranh -> (13212) Construct factor

B graph (13213) Construct

inter-action graph • (13214) Interpret plots

f (sig. effects) -> (13215) Interpret plots

(trends)->

(12211) size nj from A (12212) size ni( from B (12213) size njk from A ' B

(132141) EaclfJLâ (132142) Factor B (132143) Interaction

'i

('43) Est s c o i a model (1431) on«-way

model (14311) write terms (14312) write equ. (143111) Xij« (1*31 12) iCar (143113) est alpha)

•>1-4 (143114) est Ei(j)

•>1.2

(144) Partition ol Sum oi Squares I 1 4 4 1 Ì o n a - w M - )1

(1442) two main

10.**

(1443) two-way , 1,2.3,4.5

(1433) two-way model (with Int.)

(1422) Additive mair ' effects model (14221) X.,-(14222) g mean (14223) alpha) (14224) b e t a * (14225) Ei(j)

(1432) mam affects (additive) model

(1423) two-way model with interactions

(142311X1,-(14232) g ma an (14233) alpha | (14234) baia k (14235) Int |k (14226) Eiü)

;

(1478) Decision (14781) Raj Ho: F>Fcril (14783) Acc HO F<Fcr.t

(143211) Xi|> (143212) ibar.. (143213) est alpha )->i (143214) est bete a »

(143215) est Ei(j)->1-4 {143311 writa t a r m i (14332) w n i a i a u .

1 3.4

(148) Preplanned tests ot painvise c o n v u t s (14711) Ho: J e f f (14712) Ho: K e f f (14713) Ho: J K int

p

?

a S'

(14721) HI: srg j eH->2 (14722) HI: s l g k e l f (14723) HI: stg intrjk)

(14785) Acc HI: F<Ferit (14782) Rei Ho: p<alpha (14784) A c c Ho: p>alpha (14786) Acc HI: p x l p h a

(1465) O F

(1433151) long lormo ., (2.>1.2).(3->1.2). 4 (1433152) ungrouped 1orm-> (1433153) snort l0rm->l-4

(1*3311) XiUkJ (143312) »bar... (143313} aal alpha )->l 2 (143314) est beta k - » l , 2 (143315) est Int |k

(143316) m£ma->1->{1 - 4). 2

(1466) Mull R (1467) W - > (1468) Pooled es! ol S E > (1466} N-> (146 10) Pooled est ol SO

14751) D r e » e h e p e - » l | n 4 8 l ) Select effect (14752) Ratio ol x * ' d . l . lor c o n t r a s t s

X (1482) Deline contrast 1 ( 1 4 7 5 2 1 ) ^ 0 , ^ , 2 « . ! .1"93" ;0, ™ " '

\ * \ statistic«-> y i 4 7 5 2 l ) D e n l sX2 (1484) Test coni/asi

hypothesis->

(14811) Select effect (14812) Define

contrast ->l (147611) f _ t a i S 5 J (147612) F ( a - O l i (147613F)o.o<acl (14761) Find values

(14762) Place values on dist. plot

(1433161) Xi|K

(1433162) . pred. pn. (14331621) • olher para ests.->1 (14331622) - c e a mean

3.6 * 32 (146 11) L S esrs ol means N

(148 12) SE'S Ol L S means > (14651) dl(A)

(14662) dl(B) (14653) dl(A-B) (14654) d f ( e )

-(146 11) L S esrs ol means N

(148 12) SE'S Ol L S means > (14651) dl(A)

(14662) dl(B) (14653) dl(A-B) (14654) d f ( e ) -(14651) dl(A) (14662) dl(B) (14653) dl(A-B)

(14654) d f ( e ) - — l * | ( i ) M e J J i î l f l r j 36

Cognitive Assessment in Coached Learning Environments

Modeling Expert Knowledge: The Development of an ANOVA Procedure Frame

A s i n the p r e v i o u s s t u d y of tutoring i n engineering, a n expert m o d e l of proce-d u r a l k n o w l e proce-d g e i n the proce-d o m a i n of analysis of variance w a s proce-d e v e l o p e proce-d . T h i s m o d e l w a s d e v e l o p e d initially f r o m a procedure frame a n d semantic analysis (Frederiksen, 1986) of tutorial dialogue between a n experienced tutor a n d a n o v i c e student i n a context of tutor demonstration a n d explanation of analysis of variance. T h i s discourse occurred over three tutoring sessions i n w h i c h the tutor m o d e l e d p r o b l e m - s o l v i n g procedures a n d p r o v i d e d a conceptual ex-p l a n a t i o n of one- a n d t w o - w a y analysis of variance for a single n o v i c e student. Subsequently, data f r o m the tutoring of other students were used to i m p r o v e the m o d e l . The discourse a n d protocol analysis led to a relatively complete a n d detailed "expert m o d e l " of analysis of variance procedures.1 This m o d e l is

d i s p l a y e d i n F i g u r e 1.

T h e p r o c e d u r e frame represents a c o m p l e x procedure b y d e c o m p o s i n g it into a hierarchy of actions a n d goals. A t the t o p level, s o l v i n g a data analysis p r o b l e m i n v o l v e s six c o m p o n e n t procedures (or tasks): (a) d e f i n i n g the re-search p r o b l e m , (b) s p e c i f y i n g the data, (c) c a r r y i n g o u t a descriptive analysis of the data, (d) p e r f o r m i n g a n A N O V A w i t h the data, (e) c o n d u c t i n g a n y post-hoc analysis, a n d (f) d r a w i n g conclusions based o n the results obtained f r o m p r e v i o u s steps. E a c h of these m a i n procedures is c o m p o s e d of s u b -procedures. F o r example, procedure (d), p e r f o r m i n g the A N O V A , is c o m p o s e d of eight m a i n subprocedures to be p e r f o r m e d : (a) s p e c i f y i n g the research design, (b) s p e c i f y i n g a linear m o d e l for scores o n the dependent variable, (c) o b t a i n i n g least squares estimates of the g r a n d m e a n a n d all effects i n the linear m o d e l , (d) p a r t i t i o n i n g the total s u m of squares according to the A N O V A m o d e l , (e) p r e p a r i n g a n A N O V A table for o r g a n i z i n g results, (f) c o m p u t i n g A N O V A statistics, a n d (g) c o n d u c t i n g F tests. A n a d d i t i o n a l procedure of c o n d u c t i n g p r e p l a n n e d tests of contrasts is o p t i o n a l .2

T h e tutor's coverage of the analysis of variance procedure w a s impressive for its consistency, completeness, a n d explicitness. T h e o n l y major topics not covered b y the tutor were (a) the procedures for testing assumptions i n A N O V A (procedure node 124 i n Figure la), a n d (b) procedures for p r e p l a n n e d tests of contrasts (procedure node 148 i n Figure l b ) . Concepts a n d theory were i n c l u d e d i n the f o r m of e m b e d d e d explanations of conceptual k n o w l e d g e a n d reasoning u n d e r l y i n g procedures used i n the analysis. T h e tutor's discourse p r o v i d e d detailed explanations of the relevant statistical theory a n d concepts, related this conceptual k n o w l e d g e to appropriate steps i n the procedure, a n d m o d e l e d statistical reasoning associated w i t h a p p l y i n g this k n o w l e d g e . T h e tutor's descriptions of procedures i n c l u d e d the same k i n d s of i n f o r m a t i o n about procedures as were f o u n d for the engineering tutor. These i n c l u d e d :

Goals (to be attained for a current procedure); Actions (to be performed); Results

(obtained f r o m enacting the procedure); Explanations (several types); a n d Tools (software o r other tools used to carry o u t a procedure a n d descriptions of h o w to use them). Explanations i n c l u d e d : Representations (formulas, equations, tables, graphs); Concepts (statistical concepts such as m e a n square, F ratio, n u l l hypothesis); Theory (statistical theory u n d e r l y i n g a procedure); Procedure (the rationale for a step i n the procedure); Results (explaining the m e a n i n g of

C. Frederiksen and /. Donin

results); a n d Pragmatic Explanations (practical i n f o r m a t i o n about contexts i n w h i c h a p r o c e d u r e is a p p l i e d ) .

W h a t w a s p a r t i c u l a r l y striking about the statistics tutoring situation as an e x a m p l e of cognitive apprenticeship w a s that this wealth of i n f o r m a t i o n about h o w to solve data analysis problems w a s e m b e d d e d i n contexts i n w h i c h it w a s b e i n g used to u n d e r s t a n d problems a n d a p p l y appropriate methods to solve them. O u r research demonstrated that it w a s possible to a p p l y cognitive m o d e l i n g a n d discourse analysis techniques successfully i n this large a n d c o m p l e x d o m a i n a n d that these techniques c o u l d be used to construct a detailed m o d e l of p r o b l e m - s o l v i n g k n o w l e d g e i n the A N O V A d o m a i n . This m o d e l c o u l d be used to d e v e l o p a database of tutoring k n o w l e d g e for a c o m -puter tutor.

Assessment of Students' Learning

T o assess students' l e a r n i n g f r o m their experiences in the statistics tutoring sessions, the students participated i n a third p r o b l e m - s o l v i n g session f o l l o w i n g the second t u t o r i n g session. This session w a s designed to resemble an i n -d i v i -d u a l c o a c h i n g session, but the coach (a gra-duate stu-dent research assistant) p r o v i d e d o n l y m i n i m a l assistance to the students. The students w o r k e d i n the same e n v i r o n m e n t , b u t were g i v e n a n e w data set to analyze. The coach g u i d e d each student i n d i v i d u a l l y t h r o u g h the analysis by means of a series of ques-tions that c o r r e s p o n d e d to higher-level nodes i n the procedure frame. If the student w a s unable to answer a question, a clarification of the question w a s p r o v i d e d , but n o other f o r m of assistance (e.g., hints, instructions, or a n actual d e m o n s t r a t i o n of that step i n the analysis) was p r o v i d e d .

The m a p p i n g of assessment questions onto the procedure frame is g i v e n by the n u m b e r e d boxes enclosing nodes i n Figure 1. Questions are n u m b e r e d consecutively w i t h i n topics: T o p i c 1 covered the research p r o b l e m a n d data. T o p i c 2 covered the descriptive analysis. T o p i c 3 i n c l u d e d questions about the A N O V A itself. T o p i c 4 c o m p r i s e d questions related to the conclusions. The sequence of questions m a p p e d onto the frame gives a trace analysis of h o w the " c o a c h " g u i d e d the student t h r o u g h the procedures. A n assessment of one student is indicated i n the frame. Boldface n o d e labels are used to m a r k proce-dures that were successfully executed by the student; u n d e r l i n e d boldface nodes i d e n t i f y steps i n the procedure for w h i c h the student m a d e errors or d i s p l a y e d misconceptions. A l l other nodes that were the subject of questions (i.e., those enclosed i n boxes) are components of the procedure that the student w a s unable to execute. T h i s figure illustrates h o w a trace analysis (in an assess-ment or coaching situation) can identify procedures that have been learned, the location of misconceptions a n d errors, and procedures that the a student can-not execute w i t h o u t coaching support. S u c h a n assessment keeps track of the extent a n d k i n d s of coaching s u p p o r t that a student m a y need for particular nodes (i.e., procedures) that have not yet been mastered.

The expert procedure frame p r o v i d e s a n e t w o r k m o d e l of the p r o c e d u r a l k n o w l e d g e that is c o m m u n i c a t e d to the student and that is needed to u n d e r -stand a n d solve problems i n the d o m a i n , a n d the trace analysis reveals h o w the tutoring session w a s organized to g u i d e the students' p r o b l e m - s o l v i n g .3 If a

c o m p u t e r coach c o u l d be d e v e l o p e d to p r o v i d e guidance a n d coaching s u p p o r t for students' p r o b l e m - s o l v i n g that are s i m i l a r to that of a h u m a n coach, it ought

Cognitive Assessment in Coached Learning Environments

to be possible to adopt a s i m i l a r a p p r o a c h to diagnostic assessment i n the c o m p u t e r e n v i r o n m e n t . I n coaching, the trace analysis is p r i m a r i l y a record of the sequence i n w h i c h the student a p p l i e d specific procedures to solve the p r o b l e m . T h u s it p r o v i d e s a n assessment of the student's ability to a p p l y the procedures systematically to solve a particular p r o b l e m . O u r A N O V A tutor is d e s i g n e d to emulate the tutor's guidance (by means of a m a p of the hierarchical p r o c e d u r e frame) a n d the tutor's explanations of c o m p o n e n t tasks (by means of buttons p r o v i d i n g access to different k i n d s of i n f o r m a t i o n about the proce-dure).

Tutor Builder: Constructing a Database of Tutoring Knowledge

T h e T u t o r B u i l d e r software is currently b e i n g used to develop a database of t u t o r i n g k n o w l e d g e that is structured according to the expert procedure frame that w a s constructed i n o u r s t u d y of t u t o r i n g i n statistics. The database is o r g a n i z e d i n terms of the procedure hierarchy, a n d associated w i t h each node (i.e., c o m p o n e n t procedure) there are semantic fields that contain text a n d g r a p h i c i n f o r m a t i o n about the procedure. (This i n f o r m a t i o n is based o n seman-tic i n f o r m a t i o n that w a s expressed b y the tutor t h r o u g h his contributions to the tutorial dialogue.) Semantic fields p r o v i d e t w o k i n d s of assistance to the stu-dent: Instruction a n d C o a c h i n g Assistance. Instruction provides several k i n d s of semantic d e s c r i p t i o n of c o m p o n e n t procedures: G o a l Descriptions, P r o b l e m State D e s c r i p t i o n s , A c t i o n Descriptions, T o o l Instructions, Theory E x p l a n a -tions, C o n d i t i o n s (necessary for c a r r y i n g o u t the step i n the procedure), a n d Result Descriptions. Coaching Assistance is p r o v i d e d i n the form of Questions, Clarifications, a n d H i n t s . A s a n example, consider the Tutor W i n d o w presented i n F i g u r e 2. T h e p a n e l at the l o w e r left of the W i n d o w provides the student w i t h a " P r o c e d u r e M a p " of the hierarchical structure of the procedure (like a site m a p o n the internet). T h e student c a n select any node (i.e., c o m -p o n e n t -procedure) f r o m the m a -p , a n d then "request" coaching assistance or instruction p e r t a i n i n g to the selected procedure b y selecting a type of assis-tance (from the p a n e l i m m e d i a t e l y above the Procedure M a p ) . F o r example, types of instruction i n c l u d e a statement of the G o a l of carrying out a n A N O V A , a d e s c r i p t i o n of the k i n d s of Results obtained f r o m d o i n g an A N O V A , o r a general i n t r o d u c t i o n to the relevant Theory. Coaching is p r o v i d e d at several different levels a n d the student c a n select the level of assistance he o r she desires: Q u e s t i o n s , Clarifications, o r H i n t s (there are u p to three levels of hints available). O n c e a type of instruction o r coaching assistance has been selected, the tutor d i s p l a y s the relevant text o r graphic i n f o r m a t i o n i n the right p a n e l of the w i n d o w . T h e buttons i n this p a n e l p r o v i d e access to other types of instruc-tion o r levels of coaching. It is possible to p r o v i d e graphics, sound (and even m o v i e clips) to a c c o m p a n y these w i n d o w s . G l o s s a r y items are highlighted i n the presented text, a n d a n alphabetized listing of a l l glossary items is a l w a y s available b y means of a p o p - u p selection w i n d o w .

A n enhanced v e r s i o n of the tutor ( w h i c h is u n d e r development) w i l l i n -clude facilities for students to (a) s u b m i t their w o r k a n d (b) perform a g u i d e d self-evaluation b y c o m p a r i n g their w o r k w i t h the "tutor's results." A f t e r stu-dents h a v e c o m p l e t e d a step (at any level i n the hierarchy), they w i l l be able to s u b m i t their w o r k (in the f o r m of a text file), a n d ask to v i e w the tutor's results at this step. T h i s opens a Result W i n d o w containing t w o kinds of

C. Frederiksen and /. Donin

, A file Edlt.Jrtt« CrppliisSttts •Wlndow^.HtlpaA-ü.... • • . HjUßic,

o r Anova

Tutor

INSTRUCTION COACHING

G o a l theory Problom Condition A c t i o n Pat vit Tool

Q u o i t i o n Clarification Hint

f m k ruAro: ntittrch trateln.

Hutttfci

Ib 4 i r î i _ . _

C l P l M í h i_ Í i ü i ü i

l U f t r e h Qbl*ctiT««

T*p-ttT C l m J i c t t i o n

llrltP.tJ '

Factor I Factor 2

cogtosytfem Fufrfci Jovui tetsnmena' Fou.

1070' 107«' 1970 :

i g ? e1

1970 19¾ '

a—.iff.i

F MO lo

1970 1970 1970 '

Goal: Research Design

To carry our in Analyst» of Variance, you need c s tan v i * a precise (lescr.poon oí the research design of the study m vluch the daia v e n collected includine the methodology and the specific research quesnons or hypotheses under invtsogiocn

Flxst you need e determine vhether a one-va v class if ic anon of subjeco lnio groups (oi conditions) is Involved, or a rvo-vav cross classificatori of s'jtsecö (In vhich each subject is classified accordine o r vo

'dimensions' of classifica coro If the (roup* of Indiquais are classified according o one facer for classification dimensioni ve refer io this as a one-facpr desMn. If (hey also are classified In arms of a second facer fot classification 'dimension'), ve say thai ve have a rvp face: des»n

(Inadvanced vorlud, m

Analysts of Variance ) e than rvo classifications can be •• ...- using

Problem Ac opa Tools Theory Cond mens fttsuji Que) Opr.

I SYSTAT Analysts ! _IBB

HMHLVSIS DF VflRIHHCE SUV-CF-SOUHFCS I» it m SO »»*

210.12300 342.33333 2421 -4 I fjC7

210.12900 271.10667 30.00813

3.72733 0.01095 7.39113 0.00127 I

Figure 2. The ANOVA Tutor environment: Screen shot of the computer desktop with the ANOVA Tutor running (in a browser window) and with Data Editor and Analysis Windows displayed by concurrently running the statistics application SYSTAT.

specific i n f o r m a t i o n : first, a correct result c o r r e s p o n d i n g to that step i n the

p r o c e d u r e for the p r o b l e m , a n d second, a checklist of items required i n a successful result a n d a list of c o m m o n errors. Students w i l l be able to compare their solutions at that step w i t h the correct result a n d check those features that are present i n their solutions. A student w i l l be g u i d e d to relevant instruction based o n this evaluation. This feature is designed to d e v e l o p s k i l l i n self-m o n i t o r i n g a n d e v a l u a t i o n of p r o b l e self-m - s o l v i n g steps.

A s s h o w n i n F i g u r e 2, the computer coach is r u n concurrently w i t h the statistical analysis software. T h e student is presented w i t h a p r e p a r e d data set and a d e s c r i p t i o n of the data, h o w the data were obtained, a n d the purposes of the s t u d y . T h e p r o b l e m s o l v i n g task i n v o l v e s p l a n n i n g a n d executing a c o m -plete analysis u s i n g the statistical software, a n d s u b m i t t i n g a report containing results obtained at each step (the report is o r g a n i z e d as the sequence of student responses to t u t o r - s u p p l i e d questions). T h e student w i l l be able to cut a n d paste i n f o r m a t i o n f r o m a n y tutoring w i n d o w , or f r o m the statistical e n v i r o n -m e n t (e.g., results f r o -m the o u t p u t w i n d o w ) into the te-mplate, a n d enter i n f o r m a t i o n u s i n g text e d i t i n g functions. This e n v i r o n m e n t closely parallels natural p r o b l e m - b a s e d classroom a n d tutoring situations. The students w o r k on p r o b l e m exercises w h i l e u s i n g the tutor. T h e goal is to l e a m to p r o d u c e correct analyses i n d e p e n d e n t l y a n d p r o d u c e accurate a n d complete reports b y practicing these exercises ( w i t h the help of the tutor).

Cognitive Assessment in Coached Learning Environments

Cognitive Assessment in the Coached Learning Environment

This c o m p u t e r coach p r o v i d e s a n interesting e n v i r o n m e n t for cognitive assess-m e n t that is authentic i n the sense that it has been designed to reflect charac-teristics of expert tutoring a n d p r o b l e m - s o l v i n g i n the d o m a i n . The authenticity of the e n v i r o n m e n t can be verified b y c o m p a r i n g it w i t h various conditions of face-to-face a n d n e t w o r k e d tutoring. If students' learning proces-ses are f o u n d to be s i m i l a r across these situations, the computer tutor c o u l d be considered to have a h i g h degree of authenticity. A r e v i e w of the characteristics often r e c o m m e n d e d b y advocates of alternative approaches to assessment reveals that m a n y of these are met, a n d so there is a reasonable l i k e l i h o o d that o u r coached l e a r n i n g e n v i r o n m e n t is at least a candidate for a c h i e v i n g a c o n -trolled e n v i r o n m e n t for cognitive assessment. In the remainder of this article w e briefly examine the types of "responses" that are obtained f r o m the stu-dents i n this e n v i r o n m e n t , h o w they can p r o v i d e i n f o r m a t i o n related to the goals of cognitive assessment, a n d finally the types of statistical measurement m o d e l s that appear to be needed. A t h o r o u g h analysis a n d d e v e l o p m e n t of appropriate measurement m o d e l s is a large task that remains to be done.

Briefly, o u r assessment goals are: (a) d y n a m i c assessment of students' p r o f i ciency i n p r o b l e m s o l v i n g a n d the extent a n d q u a l i t y of their d o m a i n k n o w -ledge; (b) diagnostic assessment of students' p r o b l e m - s o l v i n g : identify correct a p p l i c a t i o n of c o m p o n e n t procedures, location a n d k i n d s of errors, types a n d levels of coaching required, a n d pattern of a p p l i c a t i o n of procedures; (c) assess-m e n t of student learning: identify changes over practice probleassess-ms; (d) predic-tion of unassisted performance o n "transfer" problems; a n d (e) extension of assessment to collaborative learning. O u r task is to specify h o w measures or indicators relevant to these goals can be obtained f r o m the responses of students w h i l e w o r k i n g o n practice problems i n the coached learning e n v i r o n -ment. T h e responses students can m a k e for each " i t e m " (i.e., c o m p o n e n t procedure) are identified i n Figure 3.

A t any location (i.e., node c o r r e s p o n d i n g to a c o m p o n e n t procedure) i n the p r o c e d u r e hierarchy, several k i n d s of student response can occur. Successful c o m p l e t i o n of the subtask c o r r e s p o n d i n g to a superordinate node requires successful execution of all subordinate tasks p l u s c o m p l e t i o n of a superor-dinate integrative task. F o r example, w r i t i n g the m o d e l equation i n v o l v e s w r i t i n g expressions for a l l the components of the equation p l u s arranging them into a single equation. For any n o d e three types of i n f o r m a t i o n are recorded: (a) the student's self evaluation of his or her solution at that step i n the procedure, (b) the level of coaching assistance used b y the student to p r o d u c e a correct s o l u t i o n at this step, a n d (c) the type of instructional assistance the student obtained f r o m the tutor. (Note that the evaluation of a student's independent w o r k c o u l d be done b y the instructor u s i n g the tutor to "score" the student's w o r k . ) Response categories for the first t w o of these w o u l d be considered to be o r d e r e d : evaluations range f r o m i t e m correct to item failed (errors only) or not attempted, a n d levels of assistance range f r o m n o assistance, to questions d e s i g n e d to elicit appropriate k n o w l e d g e , to hints that p r o v i d e partial k n o w -ledge, to f u l l instructions (each of these c o u l d have c o r r e s p o n d i n g levels of detail). Response categories c o r r e s p o n d i n g to types of assistance are not ex-pected to be o r d e r e d ; rather, they reflect students' preferences for different

Superordinate

node

Subordinate

Component

node

Procedure

_ORD

(TEM)

ORD

(TEM)

I

or

^ "Item"

COND

Component

Procedure

Complete all component

subtasks plus superordinate

integrative task

Complete subordinate

tasks

p

S L

Type of Instruction:

• Goal

• Problem

• Action

• Tools

• Theory

• Conditions

• Result

(Nominal categories)

Evaluation:

• Correct

• Partially correct (incomplete)

• Partially correct (errors)

• Attempt (errors)

• Not attempted

(Ordered categories)

Level of Coaching

Assistance:

• no assistance

• probe question

• hint

• instruction/explanation

Levels of detail

(Ordered categories)

Cognitive Assessment in Coached Learning Environments

types of i n f o r m a t i o n about procedures. T h e question w e need to address is h o w c a n w e use these responses to obtain i n f o r m a t i o n relevant to o u r assess-m e n t goals.

Domain Knowledge and Problem-solving Proficiency

K n o w l e d g e a n d proficiency are associated w i t h major components (high-level components) of the p r o b l e m - s o l v i n g procedures. T h e high-level components for statistical d e s i g n a n d data analysis problems i n c l u d e topics such as (a) s p e c i f y i n g the research design, (b) s p e c i f y i n g the s a m p l i n g p l a n , (c) s p e c i f y i n g and testing d i s t r i b u t i o n assumptions for the data, (d) p r e p a r i n g the data for analysis, (e) c a r r y i n g out descriptive analysis of the data, a n d (f) c a r r y i n g out the analysis of variance. In each of these h i g h l e v e l components, s u b c o m -ponent procedures are specified. F o r example, the subcom-ponents of (f) the analysis of variance are: (a) specifying the appropriate A N O V A design, (b) w r i t i n g the linear score m o d e l , (c) o b t a i n i n g estimates of parameters i n the m o d e l , (d) p a r t i t i o n i n g the s u m of squares, (e) c o m p u t i n g A N O V A statistics, (f) c o n d u c t i n g significance tests, (g) constructing the A N O V A table, a n d (h) carry-i n g out tests of p r e p l a n n e d contrasts. Because each of these decomposes hcarry-ierar- hierar-chically into components, proficiency is associated w i t h successful execution of the c o m p o n e n t s a n d integration of components at each level into a s o l u t i o n of the m o r e general p r o b l e m .

T w o k i n d s of measurement models m i g h t be considered: hierarchical m o d e l s that explicitly incorporate the hierarchy into the measurement m o d e l (Embretson, 1993; G i t o m e r & Rock, 1993) a n d latent trait models that scale items (corresponding to the procedure components) o n a single d i m e n s i o n or

profi-ciency scale. A c o m p l i c a t i n g factor is that measurement based o n these items

m a y be based o n ordered categories of response (levels of correctness o r assis-tance), suggesting some k i n d of partial-credit m o d e l (Masters, 1982; Masters & M i s l e v y , 1993) o r assignment to p u r e l y categorical latent classes (e.g., based o n categories c o r r e s p o n d i n g to types of assistance, Y a m a m o t o & G i t o m e r , 1993). Test theory m o d e l s of this k i n d m a y be applicable to response measures o b -tained i n the coached learning environments, a n d w o u l d justify assessments at the level of the " l a r g e " procedure components that are i n v o l v e d i n data analy-sis u s i n g A N O V A m o d e l s a n d methods. T h e function of such assessments w o u l d be to certify that a student h a d reached a particular level of proficiency and k n o w l e d g e w i t h respect to a g i v e n c o m p o n e n t procedure.

Diagnostic Assessment of Component Knowledge and Problem-solving Procedures

Diagnostic assessment makes a less stringent d e m a n d o n statistical measure-m e n t measure-m o d e l s if the p r i measure-m a r y goal is to p r o v i d e the student or instructor w i t h diagnostic i n f o r m a t i o n about w h e r e the student is h a v i n g difficulty a n d w h a t k i n d s of k n o w l e d g e are l a c k i n g . Such i n f o r m a t i o n is p r o v i d e d b y the trace of student responses u s i n g the tutor a n d evaluations of the students' w o r k . Reliability of s u c h diagnostic assessments is less of a concern, but consistency of errors i n c o m p o n e n t tasks o r need for assistance w o u l d indicate real m i s u n d e r s t a n d i n g s o r lack of k n o w l e d g e . Properties of i n d i v i d u a l items (com-ponents) resulting f r o m item analyses p e r f o r m e d to construct proficiency measures w o u l d p r o v i d e evidence of h o w significant difficulties at the level of specific items are as contributors to proficiency o n major task components. I n

C. Frederiksen and j. Donin

a d d i t i o n to d i a g n o s i n g w h e r e difficulties occur, diagnostic assessment w o u l d i n c l u d e trace analysis of particular patterns a n d sequences i n a student's ap-plication of procedures. It m a y be possible to establish expert-like patterns a n d n o v i c e patterns to use as a basis for interpreting students' sequence of p r o b l e m -s o l v i n g operation-s (Le-sgold et al., 1990). Finally, the quality of -student concep-tual u n d e r s t a n d i n g a n d explanations m i g h t be assessed at various steps i n p r o b l e m - s o l v i n g to evaluate students' k n o w l e d g e . This c o u l d be done t h r o u g h reports, questions, or other tasks designed explicitly to obtain such i n f o r m a t i o n f r o m the students.

Assessment of Change with Practice

T h e assessment of change can occur both at the level of d o m a i n k n o w l e d g e a n d proficiency a n d at the level of i n d i v i d u a l c o m p o n e n t procedures a n d the se-quence i n w h i c h they are a p p l i e d (diagnostic level). The p r i n c i p a l question is to w h a t extent is a student's performance g r a d u a l l y a p p r o x i m a t i n g that of a n expert? A n s w e r i n g this question w i l l i n v o l v e interesting analyses of changes i n the traces of students' responses w h i l e u s i n g the tutors. In a d d i t i o n to the pattern of performance finally attained, changes i n students' performance over p r o b l e m exercises p r o v i d e s i n f o r m a t i o n about h o w a n i n d i v i d u a l progresses a n d a n y l e a r n i n g difficulties that m a y have occurred. S u c h data m a y be v a l u able i n h e l p i n g students d e v e l o p effective learning strategies for c o m p l e x d o -m a i n s s u c h as statistics.

Prediction of Performance on Criterion Problem-Solving Tasks

A s t r a i g h t f o r w a r d a p p r o a c h to p r e d i c t i o n of performance w o u l d be to assess proficiency w i t h m a i n c o m p o n e n t procedures w h i l e u s i n g the tutor w h e n s o l v i n g practice p r o b l e m s . O f particular interest w o u l d be the level of p r o f i -ciency attained for each procedure a n d the difficulty level of the practice p r o b l e m s attempted. The same assessment criteria c o u l d be a p p l i e d to a post-practice near-transfer assessment p r o b l e m , a n d then the tutor performance c o u l d be used to predict final performance. If tutor performance turns out to be a g o o d p r e d i c t o r of performance o n the criterion problems, then the use of a coachedlearning e n v i r o n m e n t for assessment m i g h t be r e c o m m e n d e d . In a d -d i t i o n , a close analysis of stu-dents for w h o m pre-dictions are poorest m i g h t p r o v i d e interesting i n f o r m a t i o n about factors responsible for d i f f i c u l t y i n m a k i n g the transition f r o m assisted performance to independent competence i n a d o m a i n . T h i s issue corresponds to one of the p r i n c i p a l arguments for a cognitive apprenticeship a p p r o a c h i n w h i c h assistance p r o v i d e d b y a coach is g r a d u a l l y r e d u c e d to enable students to d e v e l o p a u t o n o m o u s s k i l l i n a d o m a i n .

Extension to Collaborative Learning Situations

C o a c h e d l e a r n i n g e n v i r o n m e n t s can be used effectively b y pairs of students or s m a l l g r o u p s of students, w i t h the a d d e d benefits of m o t i v a t i o n a n d d e v e l o p -m e n t of s k i l l i n collaborative p r o b l e -m - s o l v i n g . A s an e n v i r o n -m e n t for cognitive assessment, the trace of students u s i n g the e n v i r o n m e n t w o u l d reflect the k n o w l e d g e of each a n d their joint p r o b l e m - s o l v i n g behavior. W h e n used this w a y , it w o u l d seem essential that students also attempt problems i n d i v i d u a l l y (also w i t h the h e l p of the tutor) a n d that they be assessed i n d e p e n d e n t l y i n order to d e v e l o p i n d e p e n d e n t competence a n d k n o w l e d g e i n the d o m a i n . Data obtained i n s u c h studies w o u l d bear o n m a n y of the l e a r n i n g issues u n d e r l y i n g

Cognitive Assessment in Coached Learning Environments

c o l l a b o r a t i v e approaches to i n s t r u c t i o n , especially o n the r e l a t i o n s h i p b e t w e e n c o l l a b o r a t i v e c o m p e t e n c e a n d i n d i v i d u a l competence a n d k n o w l e d g e i n a d o m a i n .

I n s u m m a r y , c o a c h e d l e a r n i n g e n v i r o n m e n t s of the t y p e w e describe i n this article a p p e a r to offer p o t e n t i a l as e n v i r o n m e n t s for assessment. A s assessment e n v i r o n m e n t s they c a n p r o v i d e objective a n d c o g n i t i v e l y v a l i d indicators of p e r f o r m a n c e . M e a s u r e m e n t m o d e l s c a n be d e v e l o p e d for students' responses i n these e n v i r o n m e n t s to enable assessment of gains i n k n o w l e d g e a n d p r o f i -ciency, a n d these can be v a l i d a t e d as p r e d i c t o r s of p e r f o r m a n c e o n c r i t e r i o n p r o b l e m - s o l v i n g tasks. M o r e o v e r , the p e r f o r m a n c e measures themselves c a n be u s e d to p r o v i d e instructors (and students) w i t h diagnostic i n f o r m a t i o n a b o u t s t u d e n t s ' u n d e r s t a n d i n g a n d p r o b l e m - s o l v i n g , i n c l u d i n g their errors a n d m i s c o n c e p t i o n s . I n a d d i t i o n , c o a c h e d - l e a r n i n g e n v i r o n m e n t s c a n p r o v i d e a h i g h degree of authenticity as n a t u r a l e n v i r o n m e n t s for collaborative a n d p r o b l e m - b a s e d l e a r n i n g . A s s u c h they s h o u l d p r o v i d e v a l u a b l e contexts for c o a c h e d practice i n p r o b l e m - s o l v i n g that can extend a n d s u p p l e m e n t class-r o o m i n s t class-r u c t i o n . If futuclass-re class-reseaclass-rch c a n s h o w that this a p p class-r o a c h to assessment c a n be i m p l e m e n t e d i n c o n j u n c t i o n w i t h a p p r o p r i a t e c o g n i t i v e a n d m e a s u r e -m e n t -m o d e l s to create reliable assess-ments w i t h h i g h p r e d i c t i v e v a l i d i t y , then it m a y offer one s o l u t i o n to the p r o b l e m of assessment that is objective, a u t h e n -tic, a n d c o g n i t i v e l y v a l i d .

Notes

1. The analysis of the tutorial discourse was carried out using a computer-aided analysis tool (NLUDIST) to build the frame. The complete analysis of tutorial dialogue that led to the development of the procedure frame is stored in a NU-DIST database. This database documents how each text unit (and associated actions) in the tutorial dialogue was matched to procedures in the expert model.

2. Note that Figures la and lb do not include all of the branching procedures: branching subprocedures are collapsed into lists in the figures and if they are followed by arrows, they branch further.

3. In coached tutoring sessions (such as those analyzed in our engineering tutoring study), the tutor and student participated more equally in the dialogue with the tutor occasionally guiding the student to the next high-level procedure, but with the student initiating most procedures and related dialogue exchanges.

Acknowledgments

This research was supported by the Telelearning NCE, funded by the Government of Canada. The development and testing of the statistics tutor are being supported by a grant from the Office of Learning Technologies, Human Resources Development Canada. The engineering tutoring study was supported by the National Research Council and BNH Expert Engineering Software, Inc. St. Laurent, QC. Lorraine Meilleur and Marguerite Roy assisted in the research, and Michel Décary wrote the Tutor Builder software. A special thanks to Robert Bracewell for participating as the statistics tutor in these studies.

References

Birenbaum, M . (1995). Assessment 2000: Towards a pluralistic approach to assessment. In M.

Birenbaum & F.J.R.C. Dochy (Eds.), Alternatives in assessment of achievements, learning processes, and prior knowledge (pp. 3-29). London: Kluwer.

Campione, J.C., & Brown, A.L. (1990). Guided learning and transfer: Implications for approaches to assessment. In N . Frederiksen, Glaser, R., Lesgold, A., & Shafto, M. (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 141-172). Hillsdale, NJ: Erlbaum. Collins, A., Brown, J.S., & Newman, D. (1989). Cognitive apprenticeship: Teaching the craft of

reading, writing, and mathematics. In L. Resnick (Ed.), Knowing, learning and instruction: Essays in honor of Robert Glaser (pp. 453-494). Hillsdale, NJ: Erlbaum.

C. Frederiksen and J. Donin

Derry, S.JV & Lajoie, S. (1993). A middle camp for (un)intelligent instructional computing: An

introduction. In S. Lajoie & S.J. Derry (Eds.), Computers as cognitive tools (pp. 1-11). Hillsdale, NJ: Erlbaum.

Embretson, (1993). Psychometric models for learning and cognitive processes. In N . Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Tesf theory for a new generation of tests (pp. 125-150). Hillsdale, NJ: Erlbaum.

Frederiksen, C.H. (1986). Cognitive models and discourse analysis. In C R . Cooper & S.

Greenbaum (Eds.), Studying writing: Linguistic approaches (pp. 227-267). Beverly Hills, CA: Sage.

Frederiksen, C H . , & Roy, M . (1999). Tutors' use of discourse to model and explain problem-solving procedures in statistics. Unpublished manuscript.

Frederiksen, C H . , Roy, M., & Bédard, D. (1999). Expert tutoring and learning: Analysis of tutorial dialogue in engineering. Unpublished manuscript.

Frederiksen, N., Glaser, R., Lesgold, A., & Shafto, M . (Eds.). (1990). Diagnostic monitoring of skill and knowledge acquisition. Hillsdale, NJ: Erlbaum.

Frederiksen, N., Mislevy, R.J., & Bejar, I.I. (Eds.). (1993). Test theory for a new generation of tests. Hillsdale, NJ: Erlbaum.

Gitomer, D.H., & Rock, D. (1993). Addressing process variables in test analysis. In N .

Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Test theory for a new generation of tests (pp. 243-268). Hillsdale, NJ: Erlbaum.

Lesgold, A., Lajoie, S., Logan, D., & Eggan, G. (1990). Applying cognitive task analysis and research methods to assessment. In N . Frederiksen, Glaser, R., Lesgold, A., & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 325-350). Hillsdale, NJ: Erlbaum.

Masters, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47,149-174.

Masters, G.N., & Mislevy, R.J. (1993). New views of student learning: Implications for

educational measurement. In N . Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Test theory for a neiv generation of tests (pp. 219-224). Hillsdale, NJ: Erlbaum.

Mislevy, R.J. (1993). Foundations of a new test theory. In N. Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Tesf theory for a new generation of tests (pp. 19-40). Hillsdale, NJ: Erlbaum. Snow, R.E. & Lohman, D.F. (1993). Cognitive psychology, new test design, and new test theory:

An introduction. In N . Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Test theory for a new generation of tests (pp. 1-18). Hillsdale, NJ: Erlbaum.

Yamamoto, K. & Gitomer, D.H. (1993). Appliction of a HYBRID model to a test of cognitive skill representation. In N . Frederiksen, R.J. Mislevy, & I.I. Bejar (Eds.), Tesf theory for a new generation of tests (pp. 275-296). Hillsdale, NJ: Erlbaum.