E VALUATING A CTIVITIES IN E-L EARNING E NVIRONMENTS

In the previous paragraph we have focused on the influence that the second level of analysis has on the understanding of contextual factors. In particular, we have observed the influence of social interaction in school settings, which results in a series of moderating effects on students' performance in evaluative contexts. However, social interactions play a significant role also for defining, on the one hand, the relational structure that characterizes a class of students (which may be based, for instance, on collaboration, social support, information exchange) and, on the other hand, the students’ social status and their role in this relational structure (Bronfenbrenner, 2004). Again, we can refer to the second level of analysis, when Doise (1982/1987) introduces the paradigm of the communication network that “ha[s] often been used to show how the different communications systems which may

exist between a number of people allow them to coordinate the information available in a more or less efficient way in problem solving” (p. 12). Even if this paradigm is dated, since it

derives from Moreno’s sociometry (1951), which was developed during the Thirties and Forties, and from Bavelas’ studies, which were carried out in the Fifties (1948, 1950), there is now a lot of interest on Social Networks Analysis applied to Web Communities and, specifically, to Web Communities in Educational and Vocational Environment (Freeman, 1986; Garton, Haythornthwaite, & Wellman, 1997).

Web communities are one of the two key aspects of e-learning; the other is constituted by the so called Learning Objects. These two different key aspects are also representative of different ways of conceiving knowledge transmission and construction in e-learning environments. In everyday discussions, and often improperly, the concept of e-learning (electronic-learning) involves multiple aspects of distance education, which range from content selection to the organization and coordination of specific on-line courses.

On the one hand, e-learning may be identified principally with forms of learning and training which are essentially based on interactions between group or community members:

Communities of Practice (Wenger, 1998), Knowledge Building Communities (Scardamalia &

Bereiter, 1994), Learning Communities (CTGV, 1993), Communities of Learning and

Thinking (Brown & Campione, 1990), Communities of inquiry (Lipman, 1991). Learning

processes that lie behind this mode of conceiving e-learning found their theoretical references on socioconstructivism (Doise & Mugny, 1997) and sociocultural approach to human cognitive development inspired by Vygotskij. From this point of view, individual cognitive development is conceived as a result of social interaction in which:

ƒ the support and the sustain of either adult or expert peer partner is a decisive factor; ƒ there is the simultaneous presence of different points of view, and the consequent

necessity of a negotiation of meanings of the task), (cfr. the notion of sociocognitive

conflict; Doise, Mugny, & Perret-Clermont, 1974; see also Carugati & Gilly, 1993).

On the other hand, e-learning is also conceived as pure transposition via web of typical educational models of face-to-face classes. According to this approach, learning is conceived as a mere content supply. Therefore, the “e” component (electronic) refers only to the content in terms of design, supply and fruition. This is the case of Learning Objects, by which one

tends “to break educational content down into small chunks that can be reused in various learning environments, in the spirit of object-oriented programming” (Wiley, 2000, p. 7). Thus, content selection, construction and organization by educators, and content supply by web artifacts, become the very critical phases for learning processes. This idea of content modularity emerges from approaches that remind us of Mastery Learning years, which derived from that behaviorist technology, which Block (1974) proposed as the new promise for “teaching everything to everyone”.

Summing up, we may suggest that it is possible to find the same ideas and representations ofdevelopmental and learning processes both in e-learning and in “presence” situations if we consider the following two points, i.e.: a) that knowledge is elaborated “in the mind” of the single person for being then used in the interaction with others, or b) that knowledge is constructed during interaction with others as a “collective mind”, which is external to the single person, and which will be interiorized and elaborated by individuals only in a second moment.

Both of these two conceptions of e-learning require a change of perspective, i.e., a passage from “what students do in an e-learning environment” to their evaluation. Such a change is now taking place by means of the monitoring of students’ actions within a web platform. When we refer to actions, we consider the perspective of Leont’ev (1978) about human activity, in which activity is seen as always collective and sustained by some social motive or necessity. Each human activity is constituted by individual actions, which are achieved by individual or groups, and directed to specific goals. Each individual action consists of operations, i.e., automatic acts without a voluntary control performed by the individual in the execution of some action. Since actions could be performed by a single person (e.g., the student’s utilization of the resources proposed by the teacher in web platform), but also by a group (e.g., the discussions in a web forum), we can consider actions as individual (a student interacts with contents through web artifacts, e.g., a web platform) or as collective (a student interacts withother students through web artifacts, e.g., a web forum). In all of these cases, such actions related to the student’s activity may be considered in terms of competence acquisition, because they are aimed at using web artifacts for knowledge acquisition (as is the case of individual actions), and at managing on-line interactions with others for collective knowledge, sharing and construction (as is the case of collective actions). If we consider the importance of competences and learning outcomes in Dublin Descriptors, and, at vocational level, the Lifelong Learning Programme 2007-2013 launched by the European Union, in which web technologies are seen as one of the key tools for achieving the objectives of the programme (Pépin, 2007), we may easily realize that this issue is crucial not only in the field of academic research, but also in the field of professional training as defined by European policies.

Starting from these considerations, how can we monitor students’ on-line activity in both individual and collective actions?

A quantitative technique for data collection about “what user do” in an on-line environment is to be identified in the web tracking (Calvani, Fini, Bonaiuti, & Mazzoni, 2005; Mazzoni, 2006, Proctor & Vu, 2005). Through web tracking it is possible to collect a number of details about the frequency of visits and time spent on web pages during the navigation on a web artifact (e.g., web site or web platforms). This data collection technique is a feature that we can find in almost all of the existing web platforms, and it is also provided by the Italian legislative decree concerning Distance University as a means for monitoring

and evaluating students’ on-line activities. If, on the one hand, we can consider web tracking as a good technique for collecting data about individual actions, i.e., about the frequentation and the usage of web contents (Learning Objects) by students, we cannot affirm the same as far as the application of this technique to web communities is concerned. Of course, web tracking allows us to collect data on interactions between students, which may consist of, e.g., sent or received messages and sent or received replies. However, these data refer to individual characteristics (how many messages a student has sent, received, etc.) and do not provide any indication about addressees. Relational aspects, therefore, are not taken into consideration within the rough data collected by web tracking. Nevertheless, this information is available. In other words, web tracking may be employed also in order to collect data about to whom a message/reply is sent, and about the identity of the receiver of a given message/reply (the so called relational data), but these data are normally used only for summing and displaying the quantity of messages sent and received by single students. From this point of view, data obtained from web tracking could be used for analysis positioned at the first level proposed by Doise.

Now, if we consider web groups or web communities in e-learning environment, we have to consider that the final outcome of a collective activity does not derive from simple individual actions, but principally from collective actions performed by the online group or community. In this case we consider individual actions as separated from collective actions, and we have to take into account that group performance does not derive from a sum of individual actions, but rather from indicators that allow us to map the collective actions of an online group or community.

As previously outlined, relational data of web group/community could be collected by web tracking; this possibility, besides facilitating the application of quantitative analysis, allowsto construct the adjacency matrix (Figure 1) of relational data for applying the Social Network Analysis (SNA) to group exchanges.

Starting from the transposition of relational data in a matrix, SNA allows, on the one hand, to graphically represent the network of relations by sociograms and, on the other hand, to analyze this network on the basis of notions that allow to describe the relevant communicative structure. Now, a very interesting aspect is that we can develop an analysis on two levels, i.e., by focusing on the single members and their relations in the network (ego-

centered analysis) or by focusing on the network and its structural characteristics (whole network or full network analysis). Obviously, these two aspects are related. This means that

for each whole network structural indexes we have also specific individual measures.

E.g., the density of a network, i.e., “the proportion of possible lines that are actually

present in the graph” (Wasserman & Faust, 1994, p. 101) or more simply the percentage of

aggregation of its members, derives from the degree of each member, i.e., the totality of direct contacts he/she has activated or received by others. Considering the centralization, i.e., the dependence of a network from its “most important” actors, we have, together with this whole index, also the centrality index of each member, i.e., his/her importance/prominence for the communicative structure. Thus, these related networks and individual measures allow us to perform map description of collective actions of a community. On the one hand, we can monitor and depict the role and function of each member in the community knowledge exchange (e.g., wideness and aggregation of his/her neighborhood or direct contacts, central or peripheral role in information exchanges/transmission, participation in subgroups, etc.); on

Receivers

Stud1 Stud2 Stud3 Stud4 Stud5 Stud6

Stud1 0.0 0.0 3.0 4.0 0.0 0.0 Stud2 0.0 0.0 2.0 0.0 0.0 0.0 Stud3 3.0 4.0 0.0 0.0 0.0 0.0 Stud4 2.0 0.0 3.0 0.0 0.0 0.0 Stud5 0.0 0.0 0.0 0.0 0.0 0.0 S e n d e r s Stud6 1.0 3.0 2.0 2.0 0.0 0.0

Figure 1. Adjacency matrix of exchanges between students in a web forum and sociogram

representation by NetMiner1.

the other hand, we can monitor the group/community while considering the aggregation of the communicative structure, the reciprocity in discussions, the number and density of possible subgroups, etc.. In spite of web tracking data, therefore, SNA indexes represent a second level of analysis as conceived in the theory of Doise.

In order to illustrate how web tracking data and SNA indexes may be utilized, we will briefly present a study (which has not yet been published), in which we have formulated a model for representing individual and groups profiles based on both individual (coming from web tracking indicators) and collective (SNA indexes) actions. The study concerns two groups of teachers in vocational training and one group of university students. Since it would be inappropriate to provide here detailed explanations of the complex phases of data elaboration, we will simply describe our model in its main features and functions, which are basically aimed at providing useful information for representing individual and group profile.

The model consists of five areas of actions: three areas of individual actions, collected by Web Tracking (platform use; loquacity; participation to discussions) and two areas of collective actions collected by SNA (role in group collaboration; dealing with group).

All web tracking indicators and SNA indexes have been elaborated so that we could obtain a graph for each participant, which describes his/her actual performance levels in each area in relation to the maximum performance level attained by his/her group. The same may be done for the entire group, in order to obtain a graph displaying the average performance of participants in each area in relation to the maximum performance level attainable by the group (Figure 2).

In summary, this model allows us to take into consideration and represent not only the individual actions a student performs within an e-learning environment, in order to interact with contents, but also the collective actions he/she accomplishes for interacting with his/her colleagues during on-line group collaboration. Further, as we show in figure 2, we can use this model for representing group performances, and thus for comparing different groups involved in virtual learning environment characterized by collaborative activities.

From this point of view, we can analyze class/group actions in virtual environment considering the three different levels of analysis proposed by Doise. The first level of analysis is represented by individual actions derived from web tracking data. The second level of analysis is represented by collective actions mapped by whole network structural SNA indexes. Finally, the third level of analysis is represented by the students’ social roles in web

1

interactions, as mapped by SNA individual measures. Obviously, these roles are not fixed: during different periods of a web forum a student could assume different roles (for instance peripheral or central), whereas the same role could be assumed by different students.

Figure 2. An example of performance attained by a participant and by his/her group.

C

The idea that our behaviors result from processes of analysis and evaluation of a specific situation is supported by numerous studies. Drawing on Weiner’s metaphor (2006), we could consider ourselves as “judges” in a courtroom which, before delivering a judgment on a given event, and taking consequent action, evaluate all available information and evidence. Teachers’ judgments precede educational practices, feedback and evaluation. However, these judgments are not solely based on the performance of pupils. As a matter of fact, there are several “contextual” elements that play a part in such a process. As we could notice in the course of the present chapter, the process that leads to the formulation of judgments is complex, and is characterized by the action of various factors. We explained how teachers’ social representations influence the educational practices they adopt in class; how causes

attributed to succeeding or failing may influence judgments and evaluations; and how such a process involves the interaction of shared social norms and of given aspects of the school context considered. Next to these determinants of teacher judgments, we have also analyzed specific context-related elements that influence pupils’ performances directly, and that therefore compromise the quality of those evaluations that consider performance as the direct indicator of pupils’ achievement. Finally, we have dealt with an issue that is particularly relevant in today’s society and culture, i.e., that of evaluation and monitoring within e- learning contexts. We could observe how evaluation, also as far as e-learning is concerned, may be seen as a process that is based not only on the pupil’s individual performance, but also on specific information that takes into account the individual’s relationships with his/her reference group, and also his/her role in managing and transmitting such information.

With the purpose of “giving psychology away”, we believe that our contribution may offer some useful insights and ideas to be considered by teachers in their daily school activities. They may particularly contribute to raise awareness on those factors influencing the production of judgments, so that educational practices and evaluations may consequently improve the value of judgments and evaluations. This, in turn, may promote further improvement of educational contexts, and therefore encourage the creation of enhancing conditions, in which performances may take place and be evaluated according to more objective criteria.

Further considerations may be made as far as evaluation in e-learning contexts is concerned, which today is often at the center of debates and research. As a matter of fact, the data collected through web tracking may not be considered as representative of pupils’ actions within a given virtual learning environment. Rather, they reveal a quite static picture of the frequency of visits to certain resources and, possibly, the completion or non-completion of given tasks. Such logic, however, does not provide any useful elements to those analysts, who wish to explore social aspects of e-learning, which concern, for instance, the network of relations that characterizes participants. In other words, it does not consider relations among individuals, i.e., how information is transmitted among them, and what subjects occupy more central or more peripheral positions within the managing of information. Hence the search for analytical models, such as Social Network Analysis, which we suggested, and which Doise himself refers to, becomes necessary in order to provide further useful tools to control and analyze complex situations, and to suggest interesting new perspectives for evaluation.

R

Ambady, N., Shih, M., Kim, A., & Pittinsky, T. L. (2001). Stereotype susceptibility in children: Effects of identity activation on quantitative performance. Psychological

Science, 12, 385–390.

Baron, R. S. (1986). Distraction-conflict theory: Progress and problems. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 19, pp. 1–40). New York: Academic Press.

Bavelas, A. (1948). A mathematical model for group structure. Human Organization, 7, 16- 30.

Bavelas, A. (1950). Communication patterns in task-oriented groups. Journal of the

Acoustical Society of America, 22, 725-730.

Beauvois, J.L. (2003). Judgment norms, social utility, and individualism. In N. Dubois (Ed.),

A sociocognitive approach to social norms (pp. 123-147). London: Routledge.

Beauvois, J.L., & Dubois, N. (1988). The norm of internality in the explanation of psychosocial events. European Journal of Social Psychology, 4, 9-28.

Berger, P., & Luckmann, T. (1966). The social construction of reality. Garden City, New York: Doubleday and Co.

Block, J. (1974). Schools, Society and Mastery Learning. New York: Holt Rinehart and Windston.

Bonnot, V., & Croizet, J.C. (2007). Stereotype internalization and women’s math performance: The role of interference in working memory. Journal of Experimental

Social Psychology, 43, 857–866.

Bressoux, P., & Pansu, P. (2003). Quand les enseignants jugent leurs élèves. Paris : Presses Universitaires de France.

Bronfenbrenner, U. (2004). Making human beings human : bioecological perspectives on

human development. Thousand Oaks, CA: Sage.

Brown, A. L., & Campione, J. C. (1990). Communities of Learning and Thinking, or a context by any other name. Contributions to Human Development, 21, 108-126.

Butera, F. & Buchs, C. (2005). Reasoning together: From focusing to decentering. In V. Girotto & P. N. Johnson-Laird (Eds.), The shape of reason (pp. 193-203). Hove, England: Psychology Press.

Cadinu, M., Maass, A., Rosabianca, A., & Kiesner, J. (2005). Why do women underperform under stereotype threat? Evidence for the role of negative thinking. Psychological

Science, 16, 572–578.

Calvani, A., Fini, A., Bonaiuti, G., & Mazzoni, E. (2005). Monitoring interactions in collaborative learning environments (CSCL): a tool kit for Synergeia. Journal of E-

learning and Knowledge Society, 1, 63-73.

Carugati, F. (1990a). From social cognition to social representations in the study of intelligence. In G. M. Duveen & B. Lloyd (Eds.), Social representations and the

development of knowledge (pp. 126-143). Cambridge: Cambridge University Press.

Carugati, F. (1990b). Everyday ideas, theoretical models, and social representations: The case of intelligence and its development. In G.R.Semin & K.J.Gergen (Eds.), Everyday

understanding: Social and scientific implications (pp. 130-150). London: Sage.

Carugati, F., & Gilly. M. (1993). The multiple sides of the same tool: cognitive development as a matter of social construction of meaning. European Journal of Psychology of

Education, 8(4), 345 - 354.

Carugati, F., & Selleri, P. (1998) Social representations and development: Experts’ and parents’ discourses about a puzzling issue. In U. Flick (Ed.), The psychology of the social (pp. 170 -185). Cambridge: Cambridge University Press.

Carugati, F., & Selleri, P. (2004). Intelligence, educational practices and school reform: Organisations change, representations persist. European Journal of School Psychology,