A Multi-Agent System for extracting and
analysing users’ interaction in a Collaborative
Knowledge Management System
Doina Alexandra Dumitrescu, Ruth Cobos and Jaime Moreno-Llorena
Abstract In this paper we present a Multi-Agent System (MAS) for extracting and analysing users interaction in a Collaborative Knowledge Management system called KnowCat. The proposed MAS employs Web Use Mining and Web Structure Mining techniques in order to detect the most relevant interactions of the Know-Cat users and therefore should have more weight in the Knowledge Crystallization mechanism of KnowCat. More concretely, the MAS extracts the users interaction information and analyses whether they are working in the system in an organised or disorganised way. The obtained results in this study give us evidence that organised users contribute positively in a good performance of the Knowledge Crystallization mechanism of KnowCat.
Key words: multi-agents system, interaction analysis, agent-based interaction analysis
1.1 Introduction
It is well known that users’ interaction analysis has a great interest and utility in most computer systems, for instance to evaluate their interface usability [2] or to adapt the system to each user. In the computer supported cooperative work systems (CSCW) the interaction analysis takes a special relevance, since it can help in the study of the collaborative tasks carried out [11, 17] and in the analysis of the achieved individual results by each user [10].
The interaction analysis problem involves diverse aspects among those that high-light the Data Mining, to process the activity registers generated by the users and to
Doina Alexandra Dumitrescu, Ruth Cobos and Jaime Moreno-Llorena
Department of Computer Science, Universidad Aut´onoma de Madrid, Spain, 28049 Madrid Telephone: +34914972243, Fax: +34914972235
e-mail: (doina.dumitrescu,ruth.cobos,jaime.moreno)@uam.es
figure out structural aspects of the interface. The solution for this problem may also profit from the use of intelligent agents, which collaborate among them in order to carry out the analysis in an autonomous, flexible and less intrusive way regarding the involved systems. The low-level events that are generated by many user interfaces to analyze the users interaction are also a potent and interesting alternative, mainly because it deals with part of the problem in a generic way. In great detail, many researches shows us how Multi-Agent approaches are used in the task of analysing learner’s interactions in collaborative learning systems in order to improve the col-laboration (i.e. [14, 16]).
In this paper the interaction of the users is studied using the Collaborative Knowl-edge Management system called KnowCat [1, 5]. KnowCat (acronym for “Knowl-edge Catalyser”) is based on a mechanism called “Knowl“Knowl-edge Crystallisation that gives us evidence about which the best contributions in the users opinion are through their interaction with the system.
In this paper, it is presented the design of a Multi-Agent system (MAS) whose purpose is to extract information on how users interact with the KnowCat system and afterwards analyse that information. The main aim of the MAS is to find what the most relevant interactions of the users are and therefore should have more weight in the Knowledge Crystallization mechanism of KnowCat.
1.2 Related Work
Agents can have a multitude of features [6, 18], but we will mention here only two of the most important: the autonomy and being part of a community of agents, characteristics that distinguish agent-based applications from the rest. Multi-agents, a multitude of agents that interact in order to solve problems, are a suitable approach when dealing with complex software systems that are distributed and dynamic.
For some time a great emphasis was given to the integration of agents in collab-orative learning systems in order to improve the effectiveness of the learning effi-ciently . Furthermore multi-agents represent a powerful approach that based on the main features can support the collaborative learning systems [19]. Suh and Lee [16] present a multi-agent framework where the monitoring agents collect, analyse and process the information of the user’ collaborative learning activities and the facilita-tor agents analyse this information in order to offer learning advice. Letizia [13] is an intelligent agent that observes the users browsing behaviour in order to suggest interesting web links for the user. WebWatcher [12] based on the users behaviour and making use of learning techniques recommends the best web resource related to the users keywords. Another example of web mining agent is WebAce [9] that suggests possible interesting new pages for a user based on his browsing activity.
in the Web Logs, and Web Structure Mining techniques are also used, facilitating patterns discovering in the links structure of the Web and permitting internal struc-ture analysing the Web sites.
The interest concerning the users activity has been fundamental in learning based on technologies context, mainly from the introduction of collaborative support in this area. Some business initiatives are also interesting, such as the techniques used by the advertising system DoubleClick (http://www.doubleclick.com), the dis-tributed computing model HWME designed for macroeconomic decision-support based on intelligent information agents in [3] or the recommendation applications that monitor the user’s activity and are becoming more and more common.
1.3 The proposal Context: the KnowCat system and its Client
Monitor
KnowCat is a Collaborative Knowledge Management Web-based system, which is grounded on a client/server architecture and permits multiple instances of the sys-tem, called KnowCat nodes. Each KnowCat node deals with a knowledge area, and has its own knowledge repository and its own users community. The knowledge repository is fundamentally composed of these two elements: Documents, that are the atomic knowledge units in the system; and Topics, that are hierarchically organ-ised in a Knowledge Tree. The aim of each document is to describe the topic where it is located in the knowledge tree. More concretely, when a document is added in a KnowCat node it competes against the others to become the best description on its location (topic). This competitive environment is achieved by the Knowledge Crystallisation mechanism of KnowCat (see [5] for details).
Several previous studies carried out with KnowCat have corroborated that Know-Cat encourages communities to share their knowledge and, progressively, construct knowledge sites of reasonable quality [5]. The KnowCat system has been extended with a Client Monitor (CM). The aim of this extension, CM, is to store both the user data activity on the client side and on the server side in the server Web LOG file of KnowCat.
1.4 A Multi-Agent System for extracting and analysing users’
interaction
The main aim of the project presented in this paper is to bring out the design, devel-opment and experimental performance of a system formed of intelligent agents [18], whose purpose is to extract information on how users interact with the KnowCat sys-tem and afterwards analyse that information and report how each user behaves on certain characteristics (e.g. how an organised user is in his/her work with the sys-tem) in order to detect what interactions of the users are the most relevant ones and therefore should have more weight in the Knowledge Crystallization mechanisms of KnowCat.
A prototype of this proposed Multi-Agent system (called MAS-IA, Multi-Agent System for Interactions Analysis), which addresses whether a user is organised or not in his/her work with the KnowCat system, was designed and implemented with the idea of being extended with more agents that can be in charge of the analysis of other characteristics of the users work with the system (i.e. motivation, interest, etc.).
The technology used to implement the MAS-IA is JADE mainly because it is a set of open source libraries written in Java that offer support for the development of multi-agent systems and presents a series of important features in our case, such as: portability, openness, high scalability and a built-in messaging system. Additionally, the platforms implemented in JADE can be distributed on several machines with dif-ferent operating systems [7], attribute that is useful because KnowCat allows hav-ing the KnowCat nodes distributed in different machines. Furthermore, Jade offers various FIPA-specified interaction protocols in order to support the agent commu-nication.
The proposed MAS-IA integrates the following agent types:
• Data Extractor Agent (DEA), whose role is to obtain interaction information from the enriched Web LOG file of KnowCat for a certain user. This information is represented in an ontology that serves as means of communication between this agent and the following one.
• Organised Behaviour Interpreter Agent (OBIA), whose role is to decide whether a certain user is organised or not in his/her work with KnowCat. The organised characteristic is weighted by a numeric value that we call OrganisedValue (OV), between 0 and 1, where 0 means that the user is totally disorganised and 1 means he/she is very organised.
Fig. 1.1 Overview of the process conducted by the Agents (left-hand side) and the ontology used by the Agents (right-hand side).
1.4.1 Data Extractor Agent
The Web LOG file of a Web server has a line for each accessed resource in chrono-logical order. The content of the lines of the LOG file is fixed by the server con-figuration, but generally they contain standard data (see Fig. 1.2). The DEA deals
62.151.101.192 − −[20/Apr/2004 : 20 : 02 : 47 + 0200]”GET /5.0a/controlNT 2.pl?b = KC SOG2HT T P/1.1”...
62.36.67.34 − −[20/Apr/2004 : 20 : 02 : 35 + 0200]”GET /img/version.gi f HT T P/1.1”... Fig. 1.2 Typical lines LOG web server.
with the activity Web LOG file of KnowCat enriched by CM. CM registers some specific lines interspersed among the lines in the server LOG file. These LOG lines, which are registered by the MC, add to the standard LOG lines interaction informa-tion of the users work on the client side. As we can see in Fig. 1.3, this recorded activity is composed of three kinds of data: (i) identification data as field UsrID that shows the user identification; (ii) temporal data, as CntIni and NtfTmp that indicate the start registering time and the notification instant respectively; and (iii) activity data, as MseM, MseD, Scr, Fcs, Blr and KeyD, which are the numbers of the differ-ent evdiffer-ents through out the time interval, respectively mouse movemdiffer-ents and clicks, scroll movements, focus obtaining and losing, and Keyboard pulsations.
213.37.225.150−−[20/Apr/2004 : 20 : 02 : 35+0200]”GET /5.0a/in f ormSituacion.pl?urlrsp = ../5.0a/monitorKC.pl&MseD = 0&Scr = 0&Fcs = 0&Blr = 0&KeyD = 0HT T P/1.1”... Fig. 1.3 An example line in web Log file of KnowCat generated by MC.
1.4.2 Organised Behaviour Interpreter Agent
Beforehand we realised an inspection of the enriched Web LOG of KnowCat for some controlled users. Knowing the behaviours and the organisation of the work of these users with KnowCat, certain patters were detected, patters that were based in the following two issues:
• Repeated accesses to an element (topic or document) by a user. A repeated access to a selected element means that this element is accessed more than once in a user session, in other words between the accesses to the selected document the user has accesses to other elements.
• Time access to documents (which is the atomic knowledge unit in KnowCat) by a user.
The characteristic of being organised or not with KnowCat is directly related with the first issue: repeated accesses to KnowCat knowledge elements. In order to calculate the OV for a certain user session, the OBIA calculates the following values whether the user has repeated accesses to topics and/or documents in the session:
RPtopics= tnrat/tnat RPdocs= tnrad/tnad RP= (RPtopics+ RPdocs)/2 (1.1)
Where the repetition percentage to accessed topics, RPtopics, is the total number of
repeated accessed topics, tnrat, divided by total number of accessed topics, tnat. Similarly it is defined the repetition percentage to accessed documents, RPdocs. And
the repetition percentage of a user in a session is RP.
The agent distinguishes among these three cases: i) disorganised users (who re-peat access to topics and documents): they receive an OV between 0-0,4 (0 means that the user is totally disorganised in his/her work with the system, and 0,4 signifies that it is getting a medium organised behaviour); ii) medium organised users (who repeat access to topics or documents): they receive a OV between 0,4-0,8 (in the re-alised inspection, we have noticed that these intermediate values correspond to the behaviours of the users that are not very organised and in the same time not very disorganised); iii) organised users (who dont repeat access to topics or documents): they receive an OV between 0,8-1 (1 means that the user is totally organised with his/her work with KnowCat).
Moreover, the OBIA takes into account the average access time to documents in the calculation of the OV for a certain user session. This value is, called tavg, and it
is used by the agent to distinguish for each previous mentioned case into these three subcases: tavgis lower than 10 seconds (very short access duration to documents),
tavgis higher than 60 seconds (very long access duration to documents) and tavgis
The calculation of the OV for a user session is based on the algorithm shown in Fig. 1.4 (tavgis in seconds):
If (repeateddocumentsandtopics) then iniValue = 0 Else if (repeateddocumentsortopics) then iniValue = 0.4 In other case then iniValue = 0.8
If (0.76 ≤ RP ≤ 1) then
If (tavg< 10) then OrganisedValue = iniValue
Else if (tavg> 60) then OrganisedValue = iniValue + 0.1
Else OrganisedValue = iniValue + ((tavg− 10)/500)
Else If (0.51 ≤ RP ≤ 0.75) then
If (tavg< 10) then OrganisedValue = iniValue + 0.1
Else if (tavg> 60) then OrganisedValue = iniValue + 0.2
Else OrganisedValue = iniValue + 0.1 + ((tavg− 10)/500)
Else If (0.26 ≤ RP ≤ 0.50) then If (tavg< 10) then OrganisedValue = iniVale + 0.2
Else if (tavg> 60) then OrganisedValue = iniValue + 0.3
Else then OrganisedValue = iniValue + 0.2 + ((tavg− 10)/500)
Else If (0.1 ≤ RP ≤ 0.25) then If (tavg< 10) then OrganisedValue = iniValue + 0.3
Else if (tavg> 60) then OrganisedValue = iniValue + 0.4
Else then OrganisedValue = iniValue + 0.3 + ((tavg− 10)/500)
Else If (RP = 0) then If (tavg< 10) then OrganisedValue = iniValue
Else if (tavg> 60) then OrganisedValue = iniValue + 0.2
Else then OrganisedValue = iniValue + ((tavg− 10)/500)/2
Fig. 1.4 The calculation of the OrganisedValue for a user session.
1.5 Experimentation and Results
In order to test the proposed approach, we have carried out a research study with a community of 120 students enrolled in an “Information Systems course at the Computer Science Department, at Universidad Aut´onoma de Madrid (UAM). This study was carried in the last three months of the course. The students tasks during this period were executed in the following two phases:
• Creation Phase: in the first month they had to contribute with 2 documents in two assigned topics. They participated like knowledge creators.
• Evaluation Phase: in the following 2 months they had to evaluate (with votes and annotations) all documents of others 3 assigned topics. They participated like knowledge evaluators.
Our study began with the following two initial research questions:
2. How does this influence the way in which students interact with the knowledge crystallisation mechanism? We wanted to know whether organised student inter-actions with the system had any influence in the successfulness of the knowledge crystallisation mechanism.
MAS-IA calculated for all students the following values: i) the OV like a knowl-edge creator: the average value of this characteristics taking into account their inter-actions during the creation phase; ii) the OV like a knowledge evaluator: the average value of this characteristics taking into account their interactions during the evalu-ation phase; and iii) the OV global: the average value of this characteristics taking into account their interactions during the whole study.
At the end of the research study, course instructors evaluated the students work in the following way: i) at first, they assigned a qualification to each student like a knowledge creator, taking into account the quality of their documents; and ii) sec-ondly, instructors assigned a qualification to each student like a knowledge evaluator taking into account the quality of their evaluations. The qualification in both cases was a grade between 0-10, where 10 is the maximum value.
The OV global was the most significant value in order to know whether a stu-dent worked in an organised way with the system during the study. Moreover, we compared the other two calculated values and had evidence that 90% of students re-ceived a higher OV like a knowledge creator than like a knowledge evaluator. This result was expected because in the second task the students had to think about their contributions and compare them with classmates contributions, and they considered to be necessary to access to previously visited topics and documents several times.
On the one hand, most of the students with a high qualification like evaluators (greater than 7.5 in the instructors opinion), received an OV global higher than 0.5, therefore they were considered organised students. On the other hand, most of the students with a low qualification like evaluators (less than 7.5 in the instructors opinion), received an OV global lower than 0.5, therefore they were considered disorganised students.
Testing whether the organised characteristic had any influence on the successful-ness of the knowledge crystallisation mechanism, the following process was made to each topic of the knowledge tree: i) a ranking of the documents was created taking into account the qualifications assigned to them by the instructors, ii) these rankings were compared with the classification offered by the KnowCat system through its Knowledge Crystallisation mechanism, which is based in the students opinions.
In 50% of topics both rankings were very similar (coincidently), all the students, who participated in these topics like evaluators, received a very high qualification like knowledge evaluators in the instructors opinion. Moreover, 90% of these stu-dents received an OV global higher than 0.5.
The previous obtained results corroborated, at first, that the students are more organised like knowledge creators than like knowledge evaluators, and secondly, that organised students were good evaluators. Furthermore, this second result give us evidence that the perception from instructors about which student is a good or a bad evaluator is directly related to the detection by the MAS-IA about what student works in an organised way or not.
In consequence, the most important outcome of this research study was that the quality of students evaluations is directly related to whether they are organised or disorganised and this could be used to improve the Knowledge Crystallisation mech-anism.
1.6 Conclusions and Future Work
In this work we have presented a Multi-agent System, called MASIA, whose pur-pose is to extract and analyse information about how a group of users of a Col-laborative Knowledge Management system, called KnowCat, is interacting with the community knowledge. More concretely, the MAS-IA extracts the users interaction information and analyses whether they are working in the system in an organised or disorganised way.
The MAS-IA has been designed to deal with agents that can operate in both tasks: users interaction extraction and interactions analysis. The proposed approach can determine per each KnowCat user its OV for some selected sessions executed during his/her work with the system.
This MAS-IA has been tested in a research study with a community of 120 stu-dents at Universidad Autnoma de Madrid (UAM). They had to work like knowledge creators and knowledge evaluators.
The obtained results in this research study were: i) generally students are more organised like knowledge creators than like knowledge evaluators and ii) quality of students evaluations is directly related to whether they are organised or disorgan-ised. These results corroborated the necessity of analysing the users interactions of the KnowCat system, in order to extract what the most relevant interactions are of the users and therefore should have more weight in the Knowledge Crystallization mechanism of KnowCat.
Acknowledgements This research was partly funded by the Spanish National Plan of R+D, project number,TIN2008-02081/TIN; and by the AECID (The Spanish Agency for International Development Cooperation ) project number A/017436/08.
References
1. Alam´an, X., Cobos, R. KnowCat: a Web Application for Knowledge Organization. In P.P. Chen and al.(Eds.). LNCS 1727. Springer, 348-359. 1999.
2. Atterer, R., Wnuk, M., Schmidt, A., Knowing the User’s Every Move - User Activity Tracking for Website Usability Evaluation and Implicit Interaction. In Procs. WWW2006, pp. 2-9, Edinburgh, 2006.
3. Cao, L., Dai, R. Human-Computer Cooperated Intelligent Information System Based on Multi-Agents, ACTA AUTOMATICA SINICA, 29(1):86-94, 2003, China
4. Chang, G., Healey, M., McHugh, J.,Wang, J., Mining the World Wide Web: An Information Search Approach, Kluwer Academic Publishers, 2001.
5. Cobos, R. Mechanisms for the Crystallisation of Knowledge, a proposal using a collaborative system. Doctoral dissertation. Universidad Aut´onoma de Madrid. 2003.
6. Franklin, S. and Graesser, A. Is it an agent, or just a program? A taxonomy for autonomous agents. In Procs. of the Third International Workshop on Agent Theories, Architectures, and Languages, pp 443-454 Spring-Verlag. 1996.
7. Garc´ıa, G., Cobos, R, ESMAP: A Multi-Agent Platform for Extending a Knowledge Man-agement System. In Nishida, T, et.al. (eds) The 2006 IEEE/WIC/ACM WI. Hong Kong, De-cember 2006. 59-65. 2006.
8. Garofalakis, M. N., Rastogi, R., Seshadri, S., Shim, K. Data mining and the Web: Past, present and future. In Procs. WIDM99, 43-47, KC, Mssouri, 1999.
9. E. Han, D. Boley, M. Gini, R. Gross, K. Hastings, G. Karypis, V. Kumar, B. Mobasher, and J. Moore. Webace: a web agent for document categorization and exploration. In Procs. of the 2nd AA, pp 408-415. ACM Press, 1998.
10. Heraud, J., France, L., and Mille, A. Pixed: an ITS that guides students with the help of learners’ interaction log. In Procs. of the ITS, Maceio. 57-64. 2004.
11. Inaba, A. Ohkubo, R. Ikeda, M. Mizoguchi, R, An Interaction Analysis Support System for CSCL: An Ontological Approach to Support Instructional Design Process, In Procs. of ICCE’02, 358-362,2002
12. T. Joachims, D. Freitag, and T. Mitchell. Webwatcher: A learning apprentice for the world wide web. In Procs. of IJCAI97, August 1997.
13. H. Lieberman and D. Maulsby. Instructible agents: Software that just keeps getting better. IBM Systems Journal, 35(3&4), 1996.
14. Marc-Andr´e, B´elanger, S., and Frasson, C. WHITE RABBIT - Matchmaking of User Profiles Based on Discussion Analysis Using Intelligent Agents. In Procs. of the 5th ITS. Vol. 1839. Springer-Verlag, London, 113-122. 2000
15. Srivastava, J., Cooley, R., Deshpande, M., Tan, P. N. Web usage mining: Discovery and ap-plications of usage pattern from Web data. SIGKDD Explorations, 1(2):1-12, 2000. 16. Suh, H. J. and Lee, S. W. Collaborative learning agent for promoting group interaction. ETRI
Journal, 28(4), 461-474.
17. Voyiatzaki, E., Margaritis, M., Avouris, N., Collaborative Interaction Analysis: The Teachers’ Perspective. In Procs. of the 6th IEEE ICALT, 345-349. 2006.
18. Wooldridge, M., Jennings, N.R. Intelligent Agents: Theory and Practice. In Knowledge En-gineering Review, 10(2), 115-152, 1995.
19. Yacine, L. and Tahar, B. (2007). Learners assessment in a collaborative learning system. Asian Journal of Information Technology, 6(2), 145-153.
