Volume 11 Issue 3, ECEL 2012 / Aug 2013

(1)

ECEL 2012: Beyond the gadget

On the 26 and 27th October 2012 the ECEL conference was held at the University of Groningen in the Netherlands. We felt it was important for the conference to focus on practical applications of e-learning. Attending lecturers and educational advisors or policy makers should be enabled to implement the innovations after the conference. Therefore the title to the conference was Beyond the gadget. We wanted to go beyond the gadgets and establish what really matters in e-learning. How to find the real value of learning, and the role of tools therein? More and more we see that e-learning takes its logical place in the design of materials for e-learning. E-e-learning gets integrated in the design process of learning. A design cycle is followed of setting goals, working out how these are assessed, and then filling out the methods of learning. In this way, e-learning is chosen as one of the methods of learning, and is selected for its true value.

We were honoured to host keynotes from Professors Eric Mazur (Harvard University, USA), Fred Mulder (Open University, Netherlands), and Johannes Cronje (Cape Peninsula University of Technology, South Africa). Video registrations of our keynotes, photos, conference abstracts and a backlog of tweets are still available online at http://academic-conferences.org/ecel/ecel2012/ecel12-home.htm. We were only able to invite our international keynotes with the support of our gold sponsor. Thank you Blackboard! Their senior manager solution engineering Dan Peters shared his ideas on e-learning and opening up education.

At ECEL 2012 we saw examples of great course designs. Good practices, showing the real value of e-learning innovations. The eight best examples are presented in this special issue. Arnesen et. al. follow up on our keynote of Eric Mazur by describing their experiences on using student response systems, categorizing them in several pedagogical uses. Two papers are about Open Education Resources. With the enormous amounts of resources now available, finding resources is a problem. Mahoudi et. al describe a frame based approach to help find materials. Gruszczynska et. al. describe ways to build an open source curriculum in the area of teacher development. Developing digital literacy skills is the focus of the paper by Hall et. al.

The use of social media cannot be missed by a conference on e-learning. Grosch provides an overview of student use and ranks 53 media services. Rockwell et. al. describe social backgrounds for minority groups to improve their chance in the job market by making use of social media. Making social connections is stressed in Garcia et. al. Via the use of blogs students focused more on peer-critique and feedback. Providing feedback can be helped by using a diagnostic table, which is described in the paper of Wang and Chen.

The organizing committee of the ECEL2012 conference hopes you will enjoy reading this special issue. Via this issue we hope you will be able to take a small first step to implement ideas as presented here.

Hans Beldhuis and Koos Winnips University of Groningen

(2)

ISSN 1479-4403 169 ©ACPIL Reference this paper as: Arnesen, K., Sivertse K G “ H J E B “ J E W Various Pedagogical Methods Utilizing a Student Response System M L O The Electronic Journal of e-Learning Volume 11 Issue 3 2013, (pp169-181), available online at www.ejel.org

Experiences with use of Various Pedagogical Methods Utilizing a

Student Response System Motivation and Learning Outcome

Ketil Arnesen, Guri Sivertsen Korpås, Jon Eirik Hennissen and John Birger Stav

Sør-Trøndelag University College, Trondheim, Norway

[email protected]

[email protected] [email protected] [email protected]

Abstract: This paper describes use of an online Student Response System (SRS) in a pre-qualification course for engineering studies in Norway. The SRS in use, where students answer quizzes using handheld mobile devices like Smartphones, PADs, iPods etc., has been developed at Sør-Trøndelag University College. The development of the SRS was co-funded by the Lifelong Learning Program KA3-ICT in 2009-2010. SRS has been designed to help teachers effortlessly i) break the monotony of a lecture and allow the students to actively take part in the lecture, ii) increase teacher-student interaction, and iii) give teacher and students immediate anonymous feedback on learning outcome. The response system was used in mathematics in two groups with different lecturers during two semesters in 2009-2010. The pedagogical methods in use

w P I C I

their mobile devices. In both cases the result of the quiz will immediately appear as a histogram on a screen in the classroom. The closing parts will also be identical. The lecturer then highlights the correct option in the histogram and explains why this option actually is the correct one. In the Peer Instruction method there will be an additional element. The first poll will be followed by a discussion in student groups, where the students are urged to defend their choice and convince their fellow students that their chosen option is the correct one. The discussion is then followed by a new individual voting session before the final results are shown and the closing part takes place. The paper will compare this method with the peer instruction method as described in existing literature. The learning outcome will be discussed xperiences from the classroom. In addition we will analyze

We

will present results showing that when students are arguing their point of view, they will have a stronger tendency to convince their fellow students when they themselves already have found the correct option in the quiz. Finally we will suggest pedagogical improvements for future use of response systems in mathematics. Input from lecturers and from students has already been used in the process of developing a new version of SRS, finished in January 2013.

Keywords: student response systems, mobile learning, smartphones, peer instruction and learning, peer learning assessment systems, learning outcome

1. Student response system (SRS)

1.1 Technical equipment

The SRS in use was developed at Sør-Trøndelag University College (HiST), cofounded by the European Commission (EduMecca 2009). In the current version the lecturer must be equipped with a computer, preferably connected to a projector. Each student needs a device capable of accessing a web page, for example a smartphone, iPod, iPad, laptop etc. In addition there must be a wireless network present.

The major advantages with this system compared to traditional clickers are independence of software and flexibility in use of voting device. A quiz might be presented through any kind of computer application. The quiz might also be written on the fly, even on paper or a traditional blackboard. In the testing phase described in this paper we handed out iPods for a one year loan to all students, although some of the students preferred to use their own devices. In the following semester the students could choose between borrowing an iPod or using their own smartphone. As a result more than 80 % preferred to use their own equipment. The advantages of a system using

T

(3)

1.2 Voting procedure

We will differentiate between two methods, Classic and Peer Instruction (PI). Figure 1 shows a flowchart for the two different variants of the SRS session.

Figure 1: Illustration of the SRS session

In both methods the students were given a quiz, usually at the start at the lecture. They were given a couple of minutes to make up their opinion and optionally discuss the problem before casting

individual vote T

computer and be projected onto the screen. The closing session is also similar in the two methods. The lecturer opens for a class discussion where the students are urged to defend their own choices. Finally the lecturer highlights the correct option in the histogram and explains why this is the correct one why the others are not.

The lecturer can choose to arrange a second voting session involving a student discussion in groups consisting of three to five students. This option can be decided upon in advance or on the fly depending on the result of the first vote.

The peer instruction method used here is a slight modification of the original method described by Mazur (Mazur 1997). Originally, it is seen as important that the students make individual decision before the discussions, where they are supposed to defend their own choice for solution and convince their fellow students of its validity.

(4)

Electronic Journal of e-Learning Volume 11 Issue 3 2013

www.ejel.org 171 ©ACPIL

Our reason for suggestion a different approach is mainly to add flexibility. It is usually not necessary to open for student discussion when most of the students agree on the correct solution.

A similar method is suggested by Dufresne et al (Dufresne 1996), but without the lecture explanation before closure.

1.3 Categories of quizzes

We will differentiate between three categories of using SRS:

 R T

 W T

 Introduction: Preparation for a new learning session. 1.3.1 Repetition

The majority of events have been in this category. A voting session in this category is usually held at the start of a new lecture. The basic idea is that if most of the students give the correct answer, it will not be necessary to give any further explanation, but rather move on to the next theme. If the result of the quiz shows that a significant part of the students does not choose the correct solution, this will be a strong incentive for the lecturer to return to the previous lecture and give a new explanation. This may be important particularly in subjects where the learning material is of a sequential nature, as in mathematics. This method will allow the lecturer to vary the lecture

acc D D

1.3.2 Wake-up

The question of whether the students really are following the lecture should be the concern of any lecturer. The most common method is plainly to ask a question and wait for the students to raise

T

understanding, since only a few students will raise their hands, while the majority will remain inactive. Use of SRS provides a method to get answers from the entire group of students, since everyone is guaranteed anonymity. This feedback allows the lecturer to decide whether the lecture should continue as planned or if it is necessary, for instance give one more example or look at the theme from a different angle.

1.3.3 Introduction

In the third category student activity will be more important than the actual feedback. The desired feedback is that a new concept will be introduced to the students either through some sort of open question or a problem that can be solved combining existing knowledge in new ways. One example

T

W troduction to equations. The solution can

be found the hard way by trial and error, but is found with less effort for the ones capable of using mathematic language and set up the equation for the problem. It is interesting to note that only 12 % managed to find the correct solution before they were introduced to the theory of equations, which should give a strong motivation to learn a systematic method for solving the problem.

(5)

2. SRS and mathematics

From autumn 2011 the SRS has been in systematic use at the pre-qualification course at the Faculty of Technology at Sør-Trøndelag University College. The course is aimed at students from vocational schools who need this one year course to be allowed to study for a bachelor or master degree in technology. The annual acceptance of students has been 240 and the students are divided into four groups with approximately 60 students in each group. This study covers the subject mathematics in two groups during the testing period for 2010-11. The two lecturers involved were responsible for one group each, and had the responsibility for developing the quizzes in mathematics and physics respectively. This led to the establishment of a database consisting of 276 quizzes in mathematics and 89 quizzes in physics.

All data present

student interviews are collected from both groups. The subject lasted for two semesters, and the students were given one lecture each day, consisting of 2x45 minutes. Usually the students were given one to three SRS quizzes in each lecture.

Two thirds of the quizzes were held as Repetition, while the last third was almost equally divided into the Wake-up and the Introduction categories. Many of the quizzes were not consisting of conceptual problems, instead focusing on calculation techniques.

2.1 Classic versus peer instruction

There were a total number of 123 quizzes in the course, most of them using the Classic method as can be seen in figure 2. In most cases of Peer Instruction this method was decided upon after the result of the first voting. An unsatisfactory result gives the lecturer the opportunity to choose a student discussion and a new voting. Figure 3 shows the relative distribution of correct answers from each quiz. All quizzes that were expanded to PI have a result in the lower part of the distribution. When the method is decided upon according to the first voting, the students will see the result of the first voting, which is not recommended in earlier literature (Mazur 1997). It is important to notice that the SRS is compatible with traditional Peer Instruction. This method can be chosen in advance to ensure that the result of the first voting sessions will not be shown until the last voting is finished. Then both results will be sequentially displayed.

(6)

Figure 3: Correct answers in classic versus peer instruction 2.2 Improvement after peer instruction

Figure 4 shows the number of correct answers before and after the discussion for the 14 cases where Peer Instruction was used. The percentage of correct answers did improve in all quizzes except in one. The average percentage of correct answers in the first voting session was 34 %, while the number increased to 61 % in the second session. These results strongly support our hypothesis that the students with the correct solution are more likely to convince fellow students to agree with their point of view than the other way around. Earlier research on Peer Instruction in physics teaching for college students (Crouch 2001) also gives similar results.

T Y

your own opinion before the discussion. Then you compare your own answer with the student

Y A

might happen that if you have a theory, it will be proven to be false. And then it perhaps is easier to

remember

Figure 4: Increase in correct answers in 2nd voting session

Even though PI is a very effective teaching method, we will still emphasize the importance of the sion. The students identify the explanation as the most important part of the session in interviews (Hansen-Nygård 2011). A research among biology students actually shows that the combination between discussion in student groups and the

ion will produce maximum learning outcome (Smith 2011).

3. Experiences from use of SRS

T “R“

(7)

We have registered a positive attitude towards SRS from the students. A significant majority of the students are of the opinion that SRS increases both engagement and learning.

F “R“ n to increasing

engagement in classes. More than 70 % of the students claim increased engagement in classes where SRS was used.

Figure 5: Results from student poll on engagement

This is also confirmed in the student interviews. “ I Yes, I will pass it. So, I try to do my best And you suddenly wake up instead of just waiting for the lecture to end. So it definitely helps in raising my spirit

The lecturers also report increased student activity. Starting the lecture with a SRS session is found to provide engagement and participation throughout the entire lecture, not only the SRS session.

F “R“ A % agree,

while none of the students disagree.

(8)

The fact that students claim higher learning outcome in polls does of course not guarantee that this really takes place. There have been many studies on the effect of response system on learning outcome measured by tests and exams.

Mazur reports significant improvements in learning outcome. This is supported by other studies (Caldwell 2007), while others show small improvements (Chen 2010). Some studies still show a neutral effect (James 2012). The variations in learning outcome suggest that the result is dependent

F

methods and relevan

engagement. We find support for this in the student interviews:

I

discussion afterwards. On the other hand, if only half of the class or even less are wrong there will

I T A

their way in the old system with blackboard, lecturer and students. It is working for them. Variation

“ W

This study has focused on different level of the students based on their acceptance points, and we have found support for this hypothesis.

A

leading to fewer primitive errors in assessments, even if there are no statistical evidence so far to confirm this assumption. An analysis of the final student grades including a control group not using SRS (Arnesen 2012) seem to indicate that students with lower grades from vocational school have a lesser tendency to fail the course in the SRS groups than in the control group. Since those students generally suffer most from lack of calculation technique, this result might be interpreted as a support for the assumption on improved skill in performing simple arithmetic operations

In this study we had two groups with different lecturers using SRS and a third control group that did not use SRS. The two SRS groups differed in use of methods. The group analyzed in this paper differed from the second by more frequent use of Peer Instruction (16 quizzes) and more variation between Repetition, Wake up and Introduction. The other group had a more limited number of Peer Instruction (5 quizzes) where almost all quizzes were given as Repetition. The first group will be labeled SRS-P, and the second SRS-C. We also note that there were more quizzes given in SRS-P (126) than in SRS-P (87).

The grading for the subject follows the ICTS standard, A being the highest grade and F representing a failed course. The average grading is obtained by giving 5 points for each A, 4 for each B and so on. An inspection of the final grades for the three groups as shown in table 1 does not strongly support any progress in student performance in the SRS groups.

Table 1. Final grades in mathematics

Group A B C D E F Total Percentage

failed

Average score

Control 7 7 10 12 1 7 44 15,9 2,7

SRS-P 2 6 11 12 1 7 44 15,9 2,2

SRS-C 7 8 8 11 7 5 46 10,9 2,6

We observe that the control group actually has the best average grade score, while the number of failed students is lowest in one of the two SRS groups.

(9)

There are of course a number of factors affecting the grades, examples being performance of the lecturer, student effort and educational level in advance to the course. The most important and available factor is the earlier grading score from vocational school. This is converted to a number between 0 and 60, the acceptance points, when the students apply to Sør-Trøndelag University College. Figure 7 shows the distribution of the performance of the students in the pre-qualification course of 2008-2009 depending on acceptance points. As expected higher acceptance points seem to increase expected grades. The Pearson correlation between acceptance points and grade is 0.33.

Figure 7: Grading depending on acceptance points

The impo

course. Of the 268 students in 2008-2009 180 passed the course, 88 did not finish the course and 60 did not pass the final exam. Figure 8 shows the distribution of students who did pass the course sorted by acceptance points.

Figure 8: Passing ratio based on acceptance points

W

exceed 50. When we compare the three groups we should also take the acceptance points into consideration. As can be seen from table 2 the students in the control group have a higher average level than the two SRS groups.

Table 2: Average acceptance points

Student group Average acceptance points

Control 42.3

SRS-P 40.2

(10)

The acceptance points become even more important when we consider the distribution shown in figure 9.

Figure 9: Distribution of acceptance points

The control group is the only group without a fair share of students in the critical area below 35 points where academic achievements tend to drop drastically (Arnesen 2012). We should then expect better results in this group, both in better grades and higher passing ratio.

The student polls indicate that the students are of the opinion that SRS increases their learning outcome. In the student interviews there are some suggestions that students whose performances usually lies in the lower half, will benefit more from SRS than the better students.

It is therefore of interest to examine the results for the students with the lowest acceptance points in the three groups. We restrict the sample to students with acceptance points below 40.The reason for this is that due to a minor change in the calculation procedure from 2008, points from 2011will be from 5 to 10 points higher than earlier. The students from 2011 with points below 40 would then be expected to be similar to the students from 2008 with points below 50. Those are the students we want to examine with increased learning outcome as SRS effect in mind. The results can be seen in table 3.

Table 3: Results for students with acceptance points below 40

Group Average acceptance

points

Average grades Relative failure

Control 37.5 2.2 0.21

SRS-P 35.1 2.2 0.11

SRS-C 34.5 1.7 0.33

As can be seen from the average acceptance points and figure 9, the control group still has the

“R“ nd most varying use of SRS has better

performance. The last SRS group comes out with the lowest performance, but does also contain the students with the lowest acceptance points and therefore arguably the lowest potential for success. As a final inspection of which kind of students that benefits from use of SRS, we will look into three tests given throughout the semester. The students were graded on a percentage scale, which can give more information than the crude scale of ECTS grades. Table 4 shows the Pearson correlation between and acceptance points and percentage score on each test.

(11)

Table 4: Correlation between acceptance points and results

Group Test 1 Test 2 Test 2

Control 0.56 0.55 0.35

SRS-P 0.05 0.15 0.11

SRS-C 0.31 0.31 0.29

These results strongly support the theory that SRS helped improving results among the students with the lower acceptance points. All groups have a positive correlation, but it is considerably lower in the SRS groups, especially in the SRS-P with the most frequent use of SRS.

T

method (Mazur 1997). We have found that while few students usually will contribute to a discussion involving the entire class, these inhibitions seem to lessen when they are urged to defend their answer in a SRS session. We also have the impression that starting the lecture with the use of SRS will more easily get the students into a learning mode and to participate more actively in the entire lecture.

Another important reason for having the class discussion is the opportunity to reveal the possibility that the correct solution is reached based upon erroneous reasoning or conceptual misunderstandings. Recent research (Nielsen 2012) shows that in some cases the majority of students will reach the correct answer based on a misconception of fundamental theories. The class discussion can then be an opportunity for the lecturer to address and correct these misconceptions. One specific challenge is when the presented problem consists of performing basic mathematical operations, as for instance solving an equation or an integral. In these cases it will often be easier to find the correct answer by testing all the options in the q

solving the problem independent of presented options. One way to avoid this is to hold back the options until the students are casting their votes. In this case the students will have to rely upon their calculated answer instead of testing the different options. When this procedure is used, many students will reach a solution not listed among the given options. Therefore, we suggest adding

N “ then need to have this

option as the correct answer for the students to be able to trust their answer even if it is not listed as a possible solution.

Unfortunately we do register a decrease in the number of participating students during the two semesters. Figure 10 shows the development of participants from the first to the last quiz.

(12)

This can partly be explained by a decreasing number of students. While 61 students started, only 43

I

semester. In some of the quizzes the number of present students was recorded, and figure 11 shows the percentage of participating students in these cases.

Figure 11: Development in percentage of voting students

The 95 % confidence interval for the slope of the regression line is given by (-1.15 ± 1.77) %. The probability for a negative valued slope is 80 %, meaning that the hypothesis of a possible decline in the percentage of participation is not statistically verified. The statistics for the regression line are shown in table 5.

Table 5: Regression for development of participation Coefficients Standard

Error

t Stat P-value Lower 95% Upper 95% Intercept 0,71544556 0,06564733 10,8983186 3,9501E-12 0,58155694 0,84933418 Slope -0,00114511 0,0008703 -1,31576487 0,19790098 -0,0029201 0,00062988 Another interesting angle is to compare the amount of participation during the two semesters. The data show a decrease from 66 % to 62 %. There is no statistical evidence for a difference between the two samples, This is seen in the statistics in table 6,

Table 6: t-test on difference in participation by semesters

1st semester 2nd semester

Mean 0,65726382 0,61642325

Variance 0,02547268 0,02146644

Observations 16 17

Hypothesized Mean

Difference

0

df 30

t Stat 0,76437401

P(T<=t) one-tail 0,2253076 t Critical one-tail 1,69726089

(13)

Still there is a fact that a significant part of the group is not participating. Explanations may be technical problems or lack of interest from the students. One of the interviewed students suggested

Some time you can blame it on technical problems. You might get

P B

Y A

given in the interviews is that they often will choose not to answer if they do not have any preferred

I D P

to make a wild guess, because if they are lucky and get the correct answer, the lecturer will get the impression that everyone has learned the stuff and just carry on

T W D

participation increases from 60 % to 68 %. Th D

participation is still not statistically valid within a significance level of 0.05. As can be seen from table 7, the actual significance level is 0.08

Table 7: Relevance of "Don't know" option

W D W D

Mean 0,678662254 0,603140112

Variance 0,0223179 0,022523249

Observations 14 19

Hypothesized Mean Difference

0

df 28

t Stat 1,432469891

P(T<=t) one-tail 0,081540007 t Critical one-tail 1,312526782

4. Emerging technologies

A new version of the SRS is finished in january 2012. The most notable changes are a new and improved user interface, allowance for more multiple users and access to earlier voting results. The Done-IT project, cofounded by the European Commission, aims at developing an assessment system based on the described response technology using smartphones or pods (Done-IT 2011). The pedagogical idea is to change the assessment into an area for learning as well as evaluation. This can be achieved by using SRS technique for immediate feedback on student scores to individual questions. If one or more of the questions are unsatisfactory, the lecturer may then give a short lecture, allow group discussion etc. followed by a new voting session giving the students a second chance to improve their assessment (Thorseth 2012). The assessment system will be tested in the pre-qualification course during the autumn semester 2012.

5. Conclusion

Studies have shown that SRS either has benign or neutral effect on learning outcome. We have suggested three methods for use of SRS in mathematics, Repetition, Introduction, and Wake up. All methods depend on the method introduced by Mazur. We have focused on what type of students will benefit from SRS, and our findings support the theory that SRS will increase learning outcome among students who usually have academic achievements in the lower part of the group. We still emphasize that this may not be true in other subjects and for other student groups. The varying

(14)

results from different studies indicate that more research is needed to find and systemize the factors of success for use of clickers in education.

6. Acknowledgements

These results have been obtained with support from the European Commission. This publication reflects the views only of the authors, and the Commission cannot be held responsible for any use, which may be made of the information contained therein.

7. References

Arnesen K., Stav J. B., Hansen-Nygård G., Korpås G. S. and Talmo T. (2012) Evaluation of Use of Student Response System in Pre-Qualification Classes for Engineering Education, Proceedings from the International Technology, Education and Development Conference (INTED 2012), 5-7 Mars, Valencia, Spain, pp. 5077-5083, International Association of Technology, Education and Development (IATED).

Caldwell J. E. (2007) Clickers in the Large Classroom: Current Research and Best-Practice Tips, CBE Life Science Education, Vol. 6, Spring 2007.

Chen J. C., Whittinghill D. C. and Kadlowec J. A. (2010 C T C F R F E “ L and Satisfaction. Journal of Engineering Education, 99(2), pp. 158-169.

Crouch C. H. and Mazur E. (2001) Peer Instruction: Ten Years of Experience and Results, American Journal of Phsycs, 69 (9), p. 970-977.

The Done-IT Project (2011) online at www.histproject.no. This is a LLP KA3-ICT Project, contract 511485-LLP-1-2010-NO-KA3-KA3MP, which was cofounded by the European Commision

Draper S. W. and Brown M. I. (2004) Increasing interactivity in lectures using an electronic voting system, Journal of Computer Assited Learning 20, pp 81-94.

Dufresne R. J., Gerace W. J., Leonard W. J., Mestre J. P. and Wenk L. (1996) Classtalk: A Classroom Communication System for Active Learning, Journal of Computing in Higher Education, 7, pp. 3-47.

The EduMecca Project (2010) online at www.histproject.no. This was a LLP KA3-ICT Project, contract 143545-2008-LLP-NO-KA3-KA3MP, which was cofounded by the European Commision.

Hansen-Nygård G., Nielsen K. L., Stav, J. B., Thorseth T. M., and Arnesen, K. (2011) Experiences with Online Response Technologies in Education of Engineers, Proceedings from the International Conference on Computer Supported Education (CSEDU 2011) conference, May 6-9, 2011, Noordwijkerhout, Netherland, CSEDU2, p 383-391, SciTePress. James M. C. and Willoughby S. (2011) Listening to Student Conversations During Clicker Questions: What You Have Nor

Heard Might Surprise You!, American Journal of Physics, 79, 123; doi: 10119/1.3488097.

Nielsen K. L., Hansen-N G “ J B I P I H I V “ A “ E G D ISRN education, Volume 12, Article ID 290157, 8 pages.

Mazur E. (1997) P I A U Manual, Upper Saddle River, NJ, Prentice Hall

Smith M. K., Wood W. B., Krauter K. and Knight J. K. (2011) Combining Peer Discussion with Instructor Explanation Increases Student Learning from In-Class Consept Questions, CBE Life Sciences Education, vol. 10, no 1, pp. 55-63. Thorseth, T. M., Hansen-Nygård G., Pein R. P., Stav J. B., Arnesen K. (2012) Designing and developing peer learning

assessment services for smartphones and pad, Proceedings from the International Technology, Education and Development Conference (INTED 2012), 5-7 Mars, Valencia, Spain, p 1354-1360, International Association of Technology, Education and Development (IATED).

(15)

ISSN 1479-4403 182 ©ACPIL

R M M T T F B K A “ R -based Framework for Efficient

R E M

Educational Materials

Maryam Tayefeh Mahmoudi

1, 2,

Fattaneh Taghiyareh

1

, Kambiz Badie

2 1

School of Electrical and Computer Eng., College of Eng., University of Tehran, Tehran, Iran

2

Knowledge Management & E-Organizations Group, IT Research Faculty, Cyber Space

Research Institute (Ex. Research Institute for ICT), Tehran, Iran

[email protected] [email protected] [email protected]

Abstract: Retrieving resources in an appropriate manner has a promising role in increasing the performance of educational support systems. A variety of works have been done to organize materials for educational purposes using tagging techniques. Despite the effectiveness of these techniques within certain domains, organizing resources in a way being adequately reusable for support purposes is still in the offing. In this paper a semantic approach is proposed to increase performance of retrieving educational materials based on using frames. Here, frames are used to represent the very knowledge necessary for realizing the similarity/ relevance between query and supportive materials. Owing to the complexity in semantic handling of the entire text, the suggested frame-based approach is applied only to the titles or sub-titles, or in general the main headings, in the material. To make these frames comprehensive, we have made use of two

attributes call M C B C

W W

entity (How / in What way a C entire ideas belonging

to headings (titles or subtitles) in a material. These attributes seem to have enough potential for representing the knowledge of titles and sub-titles in a way reflecting the content of the paragraphs in a reasonable way. To evaluate the capability of the proposed approach, retrieving materials within the domain of Multi-Agent Systems (a subject of high concern in Artificial Intelligence) was picked out as the benchmark problem. According to this benchmark, materials are

retrieved T M -Agent

Systems as an educational resource in academia, within which a number of use

considered as possible queries, and the corresponding materials were then retrieved using the proposed approach. Computer experiments show acceptable precision and recall values for these queries with a quite good balance between them which is represented in terms of F-measure. . The findings lead us to the fact that "Major Characteristics" and "Basic Constituents" have the ability to increase the status of re-usability for the stored materials. Moreover, the fact that materials can be reused efficiently, leads us to the point that our proposed representation scheme can be useful for

to the extent that several materials

ought to be merged together to yield the requested material.

Keywords: Semantic retrieval, material retrieval/ reuse, educational materials, frame-based representation, frame attribute, major characteristics, basic constituents

1. Introduction

Nowadays, increasing digital resources, including educational materials, have made a great challenge for innovating techniques to solve the problem of finding the truly-necessary information. It is interesting to notice that, when it comes to educational support systems, facilitating the way that learners get their appropriate resources becomes remarkably important.

Within the above scope, many annotating, tagging and indexing techniques have been created for this purpose. These techniques generally make use of WordNet and meta-data as appropriate knowledge representation schemes to describe the resources (Zhao et al, 2008) (Kohler et al 2006) (Dobsa, 2007) (Roy et al, 2008). Also, Latent semantic indexing (LSI) and concept indexing (CI) are among those techniques which are capable of organizing educational assets and offer effective search and categorization services (Dobsa, 2007)(Gómez et al, 2004). While these methods improve the detection of relevant documents on the basis of the terms found in queries, there also exist

(16)

some query reformulation techniques whose effort is to make mapping between tagging and querying vocabularies (Bischoff et al, 2010) in an acceptable manner.

In sum, statistical & semantical techniques are the two major categories that however belong to indexing, annotating and tagging (Moreda et al, 2007) (Zhang et al, 2011). It should be noted that, despite the advantages & effectiveness of these techniques within certain domains, they suffer from their own limits and deficiencies particularly when it comes to organizing resources with regard to inadequacy of reusability for support purposes.

To overcome these deficiencies, in this paper, we introduce a frame-based semantic technique to enhance the retrieval mechanism for educational resources. The proposed technique is, not only capable of resolving the existing ambiguity in tags, but can also embed the focal knowledge for exploring the similarity relevance between query and supportive materials. The frames used in the suggested technique are benefited by two attributes called "Major Characteristics" and "Basic Constituents" which stand respectively for "the goal behind a concept" and "the elements that support a concept to be realized" (Mahmoudi et al,2004) (Badie et al, 2008). These attributes seem to be comprehensive enough to reveal the knowledge behind headings of a material as well as the status of learner's query. With regard to this, improving the semantic ability of retrieving educational resources through Major Characteristics (MJ) and Basic Constituent (BC) becomes the prime concern of our paper.

The rest of the paper is organized as follows: Section 2 reviews some of the previous works that have been done in the areas of indexing, annotating and organizing educational resources. Section 3 describes our proposed approach, while, in Section 4, experimental results are analyzed. Section 5 includes conclusion and future works.

2. Related works

Mankind is facing too many information resources such as: tutorials, books, learning materials, reports, case studies and practices, etc. which are to be used for educational purposes. Either organizing and retrieving appropriate educational resources or classifying them are thus major issues in learning environments. Within this context, there exist various systems based on semantic retrieval that are capable of organizing educational assets. Content management (Shao et al, 2003), information retrieval (Liu et al, 2008), question-answering (Moreda et al, 2011), classification (Thorleuchter et al, 2013), recommender (Zheng et al, 2011), educational support and intelligent tutoring (Günel et al, 2010) systems, can also be enumerated as means for this purpose. These systems are equipped with various types of semantic approaches such as annotating, tagging and indexing, that may facilitate resource organization process.

One approach to such issue is to make use of ontology-type structures such as WordNet and meta-data which is in fact a way to describe the resources in a neat and efficient way (Lohmann et al, 2008), while tag and time are mostly useful in predicating user's preference and recommending related resources. It is to be noticed that a tag performs as a bridge between a user and a resource

and the more frequently a tag is used, it means that the more the user is interested in the related resource (Zheng et al, 2011). Generating tags in a collaborative way that is called folksonomy (Bateman et al, 2007) from the one side and using tags and tags clouds to discern credible content in online message forums (Grady et al 2012) from the other side, can also have a significant role in annotating and categorizing resources, especially for adaptable online learning purposes. Apart from simple annotation methods, there also exist some co-constructed semantic space for information fusion which exploits effective annotation (Lee et al, 2012). Besides, some light-weight techniques and tools such as Cerno have been proposed

(17)

for legacy code analysis and mark-up towards semi-automatic semantic annotation of textual documents according to a domain-specific semantic model (Kiyavitskaya et al, 2009).

Indexing has also a significant role in retrieving and processing the educational contents and resources (Mahmoudi et al, 2011). Various algorithms, approaches and networks are applied for indexing purposes such as: Latent Semantic Indexing (LSI) (Thorleuchter et al, 2013), Enhanced Instance Retrieval Network (Lourenço et al 2010), and terminologies (Dinh et al 2012). In all these

are semantically related to this content.

Statistical models, natural language processing, multi-label classifiers, and collaborative techniques are in the meantime most commonly used for tag recommendation (Alepidou et al, 2011). It is obvious that bridging the gap between tagging and querying vocabularies can also yield improving the potential of resource organization systems (Bischoff et al, 2010).

Web query analysis for different application domains using semantic and linguistic knowledge have also the ability to illustrate how far a higher number of relevant resources can be retrieved (Conesa et al, 2008). Syntax-based query reformulation (SQR) and query cluster summarization (QCS) have in the meantime the ability to enhance the performance of information retrieval (Lioma et al, 2008) in this regard. There is no doubt that structured document retrieval (SDR) leads to a better retrieval performance in terms of both precision and functionality especially for textual resources (Liu et al, 2008). Semantic roles extracted from natural language texts have also been shown to be important for improving the semantic information performance of question answering systems (Moreda et al, 2011). Moreover, corpus-based approaches, that make use of statistical models to determine the semantic role of constituents of a sentence, have also been shown to be useful for both information retrieval and question answering purposes (Moreda et al, 2007).

As a conclusion, to increase the speed and efficiency of education supportive systems as well as to have a flexible and reusable repository of e-learning materials, it would be crucial to perform annotation of the document with special metadata in a way as automatic as possible (Roy et al, 2008). Although the mentioned approaches to annotating, tagging and indexing are widely used in different search, retrieval and text processing applications, they still suffer from deficiencies in their semantic potentials. To enhance their semantic capabilities, structures such as frames with comprehensive attributes and values may help a lot. In this respect, determining informative attributes as those we have proposed, seems to be an appropriate solution and capable enough to reveal the main purposes behind phrases. . In this regard, enhancing the semantic ability of the retrieval process and reusability of education supportive materials through using frames with particular attributes are our main concern in this paper.

3. The proposed approach

3.1 Basic idea

As it was mentioned before, in large-scale databases of education supportive materials, using a well-defined semantic approach plays a significant role in retrieving desired resources. In this respect, we propose a frame-based semantic retrieval approach that seems to be capable enough to facilitate such a process. Figure 1 illustrates the details of the proposed framework.

As illustrated in Figure 1, each time a query is presented; it should first be grammatically analyzed and be compared with the "titles" of the existing supporting materials within the data base. For this purpose, extracting the title of supporting material and its parsing are necessary, in order to figure out the values of corresponding "Major Characteristics" and "Basic Constituents" as the major attributes.

(18)

Insert a Query

Extracting query's MJ & BC automatically

Search for synonyms of Query's terms in WordNet

Search in DB of supporting materials based on MJ & BC Rules

Return the corresponding material POS

Tagger

Identifying Basic Constitute Part; { Applying Part of Speech Tagging; Analyzing Prepositions;

Identifying Basic Constituents; }

Identifying Major Characteristics;

{ Determining the layers of MJ by analyzing prepositions; Determining the grammatical role of each layer by POS tagger;

{ Identifying Action Part; Identifying Adverb Part; Identifying Direct Object Part; Identifying Indirect Object Part; }

}

Found it? N

Y Data

Base

WordNet

Figure 1: Details of the proposed framework

Figure 2 illustrates the pseudo code of our suggested approach. Here, semantic rules as well as frames can be used to realize such a process. Frames are helpful in the sense of determining grammatical roles of the existing terms in queries and titles with regard to education supportive materials in a database. Two significant attributes that are considered for this purpose are "Major Characteristics (MJ)" and" Basic Constituents (BC)". "Major Characteristics" is the attribute which explains the main objective behind using a material, while "Basic Constituents" mainly focuses on the methods, techniques or tools which are used to realize this objective (Badie et al, 2008)(Badie et al, 2004).

(19)

Inserting a query; Parsing the query;

Extract (MJ, BC) of query;

{ Identifying Basic Constitute Part; { Applying Part of Speech Tagging; Analyzing Prepositions;

Identifying Basic Constituents; } Identifying Major Characteristics;

{ Determining the layers of MJ by analyzing prepositions;

Determining the grammatical role of each layer by POS tagger; { Identifying Action Part;

Identifying Adverb Part; Identifying Direct Object Part; Identifying Indirect Object Part; } }

}

Search in Database of supporting materials' titles (MJ, BC); If (query (MJ, BC) = supporting materials' titles (MJ, BC));

Send existing response based on relatedness of query (MJ, BC) to title (MJ, BC); Else

Finding synonyms of terms of query in WordNet;

Repeat the whole process to extract appropriate supporting materials for that; Figure 2: The pseudo code of suggested approach

Having studied several titles within supporting materials, we acquired some rules, which were subsequently used to extract "Major Characteristics" and "Basic Constituents" from a "title". Certain conjunctions and prepositions can be in charge of specifying the values of these attributes. (Mahmoudi et al, 2011). It has been found out that some propositions being used in a conjunction, like "in" and "for" followed by a verb usually yield two layers for "Major Characteristics", which follow the same structure including four main parts of "Action", "Adverb/Adjective", "Direct object" and "Indirect object". "Action part" is mainly a verb, while, "Direct object" is a noun or a pronoun that becomes subject to this verb or shows the result of the related action. It is able to answer "What?"s or "Whom?"s relating to this verb. In addition to the grammatical roles, "Indirect object" is also the recipient of the "direct object" and has the ability to answer "To whom?"s or "For whom?"s and it usually follows a preposition. The last part belongs to "Adverb/ Adjective", which can modify verbs, adjectives, clauses, sentences, and other adverbs. It typically answers "How?"s, "In what way?"s, "When?"s, "Where?"s, and "To what extent"s, etc.

For BCs, most of the time, one layer at maximum seems to be sufficient. Determining BC is therefore closely related to the conjunctions which are considered for this purpose. Some of these conjunctions are "based on", "on the basis of", "on the ground of", "using", "making use of", "taking into", etc (Mahmoudi et al, 2011).

It is to be noticed that, employing a POS tagger can facilitate the process of determining grammatical role of terms in query as well as supporting materials. For example, consider the title "Considering agent mobility architecture for controlling transportation based on FIPA standards". As it was mentioned, the terms coming after "based on" would stand for BC, while those coming before "based on" stand for MJ. It is to be noted that Major Characteristics in this example includes two

(20)

layers which are separated by "for". The grammatical role of each part and their components are represented in Figure 3.

Major Characteristics (MJ) Action-

part Adverb-part

Direct Obj. Indirect

Obj. Layer 1

(MJ1)

Considering

-

agent mobility architecture - Layer 2

(MJ2)

controlling - transportation -

Basic Constituents (BC)

1st Layer 2nd Layer

FIPA standards -

Figure 3: Major characteristics & basic constituents for the example of "considering agent mobility architecture for controlling transportation based on FIPA standards"

Having reviewed large amount of titles, several semantic rules are yielded for distinguishing MJs and BCs, For example:

 IF a "word" or "phrase" comes after "via" or "based on" THEN it is most probably a BC.

 IF the rest of the title consisting of a verb comes after "for" THEN it is most probably a MJ's 2nd layer.

The same rules can be applied both for analyzing the queries and the titles as well. After extracting MJs and BCs in a query, based on the rules discussed above, the process of searching for the learning materials (whose "MJ"s and "BC"s are identical to those in the query) would become subject to performance. Having found the corresponding terms, the related material will then be retrieved. Otherwise closely related words or synonyms from WordNet have to be extracted for the same purpose. For the moment a WordNet with simple relational structures has enough potential to respond successfully to our study. Retrieving process will continue in this way, and in the cases where no related term was found, the process will be terminated and a failure notice will then be issued. Types of conjunction/preposition and their status with regard to the attributes, and Rules distinguishing MJs and BCs are illustrated respectively in Table1 and Table2.

Table1: Types of conjunction/proposition and their statues with regard to the attributes Name of Attribute Possible Types of Conjunctions or

Propositions

Status with regard to the highlighted attribute

Major Characteristics

With the purpose of With the purpose of X With the aim of With the aim of X

In order to In order to X

With the objective of With the objective of X

For/ in Y for/in X (sentence with Action part)

Basic Constituents

Based on Y Based on X

On the basis of On the basis of X Using/ via/ by Y Using/via/by X On the ground of On the ground of X

Making use of Making use of X

(21)

IF

X for Y(Y is non-action type noun)

THEN

Y is Adverb part in the value of MJ X for/ in Y (Y is a phrase including action part) Y is second layer value of MJ

X based/ -oriented/-inspired Y X is Adverb & Y is Direct Object in the value of MJ

X of Y(Y is non-action type noun) Y is Indirect Object in the value of MJ X based on/via/by/using Y Y is Basic constituents

A T X X

X X M

C A Y X X Y

X X

B C L

basic motive for extracting such values is to see what the main purpose of the entire heading is, and

C T X Y Y

X M

C Y

M C A X Y

Y - Y

4. Assessment of the proposed Approach

4.1 Experimental set-up

In order to evaluate the proposed approach, we made a data set including 134 supporting materials in the domain of Agent Science and Technology. To perform our tests, we designed some questions to help finding proper materials as the answers for these questions. Results were evaluated through comparing the responses made by our approach with the real responses obtained from experts in the domain of Agent Science & Technology.

4.2 Experimental Requirements

To apply frames to headers, we first split the header into the corresponding "Basic Constituents" and "Major Characteristics". Based on the information obtained in such a manner it is determined whether the value of "Major Characteristics" holds one layer or two layers. Here, it is essential to find out the grammatical role of the terms included in the values of "Major Characteristics" as well as those included in the value of the "Basic Constituents".

To realize the grammatical roles of the terms, we may make use of rules to decide what role a term can hold. For instance, to realize an "action-part" (in the value of "Major Characteristics"), rules can take into account the information regarding suffixes like "tion", "sion", "ment", "ing", etc. which are

L PO“

verb, and that word to be not located between two nouns, one may conclude that the term must be an "action". An example for applying such a rule is illustrated in Figure4.

To identify the "adverb" part, we make use of adverb and adjective tags already produced by Stanford POS tagger.

4.3 Analysis of the Results

To evaluate our approach, a dataset comprising of supportive materials was used. .Results demonstrate the fact that our proposed approach has the potential to function well with regard to detecting the values of "MJ"s and "BC"s, included in the titles. It should be mentioned that out of the 134 titles used in our experiments, our approach has been able to function properly in 107 cases. As we shall show, the amount of precision in detecting the values of "MJ"s and "BC"s is 93%, while the same amount for recall is 79%.

(22)

"Simulation of Dialogue Management"

Simulation

Simulate



is a verb

Not between two nouns



is an action

"Reliable transactions in multi-agent systems"

Transaction

Transact



is a verb

Not between two nouns



is an action

Figure 4: Samples of a rule-based approach for identifying the action part

In addition to the above experimentation, we designed some questions that cover a large variety of possible cases. Respecting this, both "verbs" and "objects" were given as the input, and the names of "appropriate materials" were then returned as the output. Table3 illustrates the detailed information regarding each question.

As the final stage in evaluation, we made use of "F-measure" to totally assess the efficiency of our approach. "F-measure" is indeed for the purpose of measuring the very essential balance which is to exist between "recall" and "precision". To determine "F-measure" we need both "precision" and "recall" values. "Precision" is measured as the proportion of relevant retrieved documents to the number of retrieved documents, while "Recall" is measured as the proportion of relevant retrieved documents to the total number of relevant documents. In the meantime F-measure shows the harmonic mean of these two functions.

Experimental results reveal that the value of precision belonging to our approach is equal to (31/33) = 0.93. Also, the value of recall was found to be 31/ (31+8) =0.79, since eight materials were left out (Figure5). Taking these two values into account, the value of "F-measure" was determined in the following way.

F-measure = 2 * (Precision * Recall) / (Precision + Recall)

(23)

Table3: Precision & recall of some queries Query Automatic approach Non-automatic Approach P re ci si o n R e cal l R e le v an t R e tr ie v e d R e tr ie v e d R e le v an t P re ci si o n R e cal l D e sc ri p ti o n T y p e Relation-determination

What is the relation between communication and agents?

1 1 4 4 4 1 1

What is the relation between protocols and agents?

1 0.5 2 2 4 1 1

What is the relation between web and agents?

1 1 6 6 6 1 1

Causality- determination

What are the reasons of deception in networks of mobile sensing agents?

1 1 1 1 1 1 1

Outcome- determination

What is the outcome of Engineering of multi-Agent?

1 0.33 1 1 3 1 0.66

Historical- determination

What is the history of

multi-agent systems? 1 1 2 2 2 0.5 0.5 How was the evolution

of applications with agents?

1 0.5 1 1 2 1 1

Definition

What is the definition of multi-agent systems?

0.

5 0.4 2 4 5 1 0.8

What is the definition of

an assistance agent? 1 1 2 2 2 1 1

Comparison

What are the differences between soccer-playing intelligent robots with a multi-agent system and regular one?

1 1 1 1 1 1 1

Solution- Determination

What are the multi-agent solutions in power engineering

applications?

1 0.3 1 1 3 1 1

In cl u si o n

Role What is the role of

ontology? 1 1 1 1 1 1 1

Application

What is the application of multi-agent in knowledge retrieval?

1 1 5 5 5 1 0.8

What is the usage of

conceptual maps? 1 1 1 1 1 1 1

What is the usage of

learning automata? 1 1 2 2 2 1 1

Advantages& disadvantages

What is the problem of

(24)

0.7 0.75 0.8 0.85 0.9 0.95

1

Precision Recall

Figure 5: Precision & recall of the proposed approach

As it is understood from the achieved F-measure, our proposed approach has been capable of avoiding irrelevant results. That is because of following a specific pattern in MJ and BC parts for each type of question. In fact, we tried to design some patterns that avoid producing irrelevant results. One should however not forget the fact that applying WordNet itself has also been helpful in this regard.

5. Concluding remarks

In the paper, we demonstrated how attributes called "Major Characteristics" and "Basic Constituents" can be used to realize the process of semantic retrieval of education supportive materials in an efficient way. These attributes were shown to be potential enough for representing the knowledge belonging to titles and sub-titles in a way reflecting paragraphs content in a reasonable way. Here, some rules based on a variety of grammatically significant propositions and conjunctions, were used to detect the values of "Basic Constituents" and "Major Constituents" in the titles and subtitles. Rules can be constituted based on the existing linguistic knowledge. It however should be noted that the higher number of propositions in a rule, a higher expectation may exist with regard to its effective role in detection. For the moment to avoid extra computation, rules have been decided to include only a few predicates. However, developing more potential rules through considering further predicates and applying complicated thesaurus to match alternative words can be regarded as major research works for future. This calls for further analysis of the existing titles as well as sub-titles in the existing materials with the purpose of discovering a wide range of conjunctions and prepositions as essential requirements, for constituting adequate rules.

As the final point, it should be noted that the suggested approach to retrieval of supportive materials can be adopted as a popular approach to retrieval due to its ability in processing texts (textual information) with no particular emphasis on using natural language processing, which can be both complicated and time-consuming in nature.

References

A ) I V K N M P A A “ T R F or Collaborative

T “ IEEE I C P “ R T IEEE I

Conference on Social Computing, pp 633-636.

Badie, K. and Mahmoudi, M.T (2008) A Computational Framework for Manipulating an Issue from the View-Point of Other Issues, 14th Intl. Cong. of Cybernetics and Systems of WOSC ICC“ W P -12.

B K M M T A ECCBR

(25)

Bateman,S., Brooks,C., McCalla, G. and Brusilovsky.P.(2007) Applying Collaborative Tagging to E-Learning, In the Proceedings of the Workshop on Tagging and Metadata for Social Information Organization, Banff, Canada.

B K F C “ N W P R B A

IR W “ “ “ A ents on the World Wide

Web, Vol 8, pp 97-109.

C J “ V C “ V I - D

Knowledge Engineering, Vol 66, No.1, pp 18-34.

Dinha, D., Taminea, L., Boubekeurb, F. (2012) "Factors affecting the effectiveness of biomedical document indexing and retrieval based on terminologies", Artificial Intelligence in Medicine, Article in Press.

Dobsa, J. (2007) "Comparison of information retrieval techniques: Latent semantic indexing (LSI) and Concept indexing (CI)", Solomonovi seminarji, Faculty of Organization and Informatics, Varazdin, University of Zagreb.

Gómez, J. M., Cortizo, J.C., Puertas, E. and Ruiz, M. (2004) "Concept Indexing for Automated Text Categorization", Natural Language Processing and Information Systems, Lecture Notes in Computer Science, Vol 136, pp 495-502.

Grady, L. O., Wathen, C.,N., Burger, J., C., Betel, L., Shachak, A., Luke, R.,Hockema, S., Jadad, A., R. (2012) " The use of tags and tag clouds to discern credible content in online health message forums", International Journal of Medical Informatics, Vol 81, pp 36 44.

Günel, K. and A R E L C E T I T “

A E “ A V -5022.

Kiyavitskaya, N., Zeni, N, Cordy, J., R., Mich, L., Mylopoulos, J. (2009) "Cerno: Light-weight tool support for semantic annotation of textual documents", Data & Knowledge Engineering, Vol 86, pp 1470-1492.

K J P “ “ M R A O

Knowledge-Based Systems, Vol 19, pp 744 754.

Lee,C.H, Wang, S.H.(2012) "An information fusion approach to integrate image annotation and text mining methods for geographic knowledge discovery", Expert Systems with Applications, Vol 39, pp 8954-8967.

Lioma,C O I A - I

Processing and Management, Vol 44, No.1, pp 143-162.

L “ M M C A C “ J A l (SDR) technology to improve

C I V 16.

Lohmann ,S., Thalmann ,S. , Harrer A. and Maier,R.(2008) Learner-Generated Annotation of Learning Resources Lessons from Experiments on Tagging, Proceedings of I-KNOW I-MEDIA '08, Graz, Austria, pp 304-312.

Lourenço, A., Carreira, R., Peña, D.G., Méndez, J.R., Carneiro, S., Rocha, L.,M., Díaz, F., Ferreira, E., C., Rocha, I., Riverola, F., Rocha, M. (2012) "BioDR: Semantic indexing networks for biomedical document retrieval",Expert Systems with Applications, Vol 37,pp 3444 3453.

Mahmoudi, M.T., Taghiyareh, F., Rajavi, K. and Pirouzi, M.S. (2011) A Context-Aware Framework for Semantic Indexing of Research Papers, The Fourth Inl. Conf. on Information, Process, and Knowledge Management (eKnow 2011), Valencia, Spain.

M M T B K C

IKE I C nf. On Information & Knowledge Engineering, Las Vegas, USA.

M P L H “ E P M C

I P M V 885.

Moreda, p., Navar B P M C - D

Knowledge Engineering Vol 61, pp 467-483.

R D “ “ G “ A E P M L C I J A ificial

Intelligence in Education, Vol 18, No.2, pp 97-118).

Shao, N.W.Y., Yang, S.J.H. and Sue, A.Y.S. (2003) A content management system for adaptive learning environment. Multimedia Software Engineering. Proceedings. Fifth International Symposium on Multimedia Software Engineering

I“M“E -214.

Thorleuchter, D., Poel, D.,V. (2013) " Technology classification with latent semantic indexing", Expert Systems with Applications, Vol 40,pp 1786 1795.

Zhang, W., Yoshida, T. and Tang, X. (2011) A comparative study of TF*IDF, LSI and multi-words for text classification, Expert Systems with Applications, Vol 38, No.3, pp 2758-2765.

) N F F F L A O - M T M M I

Conference on Computer Science and Software Engineering, pp 483-486.

) N L Q A E