Mining Students’ Behaviors in Web-based Learning:
Matching vs. Mismatching
995204009張莉苹 995204010陳宏勳 995204011黃霈仁
網學所 網學所 網學所
Abstract
This study examines the effects of matching and mismatching on web-based learning. In particular, we emphasize on how prior knowledge affect students’ reactions to a matching/mismatching environment. More specifically, reading strategies and learning performance are examined in this study. The results indicate that students in mismatching environment tend to take a concentrated reading strategy while students in matching environment tend to take an efficient reading strategy. However, the former does not perform as well the later. In summary, the matching/mismatching have great effects to student learning.
Key Words:
Matching/Mismatching, prior knowledge, learning performance1. Introduction
In recent years, there is a quick development of Internet technology, which makes Web-based learning system become a major learning tool. Thus, a lot of effort has been put for the design of Web-based learning system. Previous research showed that the design of Web-based learning systems matched with students’ needs can improve their learning performance [1]. On the other hand, mismatching Web-based learning systems has negative influences on their learning performance. However, previous research examining the effects of matching and mismatching mainly focuses on cognitive style and neglects prior knowledge.
Prior knowledge is another influential human factor. Previous research indicates that students with different levels of prior knowledge show different learning behaviors [2]. In general, the students with low prior knowledge do not perform as well as those with high prior knowledge. Therefore, it is necessary to understand how to help the students with low prior knowledge to improve their learning performance. In particular, there is a need to identify difficulties they face in a learning environment mismatched with their needs.
Hence, this study aims to discuss the effects of matching and mismatching environments on student learning. More specifically, students with low prior knowledge will have to use two kinds of learning environments that are matched with and mismatched with
their needs, and then their learning behaviors showed in these two environments will be compared. By doing so, the difficulties they will meet in a mismatching environment can be identified. Such knowledge will be useful to understand how to design a learning environment that matched with the needs of students with low prior knowledge so that their performance can be enhanced.
2. Related Works
A Web-based learning system provides students with multiple ways so that they can decide how to interact with the Web-based learning system by themselves. Therefore, students may have diverse preferences when using the Web-based learning system. More specifically, students who have various backgrounds may prefer to interact with the Web-based learning systems with different ways. Thus, human factors play an important role in the development of the Web-based learning systems. Among various human factors, previous research indicated that prior knowledge greatly affect student learning. For instance, Fethi Calisir and Zafer Gurel [3] investigate the relationships between text structure and prior knowledge of the students. Their research focused on reading comprehension. As showed in their results, students with high prior knowledge demonstrated higher performance than those with low prior knowledge.
The other study by Amadieu, Tricot, Mariné [4] also tended to discover the relationships between students’ prior knowledge and learning performance. The results showed that students with high prior knowledge got good performance and followed coherent reading sequences in the network structure. In contrast, students with low prior knowledge obtained low performance, but they could demonstrate good free recall performance and experience few disorientation problems in the hierarchical structure.
In addition to learning performance, several studies also tried to investigate the relationships between students’ prior knowledge and their search behaviors. Rouet investigated the impacts of prior knowledge on students’ search strategies [5]. The students were asked to search for answers of some specific questions within hierarchical hypertext. The results showed that students’ search strategies depend on their prior knowledge within the subject domain and question characteristics. The
students with high prior knowledge demonstrated fast and precise searches when they used a hierarchical hypertext to answer specific questions. Conversely, students with low prior knowledge needed not only to spend much time searching answers for general questions but also frequently to look into them again.
3. Problem Definition
The results of the studies presented in Section 2 suggest that prior knowledge influences users’ learning performance and the search behavior. They imply that students’ learning effectiveness may be negatively affected if learning environments do not match with the levels of their prior knowledge. In particular, there is a need to provide proper learning environments for students with low prior knowledge because their performance is not as good as those with high prior knowledge. To this end, this study addresses this issue by investigating the behavior of students with low prior knowledge in environments that match and mismatch with the needs of students with low prior knowledge.
More specifically, this study addresses a research question, how students with low prior knowledge react to a matching or mismatching environment. Hence, two versions of keyword search tools had been given to students. Subsequently, how students with low prior knowledge interact with these two tools had investigated. The section describes the experiment design of this study, including participants, research instruments, experimental procedures and data analyses.
4. Experiments
4.1. Participants
A total of 44 students, which are undergraduate and graduate students at some university in Taiwan, joined our experiment voluntarily. All participants have the basic computer and Internet skills necessary to use Web-based learning system we established. The content of the Web-based learning systems is “Interaction Design” in this experiment, and these 44 students never study the course of Interaction Design. Hence, they can be considered as students with low prior knowledge.
4.2. Research Instruments
The research instruments used include: (1) Web-based learning system to provide all participants to operate, (2) Task Sheet to allow all participants to finish it by using the Web-based learning system, (3) Post-test to evaluate what knowledge about Interaction Design these participants had known after finishing the task sheet. The
details of these instruments are described in following sections.
4.2.1. Web-based Learning System. As showed in a comprehensive review by Chen and Macredie [6], students with low prior knowledge are less comfortable to use technology-based learning tools than students with high prior knowledge. Moreover, the former require more additional support than the latter. Thus, it is necessary to use a simple design and more visual cues to support the former whereas a sophisticated design and fewer visual cues can be applied to take care of the latter [7]. Consequently, the Web-based learning system used in this study includes two versions, which provide both of the keyword search and hierarchical map. However, their design approaches are different. The differences between Version A and Version B are detailed in Table 1. More specifically, Version A provides a simple design and includes some visual cues while Version B offers complex functions and contains fewer visual cues. In other words, Version A can match with the preferences of students with low prior knowledge and females while Version B can match with the preferences of students with high prior knowledge and males. Figures 1 to 6 illustrate the design of these two versions.
Table 1. The differences between Versions A & B
Version A
Version B
Keyword
Search Single Search Box Advanced Search Tool with Boolean Operators Result Presentation Keywords Highlighted Keywords Un-highlighted Hierarchical Map Main Categories& Sub Categories Main Categories& Sub-Categories & Other Related Categories
Figure 2. Keyword search (Version B)
Figure 3. Result presentation (Version A)
Figure 4. Result presentation (Version B)
Figure 5. Hierarchical map (Version A)
Figure 6. Hierarchical map (Version B)
4.2.2. Task Sheet. The purpose of this study is to identify whether students will be affected by the matching or mismatching learning environment when they want to learn by operating the Web-based learning system. By doing so, tasks can be used to guild students to use the keyword search to find information from the Web-based learning system. Previous research also indicates that participants’ motivation can be maintained by tasks.
In this study, there are two different kinds of tasks. One is a factual question, and the factual question’s answer is a standard answer. Its design is more matched with a basic keyword search tool, because each factual question only focuses on one session’s concept. In other words, students can solve the factual question by using a simple query. On the other hand, another is an essay question, which is designed with more complex concepts. Students have to analyze the logical relationships of the keywords appeared in the essay questions. Hence, the advanced keyword search tool is more suitable for dealing with the essay questions and students can use Boolean logical operators to combine two different keywords.
During the experiment, these students needed to interact with the Web-based learning system and found the answers of each question. The aim of task sheet design is offered the opportunities of using keyword search tool, and collected data of these students’ search behavior.
4.2.3. Post-test. The post-test was designed to evaluate these students’ degrees of knowledge about “Interaction Design” after learning form the Web-based learning system. It also investigated students’ learning performance after using the Web-based learning system by finishing the task sheet. It included 20 multiple-choice questions about the subject knowledge, and each questions with three different answers and an “I don’t know” option. The post-test score is regarded as student’s learning performance and whether the learning environment can help student or not.
4.3. Experimental Procedures
In this experiment, they had been divided into two groups. One group uses the basic keyword search tool, and it is a learning environment that matched with the needs of students with low prior knowledge. On the other hand, the other group uses the advanced keyword search tool which is a learning environment that mismatched with the needs of students with low prior knowledge. Furthermore, each group of students also had been divided into two small groups. The two small groups required to complete two types of tasks assigned to them, i.e., factual question tasks and essay question tasks, respectively.
All students interacted with the Web-based learning system, and they needed to finish the assigned task sheet. Simultaneously, students’ learning behaviors had been recorded in a database. These data include which keywords they typed, how to use Boolean logical operators, how to select search result, what their extensive reading behaviors, how many pages they visit and how much time they spend for searching keywords or reading each searched page. These data had been used to data analyses.
After all students had finished the task sheet, they required to go into the final step: the post-test. They need to take the post-test to evaluate how much they had learned from the Web-based learning system, and it is regard as their learning performance.
5. Methods
In this experiment, both students’ searching behaviors and learning performance were regarded as the input data to data analysis. These data stored in the database had been analyzed with the K-means algorithm. The reason for using K-means is that it was widely used to analyze students’ on-line learning patterns. For example, the study by Chen and Liu show that K-means is an effective tool to analyze on-line learning patterns [8].
In order to utilizing the K-means algorithm, these data were necessary to extract appropriate attributes. Therefore, we determined these attributes based on students’ searching behaviors and learning performance in the matching or mismatching learning environment. In other words, this study emphasizes on the characteristics of learning environments, instead of students’ cognitive style, prior knowledge or other human factors.
There are seven attributes included: (1) the total time spent for completing tasks, (2) the total number of keyword searching, (3) the total number of visited movements, (4) the total number of visited repeats, (5) the total number of visited pages, (6) the number of pages in each keyword searching, which regarded as whether students concentrated on the Web-based learning system or not, and (7) the number of pages in the total time, which is regarded as students’ efficiency on the Web-based learning system.
Subsequently, these data had been normalized firstly before utilizing the K-means algorithm because these attributes are unbalanced. More specifically, the attributes, which have range scales dramatically different than other attributes, may determine the major trend of the results of the K-means algorithm. In other words, it may raise a bias without normalizing the attributes. In particular, to reach our aims, we respectively utilized the K-means algorithm to investigate students’ searching behaviors in two different environments (i.e. matching and mismatching learning environment). By doing so, we can further investigate students’ searching behaviors in different environments.
5.1 Search Behavior
In this section we would analyze results related to the aims of this study. More specifically, we tend to investigate students’ search behavior based on evaluating both their efficiency and attention. To this end, two major features, i.e., how many pages per searching behavior
which each student visit and how many keywords they used when interacting with the Web-based learning system, are used as input data to the K-mean algorithm.
It may be due to the fact that aforementioned features are too unbalanced to influence the results of the K-mean algorithm so we need to preprocess such features. For instance, the total time spend interacting with the Web-based learning system are dramatically bigger than other features. Thus, for addressing such issue, these features should be normalized to a scale ranging from zero to one. Moreover, there are two parts of input data set used to the K-mean algorithm, such as matching and mismatching groups. Each of these groups is divided from the input patterns according to their interface type.
After performing the k-means algorithm, four clusters are partitioned in each group. For further investigating each group, we can find that both groups have a cluster that can be treated as outliers. In other words, there are three clusters used to investigate our aims in each group. The clustering results in each group are show in Figure 7 and Figure 8. c1 c2 c3 0 0.1 0.2 0.3 0.4 0.5 0.6 total time keywor d movem ent repeat page page / keywor dpage / total time Figure 7. Clustering results (Mismatching condition)
c1 c2 c3 0 0.1 0.2 0.3 0.4 0.5 0.6 total time keyword moveme nt repeat page
1 2 3 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 page / keywor d page / total time Figure 9. Concentrated reading strategy (Mismatching
condition) 1 2 3 0 0.1 0.2 0.3 0.4 0.5 0.6 page / keywor d
Figure 10. Efficient reading strategy (Matching condition) After carefully examining Figures 7 and 8, we found that the number of pages read based on per keyword search (page/keyword) and the number of pages read per second (page/total time) demonstrate interesting trends. Thus, further analyses are conducted. Figures 9 and 10 illustrate the difference between the number of pages read based on per keyword search and the number of pages read per second. In general, students’ search behavior can be divided into two kind of pattern, including a concentrated reading strategy by using a mismatching interface and an efficient reading strategy by using a matching interface.
Regarding the former, we found that page/keyword is higher than page/total time in the mismatching interface. It implies that students in the mismatching interface concentrate on reading pages obtained from each keyword search, instead of increasing the number of pages read per second. In other words, they pay more attention to reading the pages in the mismatching interface than in the matching interface. A possible reason is that the mismatching interface may provide too many options to such users, who lack enough prior knowledge. Thus, they may lose confidence, which, in turn, they need to check whether the content meets their needs by reading many pages when they use the mismatching interface. It may take a lot of time for them to read so many pages. At the same time, they, however, are give opportunities to
absorb information from concentrating on reading such pages.
Regarding the latter, the page/keyword is lower than page/total time in the matching interface. In other words, students in the matching interface draw more attention to the number of pages they read per second than those in the mismatching interface. In other words, students in the matching interface are more concerned with efficiency than those in the mismatching interface. This is probably because the matching interface provides a simple keyword search, with which they may not be able to get exactly relevant results but this interface is familiar to them. The mismatching interface offer advanced Boolean operators, which can help them obtain exactly relevant results but this interface is unfamiliar to them. These results suggest that there is a need to provide a simple interface to increase the efficiency of students with low prior knowledge. However, this approach may stop them to concentrate on reading pages to absorb new knowledge.
5.2 Learning Performance
In addition to reading patterns, we also investigated how students’ performance is influenced by using mismatching and matching environment. To address such an issue, the students’ post-test scores were used to evaluate their performance. As shown in Figure 11, we can find that the post-test score in the matching interface is higher than those in the mismatching interface. In other words, students using the matching interface may demonstrate high performance while students using the mismatching interface may obtain low performance. It may be due to the fact that students may feel more comfortable to use functions provided by the matching interface so they can pay more attention to locate the information they want. More specifically, they may feel friendly when using the matching interface so they can easily pay more attention to using such interface to locate the information. In other words, the simple interface is suitable for students with low prior knowledge to help them study an unfamiliar area. Therefore, they may obtain high performance. The aforementioned results echoes the results of reading behavior described in Section 5.1, which indicated that students may perform efficiency when using the matching interface. These results imply that such an interface may be suitable for them so they can easily identify relevant results.
Mismatching Matching 8 8.5 9 9.5 10 10.5
Figure 11. Post-test score
Concentration may give an opportunity to let users have more chance to obtain other information which is not covered by the task. However, according to our results, such chance was not helpful for improving their performance. This is probably because they may obtain too much knowledge from the system. Therefore, students in the mismatching interface may feel difficult to select relevant issues from massive information, which, in turn, they need to spend a lot of time making sure whether the located information is right or not. In the end, they may lose confidence during their processes. Thus, we can conclude that the complex interface is not suitable for students with low prior knowledge to locate the information they want so their performance may not be improved.
The aforementioned situation may lie within the fact that the users, who browse the Web-based learning system in the mismatching interface, cannot utilize their existing knowledge to identify which searching result may meet their needs. In other words, such situation may cause their cognitive overload. More specifically, there is too much information to them to identify which content is suitable to them. In this way, they may obtain low performance. Therefore, reading in concentration may be not an appropriate reading strategy for students with low prior knowledge. In brief, mismatching environment not only have negative effects on learning performance, but also let students waste much time learning in an unsuitable environment.
6. Conclusions
This study uses two versions of web-based learning system to examine the relationships between matching /mismatching condition and students’ learning reading strategies and learning performance. In general, the results we found demonstrate that students in the matching condition tend to use the efficient reading strategies while those students in the mismatching condition tend to use the concentrated reading strategies. In summary, the matching/mismatching have great effects on students’ reading strategies and learning performance. The designers should consider the characteristics of each
student and then develop a learning tool that can match with their needs. By doing so, their learning performance and satisfaction can be promoted.
The present study shows fruitful results but there are several limitations. Firstly, the present study only incorporates a small-scale sample and limited navigation tools. Hence, it is recommended that further studies should be undertaken with a larger sample and the Web-based learning system should provide other navigation tools, such as alphabetical index, so that additional evidence can be obtained. Such evidence can not only be helpful to improve the design of Web-based learning systems, but also is useful for the development of personalized learning tools, such as electronic journals or digital libraries.
References
[1] N. Ford and S. Y. Chen, “Matching/mismatching revisited: An empirical study of learning and teaching styles,” British J. Educ. Techn, vol.32, pp.5–22, 2001. [2] J.A. Greene, L. Costa , J. Robertson, Y. Pan, V. M.
Deekens, “Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment,” Computers & Education, vol.55, pp. 1027-1043, 2010. [3] Calisir, F., Gurel, Z., “Influence of text structure and prior
knowledge of the learner on reading comprehension, browsing and perceived control,” Computers in Human Behavior, Vol. 19, No. 2, pp. 135-145, 2003.
[4] F. Amadieu, A. Tricot, and C. Mariné, “Hypertexts favor comprehension and learning for experts? The effects of prior knowledge diversity,” In Paper presented at ICLEPS conference, 2005.
[5] Potelle, H., & Rouet, J. F., “Effects of content representation and readers_ prior knowledge on the comprehension of hypertext.” International Journal of Human–Computer Studies, 58(3), 327–345, 2003. [6] S. Y. Chen and R. Macredie, “Web-based interaction: A
review of three important human factors,” International Journal of Information Management, vol. 35, pp. 379-287, 2010.
[7] S. Y. Chen, J. Fan, and R. D. Macredie, “Navigation in Hypermedia Learning Systems: Experts vs. Novices,” Computers in Human Behavior, vol. 22, pp. 251-266 ,2006.
[8] S. Y. Chen and X. Liu, “An Integrated Approach for Modeling Learning Patterns of Students in Web-based instruction: A Cognitive Style Perspective,” ACM Transactions on Computer-Human Interaction, vol.15, Article 1, 2008.
We would like to thank for the support from Professor Sherry Chen (陳攸華) for the proposed project.
