Running Head: Using LMS for Course Projects
Using Learning Management Systems for Course Projects:
A Case Study of Student Collaboration
Steven Lonn
University of Michigan
Please address all correspondence to:
Steven Lonn
University of Michigan
Digital Media Commons & School of Education 2281 Bonisteel Blvd., Ann Arbor, MI 48109-2094 Email: [email protected]
Paper presented at the
Annual Meeting of the American Educational Research Association San Diego, California, April 14, 2009
Using Learning Management Systems for Course Projects:
A Case Study of Student Collaboration
Steven Lonn
University of Michigan
Digital Media Commons and School of Education
Abstract: This case study examined how undergraduate students used project site capability
available through the campus Learning Management System (LMS) to work together to produce a required group project. Students' survey responses and aggregated event log data were analyzed to see if students’ project site use was consistent with their beliefs about project site activities and tools. Overall, students regarded the activities and related tools within the LMS very positively and thought that their assignments were more successful because of their use of LMS for their group project. Students rated activities related to the management of materials and communication within their group higher than interactive online activities such as answering questions from group members.
Students attending higher education institutions today are presented with a wide array of technological tools from email and personal web pages to clickers and iPods. Learning
Management Systems (LMS) allow instructors and students to use a variety of web-based technological tools in order to share materials, make announcements, and interact with each other online within one online space. While the typical mode of LMS is focused on instructor-directed course sites, some institutions allow students to create LMS sites of their own without instructor mediation. These sites, known as project sites, extend the online tool set available in course sites to allow student-generated, student-subscribed, student-customized sites with permissions set for more openness and collaboration among the participants.
There have been a substantial number of studies on online synchronous and asynchronous communication tools (see De Wever, Schellens, Valcke, & Van Keer, 2006). However, there has been very little research on how students use the collection of online tools found within LMS to arrive at common goals (e.g. a group presentation, paper, or other project) with their peers. This case study therefore investigated how undergraduate students in an upper-level biology course used project sites in order to complete a group project: to produce a mock National Institutes of Health (NIH) grant proposal. Students' quantitative and qualitative answers to a post-project survey were explored and aggregated event log data from the LMS was examined to see if students’ system use was consistent with their beliefs about various activities and tools as demonstrated by the survey results. Overall, this case study illuminates what students thought about the project sites, how they used the tools provided in these sites, and if they found the technology useful for collaboration.
Background
This study is specifically interested in ways that students collaborate with one another through a suite of web-based tools available in a LMS. In this case study, collaboration refers to the "social context that the group process facilitates" (Stacey, 2005, p. 151). Within this social
conception of the central problem (Rochelle & Teasley, 1995). Collaboration is most effective when student groups have a common goal, incentives to collaborate, and students are
independent from the instructor (Hathorn & Ingram, 2002), as is the case with the mock NIH grant proposal students were required to complete in this study.
Other computer-supported collaborative learning environments have effectively supported student collaboration. The Learning Through Collaborative Visualization (CoVis) environment allows students to collaboratively investigate atmospheric and environmental science through inquiry-based activities (Pea, Edelson, & Gomez, 1994). Students use a variety of asynchronous communication tools, such as the Collaboratory Notebook, to record their activities, observations, and hypotheses, scaffolding their scientific inquiry (Edelson, Gordin, & Pea, 1999). The Global Learning and Observations to Benefit the Environment (GLOBE) environment allows students to take scientific measurements of their area's airborne particulate counts and cloud cover that are used by scientists and students around the world for
environmental research (Finarelli, 1998). A variety of communication tools and shared resources allow students to collaboration with scientists while engaging in authentic science research. Finally, the Computer-Supported Intentional Learning Environment (CSILE) (or the second generation known as Knowledge Forum) allows students to share their writing with classmates and other students around the world (Scardamalia, 1989; Scardamalia & Bereiter, 1994). Students are encouraged to contribute their writing and then discuss, refine, and revise that writing collaboratively. While these and other similar environments have successfully supported student collaboration in relatively controlled, discipline-specific domains, LMS is designed to apply generally to all learners in all domains. This study is therefore interested in how this more generic set of tools facilitates collaboration, as indicated by students' opinions and supporting event log data.
This study also builds upon a growing literature investigating LMS use by instructors and/or students (e.g., Ansorge & Bendus, 2003; Harrington, Gordon, & Schibik, 2004; Herse & Lee, 2005; Holm, Röllinghoff, & Ninck, 2003; Morgan, 2003; Yohon, Zimmerman, & Keeler, 2004). Based on their survey of over two hundred instructors, three researchers identified six typologies of LMS use (Dutton, Cheong, & Park, 2003). Of these typologies, the eTeam description, virtual student groups for studying or projects, most closely resembles the type of LMS use examined in this case study, in which a group of students used project sites to interact with each other around a course-related project. By examining students' survey answers and LMS event logs, this study hopes to discover what students thought about the project sites, how they used the tools provided in these sites, and if they found the technology useful for
collaboration.
Methods
Participants
This case study focused on an upper-level biology course at a large American
Midwestern research university in which undergraduate students were required to complete a term project, a mock NIH grant proposal, a complex project involving several written sections and research on a biology-related topic. Students worked independently or in self-selected
groups of 2-6 students. Overall, there were 32 groups formed and 8 students decided to work independently, twenty-one of the groups (83 students) decided to use the LMS to create their own project site, representing 66% of all the students in the biology course. The group size averaged about 4 students per group. There were 43 students who participated in the online survey, representing 17 different groups who used project sites.
Data Sources
The online survey examined in this study was administered immediately after students turned in their final grant proposals. Students were invited to participate via email. Aggregated data on system use was generated from the event logs for all 21 project sites. Event logs capture when a user takes a particular action with a specific tool, such as downloading a document or posting an announcement.
Design & Procedure
The LMS examined in this study is based on the Sakai community-source architecture (http://sakaiproject.org). This environment is comparable to other popular systems such as
Blackboard (http://blackboard.com) and Moodle (http://moodle.org). Within project sites, student owners may elect to use any of the following tools as well as decide how to use them:
• Announcements: Non-threaded, asynchronous messages for all site participants to read. Email notifications may be sent as well.
• Chat: Synchronous conversation tool. All messages are automatically saved and are viewable for all site participants.
• Content Sharing: File storage space. Any file type may be stored; URLs to other websites may be created, and shared citation lists may be created. Email notifications of new content may be sent as well.
• Discussion: Threaded, asynchronous messages for all site participants to read. • Email Archive: All email sent via a site-specific email address is delivered to all
participants and also saved online for archival and searching.
• News: Allows site participants to view RSS feeds from external sources. • Schedule: A shared calendar used to post deadlines, due dates, etc. • Web Content: Allows site participants to view external websites.
• Wiki: A collaborative document-writing tool. Any site participant may add or modify additional pages and a history of changes is automatically recorded.
The analysis of the survey data investigated the usefulness of various activities within the project sites and the tools that are used to support those activities. These results were then
compared to patterns of use shown in the system log data that reported aggregated user actions in the project sites. User actions related to logging in and out of the project sites were removed for this analysis. Additional survey responses about benefits, overall attitudes, non-project site technologies, and face-to-face meetings were also explored.
Results
How Often Students Visited Project Sites
In order to determine if students' opinions about project sites were based on frequent or infrequent use, students were asked how often they visited their project site for their grant
proposal project (see Table 1). Most students (74%) replied that they visited their project sites at least a few times every week or daily.
Table 1: How Often Students Visited Their Project Sites
Few times a semester
Few times a
month Once a week
Few times a week Daily 5% (n=2) 9% (n=4) 12% (n=5) 51% (n=22) 23% (n=10)
Usefulness of Activities on Project Sites
Because Project sites can be used to accomplish a wide variety of activities, students were asked to rate the usefulness of various activities using a 5-point Likert scale from 1="Not useful at all" to 5="Very useful" (see Table 2). The highest rated activities focused on sharing documents and messages between group members, while the lowest rated activities focused on providing feedback to other group members.
Table 2: Student Ratings of Usefulness of Activities Completed on Their Project Site
Activity n Mean SD
Posting documents or other materials online 43 4.86 .560 Accessing documents or other materials online that
other group members have posted 43 4.84 .485
Receiving messages / announcements / notifications
from other group members 13 4.29 1.183
Sending messages / announcements / notifications
to other group members 11 4.20 1.114
Linking to a website outside of the LMS environment 31 3.71 1.101
Posting questions to other group members 18 3.61 1.197 Collaboratively writing a document with group
members online 30 3.50 1.167
Receiving comments on my work from other group
members 26 3.50 1.068
Answering questions from other group members 33 3.45 1.201
Commenting on other group members' work 30 3.20 1.215
Usefulness of LMS Tools and Corresponding Event Log Data
The activities described above are accomplished using several LMS tools. Therefore, the students were asked to rate the usefulness of the LMS tools for collaboration on a 5-point Likert scale from 1="Not useful at all" to 5="Very useful." To see if actual system use was consistent with the attitudes expressed in the survey data, the LMS event logs for the 21 project sites, aggregated by tool, were also examined. Table 3 provides the data from the students’ ratings and event logs. The highest rated tools were Resources and Announcements and the lowest rated tools were Schedule and Wiki. According to the event logs, the Content Sharing and Chat tools were used the most and the Schedule and Wiki tools were used the least.
Table 3: Student Ratings of Usefulness of LMS Tools on Their Project Site for Collaboration and Corresponding Aggregated LMS User Action Log Reports by Tool
Survey Results Aggregated Log Reports Tool
n Mean SD Mean % of all user actions per site
Content Sharing 42 4.98 .154 87.96%
Announcements 39 4.26 1.117 2.60%
Email Archive 22 3.27 1.241 0.74%
Chat 29 3.21 1.114 5.02%
Web Content
(URL Links) 20 3.20 1.361 not logged
Discussion 22 3.09 1.306 0.46%
Schedule 19 2.68 1.204 0.19%
Wiki 15 2.40 1.404 0.13%
In addition to asking students to rate the value of each tool, we also provided them with a text box to explain which tools they thought were the most useful for collaboration. Qualitative coding of these answers (40 total comments) suggested the basis for the tool ratings. The highest rated tool, Content Sharing (4.98), was identified by 39 students. For example, one student wrote, "The content sharing tool … allowed the group to pool useful articles found by individual members, and allowed work completed to be accessed by all members for review." The second highest rated tool, Announcements (4.26), was also mentioned by 6 students as useful for collaboration because, "Announcements allowed us to keep track of who had done what at what time and to assess overall group progress."
Benefits of Using Project Sites
In order to gauge the perceived benefits of using project sites, students were asked to rate several potential benefits on a 5-point Likert scale from 1="Strongly Disagree" to 5="Strongly Agree" (see Table 4). The highest rated benefits were convenience and efficiency, while the lowest rated benefit was improving learning.
Table 4: Student Ratings of Value of Benefits of Using Project Sites
"Using project sites is valuable for…" n Mean SD Accessing material any time, from any location
(convenience) 41 4.61 .703
Saving me time (efficiency) 41 4.34 .693
Managing my project-related activities 40 4.20 .723
Improving my communication with other project site
participants 41 3.80 .872
Improving other project site participants'
communication with me 41 3.78 .909
Improving my learning 41 3.41 .836
After rating each individual benefit, students were prompted to select one as the most valuable benefit of using project sites (see Table 5). Over three-quarters (76%) of students chose the benefit related to the convenience of accessing material at any time and from any location. Table 5: Student Responses About Most Valuable Benefit of Using Project Sites
Accessing material any time, from any
location (convenience) Saving me time (efficiency) Managing my project-related activities Improving my communication with other project site participants Improving other project site participants' communication with me Improving my learning 76% (n=31) 7% (n=3) 10% (n=4) 7% (n=3) 0% (n=0) 0% (n=0)
Overall Attitudes About Project Sites
To measure students' overall attitudes about project sites, they were asked four questions about their own use of project sites and the relationship of this use to their grant proposal group (see Table 6). Students rated these statements on a 5-point Likert scale from 1=Strongly Disagree to 5=Strongly Agree. The highest rated statement was about the project site helping
communication with group members. The lowest rated statement was about the grant project receiving a higher grade because of project site use. The low rating for project site use affecting grades is consistent with a preliminary analysis that found no correlation (r(49)=-.140, p=.327) between grant proposal grades (M=93.96, SD=6.200, N=51) and project site use (M=.80, SD=.404, N=55).
Table 6: Student Responses About Their Project Sites' Relationship to Their Grant Proposal Groups
Statement n Mean SD
Using a project site helped me communicate with my
other group members 43 4.19 .699
My NIH grant proposal assignment was more
successful because my group used a project site 43 4.00 .845 Using a project site helped me learn from my other
group members 43 3.60 .760
My NIH grant proposal assignment received a higher
grade because my group used a project site 42 3.45 1.017
Students were also asked about the likelihood of using and recommending project sites in the future (see Table 7). Students rated these statements on a 5-point Likert scale from 1=Very Unlikely to 5=Very Likely. The highest rated statement was about using project sites for future course-related projects, and the lowest rated statement was about using project sites for non course-related activities.
Table 7: Student Responses About Their Likelihood of Using and Recommending Project Sites in the Future
Statement n Mean SD
I will use project sites for future course-related
projects / assignments 41 3.98 1.235
I will recommend project sites to other students I
know 41 3.95 .947
I will use project sites in the future for non
course-related activities 41 2.93 1.385
Non-Project Site Collaborative Technologies and Face-to-Face Meetings
In addition to asking students about their use of project sites, students were also asked to identify any non-project site technologies they used to collaborate with their group members (see Table 8). The most popular technologies were email and phone, which 86% and 84% of students reported using, respectively. We again asked an open-ended question and provided a text box for students to explain their use of these technologies (35 total comments). Twenty-nine students commented on email and/or telephone. For example, one student wrote, "E-mail is easy to use because we all check it everyday. Phone is easy because it's instantaneous conversation and facilitates discussion better."
Table 8: Percent of Students Reporting Their Use of Non-Project Site Collaborative Technologies
Technology Percent of Students Reporting Use (n=43)
Email 86%
Telephone / Cell Phone 84%
Instant messaging (e.g., AIM) 23%
SMS text messaging (e.g, texting) 23%
Google documents 5%
Peer connection websites
(e.g., Facebook) 2%
Students were also asked to report how often they met face-to-face (see Table 9). Over three quarters (81%) of students replied that they met face-to-face at least once every week suggesting that the project site use was not a substitute for meeting.
Table 9: How Often Students Reported Meeting Face-to-Face
Never Few times a month Once a week Few times a week 2% (n=1) 16% (n=7) 51% (n=22) 30% (n=13)
In addition to reporting how often they met face-to-face as a group, students were asked to explain what they discussed at those meetings and why they conducted those meetings face-to-face instead of online within the project site. Most of the students (n=47) replied that they
divided work or went over certain portions of the grant proposal during their face-to-face meetings and they did so because it was easier than meeting online. For example, one student wrote, "(The) speed of discussion and level of comprehension are both improved in person. In addition, face-to-face contact allows our group members to know that we are all focused on the task at hand."
Discussion
Overall, students responded positively when asked to rate the value of specific activities and related tools provided by the project site capability in the campus LMS. The aggregated log data confirmed that students used several different tools within their project sites, and the most used tools also generated high ratings of usefulness on the survey. Students thought that their NIH grant proposal assignments were more successful because of their use of project sites, and most students were likely to use project sites again for future course-related assignments.
The survey results suggest that students thought activities related to the management of materials and communication within their group were more useful than more interactive online
Content Sharing, a tool that helps users manage files, and Announcements, a tool that allows users to send messages to each other, were rated much higher than other tools. Students also rated project site benefits related to materials access and management higher than
communication and learning benefits. These findings replicate similar results from previous LMS studies for both instructors (Hanson & Robson, 2004; Holm et al., 2003; Yohon et al., 2004) and students (Herse & Lee, 2005; Maslowski, Visscher, & Collis, 2000; Parker, Bianchi, & Cheah, 2008).
Limitations
One explanation for these results is that the students in this case study attended a residential campus, and this may have had easier access to each other for face-to-face meetings and most of the collaboration between students likely happened at that time. Students generally responded that meeting face-to-face was simply easier than using the various LMS tools (such as Chat or Discussion). There is some evidence in the logs that some groups did use the more interactive tools, but students in those groups still rated the Content Sharing and Announcements tools as more useful in the online survey.
Future Research
The results from this case study indicate that students, overall, find project sites very valuable for their course projects. Particularly when group assignments are complex and involve research literature gathering, group decision-making, and collaborative writing and editing of documents, the tools within LMS project sites can be particularly useful. The more interactive tools within these systems may also be useful for students who might not be able to easily meet face-to-face as the students in this case study. But while these tools are well received, the students did note that there could be substantial improvements to better facilitate group work online. Since LMS were initially designed to facilitate instructor-to-student distribution of materials and announcements in course settings, the tools in project-oriented LMS sites may need to be reconfigured and/or redesigned in order to maximize the potential for collaboration between group members. Further study of LMS use will help to make design recommendations that may lead to better facilitation of group collaboration online using these multifaceted systems.
Acknowledgements
The author would like to thank Stephanie D. Teasley for her helpful review and commentary on the proposal version of this paper. The author would also like to thank Chris Quintana, Eric Dey, and Addison Stone, members of the dissertation committee of which the survey described in this paper is a part of, for their overall guidance and support during the dissertation proposal,
implementation, and analysis process.
References
Ansorge, C. J., & Bendus, O. (2003). The pedagogical impact of course management systems on faculty, students, and institution. In R. H. Bruning, C. A. Horn & L. M. PytlikZillig
(Eds.), Web-based learning: What do we know? where do we go? (pp. 169-190). Greenwich, CT: Information Age Publishing.
De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers &
Education, 46 (1), 6-28.
Dillenbourg, P. (1999). Introduction: What do you mean by 'collaborative learning'? In P.
Dillenbourg (Ed.), Collaborative learning: Cognitive and computational approaches (pp. 1-19). Oxford, UK: Elsevier Science Ltd.
Dutton, W. H., Cheong, P. H., & Park, N. (2003). The social shaping of a virtual learning environment: The case of a university-wide course management system. The Electronic
Journal of e-Learning, 2(1).
Edelson, D., Gordin, D. N., & Pea, R. D. (1999). Addressing the challenges of inquiry-based learning through technology and curriculum design. The Journal of the Learning
Sciences, 8(3/4), 391-450.
Finarelli, M. G. (1998). GLOBE: A worldwide environmental science and education partnership.
Journal of Science Education and Technology, 7(1), 77-84.
Hanson, P., & Robson, R. (2004). Evaluating course management technology: A pilot study. Boulder, CO: EDUCAUSE Center for Applied Research, Research Bulletin, Issue 24. Harrington, C. F., Gordon, S. A., & Schibik, T. J. (2004). Course management system utilization
and implications for practice: A national survey of department chairpersons. Online
Journal of Distance Learning Administration, 7(4)
http://www.westga.edu/~distance/ojdla/winter74/harrington74.htm
Hathorn, L. G., & Ingram, A. L. (2002). Online collaboration: Making it work. Educational
Technology, 42(1), 33-40.
Hawkins, B. L., & Rudy, J. A. (2007). Educause core data service: Fiscal year 2006 summary
report. Boulder, CO: Educause.
Herse, P., & Lee, A. (2005). Optometry and WebCT: A student survey of the value of web-based learning environments in optometric education. Clinical and Experimental Optometry,
88(1), 46-52.
Holm, C., Röllinghoff, A., & Ninck, A. (2003). WebCT and elearning in Switzerland.
Proceedings of the 5th International Conference on New Educational Environments,
Luzerne. 139-143.
Maslowski, R., Visscher, A. J., & Collis, B. (2000). The formative evaluation of a web-based course-management system within a university setting. Educational Technology, 40(3), 5-19.
Morgan, G. (2003). Faculty use of course management systems (Vol. 2. Boulder, CO: Educause Center for Applied Research.
Parker, R. E., Bianchi, A. & Cheah, T. Y. (2008). Perceptions of instructional technology: Factors of influence and anticipated consequences. Educational Technology & Society,
11(2), 274-293.
Pea, R. D., Edelson, D., & Gomez, L. M. (1994). Distributed collaborative science learning using scientific visualization and wideband telecommunications. 160th Meeting of the
American Association for the Advancement of Science,
Scardamalia, M. (1989). Computer-supported intentional learning environments. Journal of
Educational Computing Research, 5(1), 51-68.
Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities.
The Journal of the Learning Sciences, 3(3), 265-283.
Stacey, E. (2005). A constructivist framework for online collaborative learning: Adult learning and collaborative learning theory. In T. S. Roberts (Ed.), Computer-supported
collaborative learning in higher education (pp. 140-161). Hershey, PA: Idea Group
Publishing.
Yohon, T., Zimmerman, D., & Keeler, L. (2004). An exploratory study of adoption of course management software and accompanying instructional changes by faculty in the liberal arts and sciences. Electronic Journal of e-Learning, 2(2), 313-320.
