The Benefit of Information Technology in Managing Outcomes
Focused Curriculum Development Across Related Degree Programs
Brian R. von Konsky, Allan Loh, Mike Robey, Susan J. Gribble, Jim Ivins, David Cooper
Curtin University of Technology Department of Computing
GPO Box U1987, Perth 6845, Western Australia [email protected]
Abstract
This paper describes an interactive, web-based tool that provides access to an Outcomes Database for managing an Outcomes Focused curriculum plan. The approach provides a rapid way to involve subject specialists, course controllers, curriculum designers and educational consultants in the curriculum design and review process. The Outcomes Database is used to capture the relationships between units (i.e. subjects), Unit Learning Outcomes, Course Learning Outcomes, Generic Graduate Attributes and assessments. This is undertaken across courses (i.e. undergraduate degree programs) sharing common units in Information Technology, Computer Science, and Software Engineering. Automatically generated database reports that show how units contribute to the development of the generic graduate attributes are compared to those derived manually on a unit-by-unit basis. Preliminary experience with this IT-based approach demonstrates that automatically generated database reports effectively highlight areas requiring further attention during subsequent curriculum development iterations. The tools and approach described here show how the Outcomes Database can assist in making the transition from an existing program to one using an Outcomes Focused approach.
Keywords: Database, Outcomes Focused Education
1
Introduction
In an Outcomes Focused approach to teaching and learning, educators place emphasis on measuring the attainment of student learning outcomes, taking a holistic approach to the design of an integrated curriculum. Effectively, this focuses attention on what students learn as a result of their participation in a program of study, rather than on the progression of topics taught in individual subjects.
Increasingly, such an approach is a requirement for professional accreditation (Bell, 2000; Engineers Australia, 1996).
Copyright © 2006, Australian Computer Society, Inc. This paper appeared at the Eighth Australasian Computing Education Conference (ACE2006), Hobart, Tasmania, Australia, January 2006. Conferences in Research in Practice in Information Technology, Vol. 52. Denise Tolhurst and Samuel Mann Eds. Reproduction for academic, not-for profit purposes permitted provided this text is included.
The transition to the Outcomes Focused approach presents many challenges to curriculum designers, who may be designing new degree programs or adapting existing programs.
These challenges include the following:
• identifying appropriate learning outcomes and effective means to demonstrate their attainment; • managing the definition of learning outcomes shared
by multiple units and across related degree programs;
• demonstrating that students have attained key competencies prior to entering a given profession (e.g. the Software Engineering Body of Knowledge (SWEBOK) for graduates of Software Engineering Programs or Computing Curriculum 2001 in the case of Computer Science and Information Technology programs) (IEEE 2004; IEEE and ACM 2001);
• communicating the criteria used in assessing and reporting performance to students and their parents; • managing limited resources and minimising the
negative impact on students when making the transition from an existing program to one using the Outcomes Focused paradigm; and
• involving relevant stakeholders in the curriculum design and review processes, including curriculum designers, domain experts, university management, educational consultants, internal review and approval committees, lecturers, tutors, and accreditation panels.
Perhaps most significantly, inter-relationships between units in courses that share common outcomes defined at different levels of generalisation make designing and managing an Outcomes Focused approach to education inherently complex.
This paper describes an interactive, on-line Outcomes Database developed to assist curriculum designers in addressing these challenges.
2
Theoretical Framework
The goal of an Outcomes Focused approach to education is to articulate and measure what students are expected to know, be able to do, and be like as a result of their learning. These expectations are expressed in terms of outcomes that take the form of descriptive statements
about the learning to be demonstrated by students. Learning outcome statements, however, may be expressed at different levels of generality from broad and general to more specific and highly detailed.
Irrespective of the level of generality, outcome statements describe the skills, knowledge and values that students should be able to demonstrate at the end of a planned teaching-learning process. The outcome statements do not specify what should be in the curriculum. Instead, they describe the result of successful student participation in the teaching-learning process.
Structuring outcome layers can provide a framework for a comprehensive curriculum design that includes common outcomes and subjects for related degree programs. More importantly, an integrated curriculum design and the means by which outcomes will be measured become possible within the context of such a framework.
The Outcomes Database described in this paper uses four layers of generality:
• Generic Graduate Attributes • Course Learning Outcomes • Unit Learning Outcomes • Assessment Rubrics These are described in detail below.
Generic Graduate Attributes describe broad characteristics that can be demonstrated by all graduates from an institution, or by graduates who will become eligible for membership in the professional society responsible for accrediting related degree programs. Barrie (2004) states that:
"[b]roadly speaking, generic graduate attributes in Australia have come to be accepted as being the skills, knowledge and abilities of university graduates, beyond disciplinary content knowledge, which are applicable to a range of contexts. It is intended that university students acquire these qualities as one of the outcomes of successfully completing any undergraduate degree at university."
For example, Engineers Australia have identified 10 Generic Graduate Attributes (Engineers Australia, 1999). Additionally, all future graduates of professional engineering programs accredited by Engineers Australia must demonstrate attainment of over 80 competency performance indicators to qualify for stage 1 status as a professional engineer (Bradley, 2005). That there are so many performance indicators suggests that Information Technology (IT) infrastructure and tools to manage learning outcomes and track performance indicators will become essential in the accreditation of Australian engineering programs.
Generic Graduate Attribute statements by themselves do not provide a comprehensive framework for developing an integrated curriculum with measurable outcomes that can be tied to these performance indicators.
In our approach, such a framework is facilitated by the Outcomes Database, in which Course Learning Outcomes further elaborate the Generic Graduate Attributes, but in the context of a given degree program. Unit Learning Outcomes are then defined for individual units, and are mapped to appropriate Course Learning Outcomes. The rubrics used to assess the level to which individual Unit Learning Outcomes have been attained are also stored in the database.
Once it can be shown that a Unit Learning Outcome has been attained, by inference the associated Course Learning Outcomes and Generic Graduate Attributes have also been attained.
Adding to the complexity of a layered outcome focused curriculum, related degree programs often share common required units. For example, Software Engineering programs are sometimes regarded as a Computer Science specialisation. In other cases, Software Engineering students share common units with Electrical, Civil, and Mechanical Engineering students (Parnas, 1999).
In these cases, Unit Learning Outcomes must be defined that are compatible with the Course Learning Outcomes from the related degree programs. This can complicate curriculum design when using an Outcomes Focused approach, particularly when degree programs cross the administrative and faculty jurisdictions used to design, review and approve the curriculum.
This suggests that the effective use of IT may help to improve collaboration both within and between university departments. Moreover, practical experience at the University of Queensland has shown that the use of IT for outcomes focused course design can foster pedagogical change and encourage university-wide communication and collaboration amongst curriculum designers (Jolly and Humphries, 2002).
Other software-based tools are available to help educators develop program curriculum, including the content and assessment instruments associated with individual subjects (Botev et al. 2005; Ong and Shepherd, 2002; Marshall 2005). However, these tools are for broader course management purposes and by themselves are not outcomes focused. This paper addresses this need by describing an IT-based approach for managing the complex relationships found in an outcomes focused curriculum plan that uses layered outcome statements. Our approach manages the inherent complexity using a database and convenient on-line tools. It is suitable for planning, reviewing, and implementing multiple degree programs that share common subjects, and facilitating collaboration amongst designers, reviewers, and other stakeholders. Describing this approach is the principal contribution of this paper.
3
Development of the Outcomes Database
The Outcomes Database described in this paper is depicted in Figure 1. It was developed to manage the introduction of Outcomes Focused Education in the Department of Computing at an Australian university offering three-year BSc (Computer Science) andBSc (Information Technology) degrees and a four-year BEng (Software Engineering) degree. These programs share common units and hence common Unit and Course Learning Outcomes. Each program also contains unique units with distinct Unit Learning Outcomes, resulting in Course Outcomes that differentiate each degree.
Additionally, the BEng (Software Engineering) program shares a Common First Year in Engineering with all other engineering specialisations.
In the nomenclature of our institution, degree programs are called courses and individual subjects are called units.
3.1
The Outcomes Database
The Engineers Australia Generic Graduate Attributes developed by a given unit were manually identified during a workshop involving all academic staff from the Department of Computing. Since 2001, this has been manually reviewed on a regular basis. Justification for the claim that a given unit contributes to the development of a Generic Graduate Attribute is included in all unit outlines distributed to students during the first class meeting in any given semester.
The manual mapping between units in Department of Computing courses and the Generic Graduate Attributes was undertaken prior to the conception of the Outcomes Database and the identification of Course Learning Outcomes. The justification for the manual identification of this relationship included teaching pedagogies and methods not directly related to the Course Learning Outcomes that were identified later.
During 2004, the Department Teaching and Learning Committee developed Course Learning Outcomes for each of the three undergraduate courses offered by the Department. These Course Learning Outcomes were entered into a web-accessible MySQL database to foster dialogue amongst committee members, educational consultants, and other stakeholders.
As of February 2005, the Outcomes Database contained Unit Learning Outcomes for all units offered by the Department of Computing. It also captured the relationships between each Unit Learning Outcome and the Course Learning Outcomes for those courses in which the unit is a core requirement, and indirectly to the Generic Graduate Attributes. These relationships are illustrated in the shaded area in Figure 1.
As of February 2005, the Outcomes Database contained 10 Generic Graduate Attributes defined by Engineers Australia, supplemented by 3 Generic Graduate Attributes defined by the Department of Computing. All 13 of these were related in the database to 9 similar Generic Attributes defined by the University. Each of the 20 Course Learning Outcomes for the 3 courses offered by the Department of Computing were mapped to the Generic Graduate Attributes. Of the 20 Course Learning Outcomes, 7 were common to all 3 courses, 2 were common to the BSc(CS) and BSc(IT) programs, 1 was common to the BEng(SE) and BSc(CS) programs, 3 were unique to BSc(CS), 2 were unique to BSc(IT), and 5 were unique to BEng(SE). There were a total of 269 Unit Learning Outcomes mapped to Course Learning Outcomes for 57 units.
The Outcomes Database was motivated by the observation that an efficient and automated mechanism for managing the scope and complexity of a layered outcome focused framework was necessary. This is particularly important because a small change in any one part of the outcome structure can have a ripple effect, impacting many other parts. With the Outcomes Database, any change is automatically propagated throughout the framework, and automatically generated database reports.
Beginning in July 2005, the Outcomes Database is being populated with data capturing the relationship between each Unit Learning Outcome, the criteria used to assess the level of outcome attainment, and the percentage significance of each assessment criterion to the overall mark in the unit.
Experience has shown that each Unit Learning Outcome usually has around 4 assessment criteria. By the time that the database is fully populated with these rubrics, it is estimated that there will be around 1076 individual criterion that must be tracked for each student.
Course Learning Outcomes
Unit Learning Outcome 1 Unit Learning Outcome 2
…
Unit Learning Outcome n Generic Graduate Attributes
Unit Learning Outcome 1- Criterion 1 Unit Learning Outcome 1- Criterion 2
…
Unit Learning Outcome 1 - Criterion n
Criteria Significance (%)
Assessment Item
Figure 1. Database relationships between Generic Graduate Attributes, Course Learning Outcomes,
Figure 2. A portion of the Course Learning Outcome Report for the BSc (Information Technology) program.
Figure 3. A portion the Generic Graduate Attribute Report showing the contribution of units in Semester 1, Year 3 of the BEng (Software Engineering) degree program to the Generic Graduate Attributes.
3.2
Data Entry and Reports
As local domain experts, individual Unit Controllers developed Unit Learning Outcomes for their units and identified the associated relationship to an appropriate Course Learning Outcome. Unit Controllers then entered this information into the database. Data entry was achieved using a secure, web-based Graphical User Interface (GUI) that directly communicates with the Outcomes Database.
When adding or modifying a Unit Learning Outcome, Unit Controllers enter the text of the Unit Learning Outcome, possibly embedding Hypertext Mark-up Language (HTML) tags for formatting purposes. This enables verbs from Bloom's Taxonomy (Anderson et al., 2001) to be emboldened, and allows the use of italics or underlining for emphasis of other keywords when the outcome is presented in a web browser.
The corresponding Course Learning Outcome must also be specified at this time from a list of potential choices automatically presented in the GUI by the Outcomes Database. Potential alternatives are in the set given by the intersection of Course Learning Outcomes for the courses in which the unit is a core requirement. The implication is that Unit Learning Outcomes may only be mapped to Course Learning Outcomes that are common to all courses for which the unit is a required unit.
The course structure for each degree program is also stored in the database. This makes it possible to
automatically generate a report showing those courses for which a unit is a core requirement, and the program year and semester in which it is taught.
The curriculum committee used this report when developing the initial set of Course Learning Outcomes in an attempt to ensure a reasonable set of common outcomes for shared units.
An alternative but equally practical use for this report is for resource planning purposes. That is, the report can be used when planning staffing allocations that will meet the requirements of the various course structures and also facilitate the mid-year intake of students into the various programs.
The Unit Learning Outcomes Report shows the number of outcomes defined for each unit. This enabled reviewers to monitor which units were being modified and potentially ready for inspection, based on the assumption that units generally have 4 to 6 Unit Learning Outcomes. It also made it possible to identify those Unit Controllers who were struggling to enter data and might require assistance, or who were not complying with requests for data entry. Generally speaking, however, Unit Controllers were quick to adopt the system once asked to do so. The Course Learning Outcome report automatically shows the outcomes that are common to each degree program. It also shows the units that contribute to each Course Learning Outcome for each course. A portion of this report for the BSc(IT) program is shown in Figure 2.
A Generic Graduate Attributes report can also be generated for each degree program. This report shows Generic Graduate Attributes developed by each unit in the context of a given course structure, based on the relationships defined in the database between Unit Learning Outcomes, Course Learning Outcomes and Generic Graduate Attributes.
For example, Figure 3 shows a portion of the Generic Graduate Attributes Report depicting the attributes developed by units in the first semester of year three for students enrolled in the BEng (Software Engineering) program. Columns a through j in the figure correspond to the 10 Generic Graduate Attributes defined by Engineers Australia. Columns k through m correspond to additional attributes identified by the Department of Computing.
The format of the Generic Graduate Attributes report conforms to the standard presentation style used in recent accreditation documents submitted by the Faculty of Engineering and Computing at the authors’ institution. The Outcomes Database also maps the rubrics used to assess outcome attainment to the corresponding Unit Learning Outcome and the contribution of related assessment criteria to the final mark. Figure 4 shows an example for a first year, first semester programming unit. It is an important feature of the database that these relationships are automatically maintained, such that if any Unit Learning Outcome, criterion, or assessment weighting were to change, this report would be automatically updated the next time it was displayed.
4
Iterative Curriculum Development
Although the process of developing an Outcomes Focused approach began with Generic Graduate Attributes from Engineers Australia and later Course Learning Outcomes defined by the Department curriculum committee, the first iteration was essentially conducted in a bottom-up fashion.
Effectively, the existing programs were revised utilising the new Outcomes Focused approach by identifying appropriate Unit Learning Outcomes for each unit and then directly mapping these to the Course Learning Outcomes and indirectly to the Generic Graduate Attributes.
It was realised that several iterations would be required, during which Course Learning Outcomes would gradually be refined. The process would then be repeated until a good fit was found, meeting the objectives of the various programs and conforming to the requirements of our institution and accrediting bodies.
This iterative process is still underway and the results reported here are preliminary. However, the Outcomes Database has proven to be indispensable for this iterative approach. In part, this is because the various automatically generated reports highlight aspects of the curriculum plan that require attention during subsequent iterations.
Aspects of the curriculum plan that require attention are generally obvious from the various database reports, and include the following scenarios.
• The appropriate Course Learning Outcome is not in the common set for those courses in which the unit is a core requirement. Sometimes this becomes apparent in the Course Learning Outcome report when there are Course Learning Outcomes that are not listed as being addressed by any unit, and when an outcome exists that would otherwise be appropriate. This is because the appropriate outcome is not included in the set of potential choices presented by the database in the GUI because the Course Learning Outcome is not shared by all courses for which the unit is a core requirement.
• Sometimes, the Course Learning Outcomes report reveals that a given Unit Learning Outcome is mapped to an inappropriate Course Learning Outcome. For example, this is the case for Mathematics 115 and Statistics for Engineers and Scientists 101 in Course Learning Outcome #1 of Figure 2.
• The Course Learning Outcome is too specific and is really more appropriate as a Unit Learning Outcome. This is sometimes apparent in the Course Learning Outcome Report when there is only one unit that maps to a given Course Learning Outcome, and/or the Course Learning Outcome simply restates the Unit Learning Outcome.
• A single Course Learning Outcome covers too many topics in a manner that does not facilitate sharing common outcomes across courses. An example would be those cases in which a subset of common outcomes are included with other more specific skills in a single Course Learning Outcome in a way that excludes other courses which do not include that skill. In this case, the common outcomes need to be restated for the other course.
• A Unit Learning Outcome has not been associated with a Course Learning Outcome, possibly because none are appropriate. This is sometimes apparent in the Generic Graduate Attributes report when a unit is shown to contribute to none of the attributes.
5
Comparison to Manual Mappings
The automatically generated Generic Graduate Attributes Report, a portion of which is shown in Figure 3, was compared to a similar report produced manually.
Initial versions of the manually produced report were written during a 2001 workshop involving all members of the academic staff from the Department of Computing. The manually generated report has been updated frequently since that time. The report gives the justification for claims that stated Generic Graduate Attributes are developed in a given unit. This information is also included in unit outlines distributed to students during the first class meeting each semester.
Figure 4. Assessment criteria used to measure the attainment of Unit Learning Outcomes in a first year, first semester programming unit, as stored in the Outcomes Database.
The manually generated report was produced prior to the time that Course Learning Outcomes were identified and was not necessarily tied directly to assessments.
As a result of the comparison, significant discrepancies between the automatically and manually generated versions of the Generic Graduate Attributes report were found to exist. These discrepancies generally occurred when a pedagogical approach or learning experience not directly associated with an assessment was used as justifications for the claim.
With hindsight, the cause of this inconsistency seems somewhat obvious, although at the onset of this exercise it was not. However, it becomes readily apparent when the manual and automatic reports are compared. Such a comparison is useful in highlighting those learning experiences that are not directly assessed.
For example, the unit outline for one unit lists a Generic Graduate Attribute (bold) and makes the claim that this attribute is addressed in the unit (italics) as shown below:
"Expectation of the need to undertake lifelong learning and the capacity to do so.
Case-studies and journal articles from the software engineering literature are required reading for this subject. This experience is meant to foster good professional habits, and to demonstrate the importance of remaining current in ones discipline by reading the professional literature."
Data contained within the journal articles are used by students to complete practical worksheets. However, these worksheets are not directly assessed but are intended to be preliminary learning experiences upon which later assessed work is based. Whether students actually read the journal articles in question is not directly or indirectly assessed. They are only asked to use the data contained within the articles when completing the worksheets. In this case, the automatically generated report has revealed that the justification for claiming that this Generic Graduate Attribute is addressed is weak because it is not directly associated with an assessed Unit Learning Outcome.
The unit outline for the unit also provides this example: "Understanding of the ethical
responsibilities of the professional software engineer with respect to maintaining a high level of product quality.
The Codes of Conduct for the Australian Computer Society (ACS) and Institution of Engineers Australia (IEAust) are presented in this unit. The ethical duty of Software Engineers to produce quality products that conform to a given specification, meet customer expectations, and are fit for a given purpose is discussed in detail."
Examples from Burnmeister (2000) are discussed during class to highlight how the Australian Computer Society (ACS) Code of Ethics might be applied in various
hypothetical scenarios. Nevertheless, this material is not directly assessed in this unit.
Dook et al. (2005) argue that it may not be necessary to directly assess every outcome. They write:
"... a discussion about ethical issues may be purposely raised in a tutorial or laboratory session, but there may be no attempt to formally assess whether or not students have achieved the outcome 'appraisal of ethical issues related to the subject matter'. The discussing itself may serve the important purposes of raising pertinent questions for ongoing reflection and consideration."
This suggests that it is reasonable to claim that a non-assessed learning experience contributes to the development of a Generic Graduate Attribute in some circumstances.
6
Future Work
As a result of these observations, a future version of the Outcomes Database is planned that will contain an additional table to capture the relationships between non-assessed learning experiences and the Generic Graduate Attributes.
With this minor modification, once the assessment portion of the database depicted in Figure 1 is fully populated (late 2005), it will be possible to automatically generate an expanded Generic Graduate Attributes report showing the following for each unit in a given course:
• the specific assessment items that contribute to the development of Generic Graduate Attributes; • the percentage contribution of each assessment to
the overall mark and the rubric used to determine the level to which it has been achieved; and
• non-assessed learning experiences that contribute to development of the Generic Graduate Attributes. It is also interesting to note that data contained in the Outcomes Database includes much of the information communicated to students in unit outlines, including the unit name, Unit Learning Outcomes, and assessment details. This suggests that unit outlines can be automatically generated from information stored in the database. Doing so has the potential to reduce the staff overhead required to maintain unit outlines while ensuring consistency of presentation.
The unit syllabus is also contained in the Outcomes Database for the convenience of curriculum designers and reviewers. It is also usually contained in unit outlines and the on-line University Handbook. Consequently, syllabus data are currently replicated in redundant locations on the University network.
A Content Management System that serves as a middleware broker between information sources and sinks is the next logical step in the use and management of learning outcomes, assessments, and related documents.
In particular, a Content Management system could: • ensure that each data item is stored in one location
only;
• eliminate the need for redundant data entry and the human overhead associated with it;
• cater to the different data formats and information sources used by different lecturers and departments while providing links to related items (e.g. marks in spreadsheets and the percentage contribution of that assessment to the final mark as stored in the Outcomes Database);
• provide for the flexible creation of new document types from distributed information sources;
• ensure the consistency of data presentation in documents that are generated on-the-fly from the required information sources;
• facilitate data approval and change management. In collaboration with a Content Management System and other information sources on the network, the Outcomes Database could provide information to the following data sinks.
• Unit Outline Generator- producing unit outlines
from data stored in the Outcomes Database and other locations on the University network.
• Student Portfolio System- storing samples of
student work and demonstrating attainment of outcomes stored in the Outcomes Database.
• Time Management System- managing due dates
and expected effort for each assessment stored in the Outcomes Database.
7
Conclusions
The Outcomes Database provides an effective means to manage complexity during the transition to an Outcomes Focused approach to curriculum design and review. In particular, automatically generated reports highlight those places where outcomes statements and their relationships to other parts of the curriculum design require further work during subsequent iterations.
As an on-line, web-based system, the Outcomes Database promotes collaboration between stakeholders, including Unit Controllers, Managers, and members of curriculum review committees.
In its present form, the Outcomes Database has been invaluable in managing the complex relationships amongst units and courses in a department offering several related undergraduate degrees. When combined with the planned Content Management System, the Outcomes Database will be able to provide outcomes information to a broader set of applications for use by both students and staff. This will enhance the quality and consistency of learning experiences and provide a measurable means for defining and archiving attainment of learning outcomes.
8
References
Anderson, LW. Krathwohl, D.R., Airasian, P.W., Cruiskshank, KA., Mayer, R.E, Pintrich, P.R., Raths, J, and Wittrock, M.C. (eds.) (2001): A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Addison-Wesley Longman, ISBN 0-321-08405-5. Barrie, S. (2004): A research-based approach to generic
graduate attributes policy, Higher Education Research & Development, 23(3):261-275.
Bell, T.D. (2000): Proven skills: the new yardstick for schools, IEEE Spectrum, 37(9):63-67.
Botev, C., Chao, H., Chao, T. Cheng, Y. (2005): Supporting Workflow in a Course Management System, ACM SIGCSE ’05, pp. 262-266.
Bradley, A. (2005): Engineers Australia National Generic Competency Standards - Stage 1 Competency Standard for Professional Engineers, Document No. P05. Engineers Australia Accreditation Board.
Burmeister, O.K. (2000): Applying the ACS Code of Ethics, Journal of Research and Practice in Information Technology, 32(2):107-120.
Dook, J., Thompson, E., McNamara, B., Hepburn, N., and Baird, C. (2005): Outcomes Based Education and Assessment at UWA, Accessed 17 August 2005, http://www.catl.uwa.edu.au/obe/outcomes_based_educ ation_and_assessment_at_uwa.
Engineers Australia (1996): Changing the Culture: Engineering Education into the Future, ISBN 858256630.
Engineers Australia (1999): The Manual for the Accreditation of Professional Engineering Programs, Revised 7 October 1999.
IEEE and ACM (2001): Computer Curricula 2001: Computer Science, Final Report, December 15, 2001, Joint Task Force on Computer Curricula, IEEE Computer Society and the ACM.
IEEE (2004): Guide to the Software Engineering Body of Knowledge, 2004 Version, SWEBOK, Abran, A., Bourque, P., Dupuis, R., and Tripp, L.L. (eds.) ISBN 0-7695-2330-7.
Jolly, L., and Humphries, J. (2003): Using an Electronic Tool to Encourage Pedagogical Change, Proceedings of AaeE Conference, 29, September-1 October 2003. Marshall University (2005):Comparison of Online Course
Delivery Software Products, accessed 27 Oct. 2005, http://www.marshall.edu/it/cit/webct/compare/compari son.html.
Ong, S.S. and Shepherd,, J. (2002): WebCMS: A Web-based Course Management System,. Workshop on Databases and Expert Systems Applications, pp. 345-350.
Parnas, D.L (1999): Software Engineering Programs are not Computer Science Programs, IEEE Software, 16(6):19-30.
