System Design

solutions may be stored in the database and handed over to grading components later. Syn-chronous grading is at least required in self-training scenarios where an exercises consists of several steps and feedback should be presented after each step without interrupting the learner for a long time. Asynchronous feedback is at least required in the mass validation of exams where each individual grading process is potentially a long-running task.

All grading components should be easy to develop, integrate and manage at run time.

They also must be able to run outside the actual assessment system to allow load balancing:

Since the students’ access to the assessment system must be as robust as possible and the system resources needed for marking are type-specific and not predictable, it is necessary to loosely couple these tasks. Thus a run time system is required that executes arbitrary grading components and connects to the assessment system for this purpose. The assessment system is responsible for delivering solutions on request of the marking process until every single solution has been assessed and marked. Running grading components in a separate run time system that connects to the assessment system raises additional security issues. To prevent fraudulent attempts, the assessment system must make sure that only valid marking run time systems may access stored solution data and may deliver results for them.

There should be a mechanism allowing grading components to reuse results from other grading components as additional input. To allow for this, proper chaining of grading com-ponents must be possible, where some comcom-ponents are only invoked if others have already produced results.

3.1.4. Review Management

Independent from any automated grading process, it is required that a teacher can access any submitted solution in order to review it and to add a manual result to it. Moreover, it must be possible for a teacher to review any comments made by an automated grading component and to remove any single error message or warning generated. This way, “false negatives” in automated grading can be detected and removed manually.

Obviously a well presented list of errors and warnings for a given solution has to be a core feature of a tutoring system. As already required in the previous section, each message should be related to a solution artefact as close as possible and this relation has to be comprehensible in the review interface. Since students have to learn just by studying the results from the grading components in pure e-learning scenarios, the feedback has to be presented in a way that is clear enough to be understandable without further explanation.

To facilitate quality assurance, students should be allowed to vote up or down on feedback, indicating whether a particular line of feedback was helpful or not to them.

Also in exam scenarios it should be possible to let students review their own solutions afterwards and see a justification of the results, both for educational and judicial purposes.

3.2. System Design

The requirements sketched above can be turned into a data model representing the entities handled by the system and an architectural model representing the system components.

Both models are discussed in short in the following subsections.

3.2.1. Data Model

The data model of JACK is shown in figure 3.1. The entities present in this model can roughly be divided into three groups: (1) Entities concerned with exercises and their ar-rangement into courses and exams, (2) entities concerned with solutions to exercises and feedback given to these solutions, and (3) entities managing users and their rights.

Exercise and Examination Data

The central concept to manage exercise and examination data within JACK is the entity

“AbstractExercise” shown on the top left in figure 3.1. It is specialized for actual exercises and courses and holds meta-data about the exercise (such as a title or rules for submission).

For exercise delivery, each exercise states a display type in the meta-data. Depending on this type, the user interface can decide how to display the exercise to the user or even to not display it at all because of missing capabilities to show it properly. Files belonging to the exercise are attached as resources, belonging to one of the four different types of resources identified in the requirements. Resources are never shared among exercises, but always belong to exactly one exercise. The checker configuration attached to the exercise states which resources are handed to which grading component for which purpose. Consequently, it is not necessary to make all resources visible to all grading components and it is also possible to invoke the same grading component with different configurations and resources on the same exercise. As part of the meta-data of an exercise it is specified how the final mark for each solution in calculated from the individual marks given by each of the grading components configured for this exercise. It is not necessary to use each grading component in this calculation, so a component can also be used to create additional feedback that does not influence the mark or is not visible to the student at all, but just to the teacher.

Exercises can be grouped to courses based on their display type and their tags, and they can be assigned to exams. Both entities carry their own meta-data.

Solution Data

In analogy to the entity “AbstractExercise” there is an entity “AbstractSolution” (top right in figure 3.1) which is the core concept for handling the user submission. Besides containing meta-data like a submission timestamp, it is mainly a container for the resources submitted by a student and the feedback generated by the grading components or by manual results added by a teacher. For the files attached to a solution there is a distinction on whether they are user generated (thus submitted by the user) or generated by a checker (thus attached later during the checking process). Feedback is stored in error records, where each individual line of feedback can be ruled out by a teacher. Moreover, a student’s vote on the feedback can be stored. The marks given are in all cases handled as integer numbers from 0 to 100.

Grading components may also return additional attributes that are stored alongside the solution. This can be used to pass additional information from one grading component to another.

Courses and exams do not carry own attached files, but just meta-data and overall results.

For each solution submitted by a student and each asynchronous checker configured for the associated exercise, a job entity is created, which is used by the grading management and deleted as soon as results from the respective checker are available. For synchronous checkers,

3.2. System Design

Figure3.1.:ClassdiagramforthedatamodelofJACK.Primitiveattributesonallentitiesaswellasmultiplicitiesofassociations arehiddenforthesakeofreadability.

no jobs need to be created, as the solution can be handed directly to them, obtaining the results immediately.

User Data

User data is the smallest chunk of data handled by JACK. It consists of entities for each user, that can be either students, teachers or administrators. Besides user name, password, and role, user data only consists of information on which entity has been created and is thus owned by which user. For teachers, additional access rights can be granted to exercises, courses, and exams.

Participation of users in exams is covered via an additional entity “PatricpationRecord”

that may also contain an individual password required for access to this particular exam.

3.2.2. System Architecture

The component based system architecture of JACK is shown in figure 3.2. The architecture is organized in two parts to be run on different servers: The part running on the core server is designed as a classical three-tier architecture and cares for data storage, business logic, synchronous grading, and various user interfaces for teachers and students. For user authentication, external services can be called. Both the web-frontend and the web-service for Eclipse integration are concerned directly with the appropriate presentation of exercises and exercise resources. They also provide to the learners the necessary means to submit a solution. Both the web-frontend and the web-service for workers also deal with presentation of solutions, either for manual review via the web frontend or for automated grading on another server. They also provide the necessary means for submitting results from manual assessment by teachers or automated grading components, respectively. Finally, the web-frontend also provides means for authoring, import and export of exercises, courses and exams by teachers.

All three components call methods of the business logic core to provide or request data.

The business logic core cares for processing of this data, e.g. permission checks for access to exercises and exams, running synchronous checks on solutions, and maintenance of the job queue for asynchronous checks. It in turn calls the persistence layer or external services to fulfill its tasks.

The part running on worker servers is designed as service-oriented system and cares for asynchronous grading. Each worker server can be equipped with different marking compo-nents. Each of the checking concepts discussed in this thesis can be realized as concrete, complete, runnable implementation in such a component. Several worker servers can con-nect to the same core server and one worker server can also concon-nect to several core servers.

They issue periodic requests for work to the server and use the methods provided by the web-service for workers to retrieve data and send back results later if a job is available for them.