tioned above, we have reached the following objectives:
Usage for learning. The exercise representation format, the diagnostic capabilities of the system and the tutorial and presentation strategies suited the needs of teachers at schools and universities. The course contents within the ActiveMath system including hundreds of interactive exercises are used for learning in various schools and Universities in several countries.
Value as research tool. The rich expressive power of the exercise lan- guage and easily extensible framework for tutorial and presentation strate- gies allowed to design and perform various research and empiric studies in the field of computer-based education.
6.2
Performance tests
In ActiveMath we have performed automated tests measuring the per- formance of the exercises, as part of general performance testing for the ActiveMath system. These tests have been developed using the JMeter framework. 1
A part of this automated performance tests were tests with artificial users. ”Stress-tests” have been performed with up to 200 artificial users using the system simultaneously. Note that the typical learning time of the exercises was not taken into account, since not every exercise had such metadata. So, a fix delay of 3 seconds was used between the actions of artificial users. This, however, is in most of the cases too little time for performing an exercise step. This means that in the case of real human users the delay between exercise steps will be longer, so the system will have to answer less requests per time unit, which would increase it’s efficiency.
The results of the stress tests are shown in Figures 6.1, 6.2 and 6.3. We present three views of these result, that are most commonly used in perfor- mance testing, showing average, median and 90% line values respectively.
The average view shows us the sum of all time measurements divided by the number of observations. The median (or 50% line) is a number which divides the samples into two equal halves - the first half is smaller then the
140CHAPTER 6. EVALUATION OF COVERAGE AND PERFORMANCE median and the other half is greater. The median value of the (multi)set of numbers can be obtained by ordering all the values and taking the one positioned in the middle of the list. The median might be a better indication of central tendency then the arithmetic mean, and the measure it gives is more robust then the mean in the presence of outlier values (i.e., values that are numerically distant from the rest of the data). The 90% line is the value below which 90% of the samples fall. This value is robust in the presence of only a few outlier values.
From the tests we can see that the exercise system scales well for up to 100 simultaneous users, which corresponds to serving up to three school classes simultaneously using the same instance of the ActiveMath server. In this case the average request time for exercises is below 2 seconds and the median and 90% line are below 1 second. For 150 users the average value is about 3 seconds, which is still tolerable, the median value is below 1 second, and the 90% line value is pretty high (9 sec). For 200 students the average value reaches 9.3 seconds, which is practically unusable.
Therefore, currently the ActiveMath exercises scale well for 100 simul- taneous users per instance of the ActiveMath system, which is feasible for practical usage in the classroom.
One of the main reasons for using the same instance of the system for the whole class is to ease the collection of the performance data of students in order to generate group performance reports. Of course, there are other practical reasons, such as the need for a separate server to host each running instance of the ActiveMath system.
6.2. PERFORMANCE TESTS 141
142CHAPTER 6. EVALUATION OF COVERAGE AND PERFORMANCE
6.2. PERFORMANCE TESTS 143
Chapter 7
Conclusion and Future Work
In this work we have defined a knowledge representation for interactive ex- ercises and developed the Exercise Subsystem within the ActiveMath learning platform. This system has several novel aspects as compared to other existing systems that serve interactive exercises. Firstly, thanks to our framework of distributed semantic services and generic query format, we can connect to several external systems simultaneously in order to obtain diag- nosis of student’s answers. A quick diagnosis can be computed by a CAS, a more detailed diagnosis can be provided by a domain reasoner.
Secondly, we support a combination of manually authored and automati- cally generated exercises. Moreover, the exercises can be reused with different automated tutorial strategies.
A variety of presentation strategies allows for custom layout of feedback and via combining various user interface components we can implement dif- ferent approaches to user interaction.
There are several directions of further research we plan to conduct, which we briefly describe in the following sections.
7.1
Authoring Tutorial Strategies
Currently, the tutorial strategies for exercises are realized as programs. We have also developed a framework in which these strategies can be combined with each other to form more complex strategies.
One of the directions of further research is to devise a knowledge rep- resentation for tutorial strategies and implement a module of the Exercise
146 CHAPTER 7. CONCLUSION AND FUTURE WORK Subsystem that interprets such a representation and applies the correspond- ing tutorial behavior to the given exercise. Further step would be to develop an authoring tool for such strategies. As opposed to the low level proce- dure representation such as the so-called flow-lines in the EON strategy editor [Murray, 2003a], we aim at more high-level declarative representation that will allow the teachers to express their wishes for the tutorial behavior of the strategy without programming.