Comparison Framework Analysis Findings

The comparison framework analysis uncovered that there is little to differentiate individual tools in the categories of Tool Design or Deployment Environment. Generally, simulation-based test-beds tend to perform better in terms of scalability, configurability and to a certain extent heterogeneity, simply because both devices and the physical environment can be modified more easily and at relatively little cost. Live environments are realistically only configurable in terms of the devices and network deployment. Live test-beds also tend to have the highest costs, particularly those that are custom built and include the expense of the physical building.

However, live environments produce the highest fidelity experiments because test-users can interact with the system in an authentic setting. This provides

the opportunity to gain deep understanding about how these environments will operate, and make informed decisions about best practice for ubicomp design. At the same time though, the development cycle for individual systems also needs more accessible tools that can be used regularly, repeatedly, and with lower investment. Simulation-based test-beds have the necessary characteristics to be well-suited for frequent testing, particularly in the early-mid stages of the development cycle.

In terms of testing ubiquitous computing systems, the comparison framework analysis uncovered that few tools address the issue of automatically monitoring system behaviour at run-time. UbiREAL and VisualRDK provide the only examples among the reviewed tools. UbiREAL does this by adopting a situation-based service specification and requirements testing approach. The aim of UbiREAL is to support end-users to prototype applications, for their own domestic environments, and visually demonstrate how a prototype will behave for specific situations at run-time. VisualRDK makes a contribution to automatic monitoring by supporting developers to debug specially compiled, localised versions of their code. Although both of these tools make contributions to this issue, both also place more emphasis on technical effectiveness, than on the more user-centric issue of appropriateness.

It was also uncovered that no tool provides automatic monitoring while also engineering the opportunity for spontaneous or unanticipated situations to arise. UbiREAL comes closest with its situation-based approach to system specification. However, UbiREAL’s approach is limited due to a lack of generality and over-specification of test cases, which make it difficult to test for the unanticipated.

Evaluation has generally been poorly addressed by early stage and low investment tools. It is probable that the trend for lower investment tools is only emerging now because earlier tools needed to first focus on supporting designers to more easily prototype ubicomp systems. The Context Toolkit was seminal in this respect, and more recently Topiary has also made a significant contribution in terms of enabling developers to rapidly create ubicomp systems.

Live environments have been demonstrated to provide reliable results in terms of evaluation, however they have not yet been demonstrated as a practical solution

for mass testing of systems on a large scale. Limitations in terms of cost and lack of flexibility, make them inaccessible to many projects. UbiREAL and VisualRDK provide some degree of evaluation for technical effectiveness. UbiREAL enables a tester to determine that for specific inputs the system behaves as intended. VisualRDK supports the tester to examine the program flow and data structures of a ubicomp system at run-time.

In recent years, a trend has emerged of simulation and prototyping that is placing more emphasis on addressing the issue of testing and evaluation of ubicomp systems earlier in the development cycle and requiring lower investment. Many of these tools are looking to more formal, systematic and automated approaches to testing. UbiREAL made a notable contribution by enabling end-users to define a formal, testable, situation-based system specification. VisualRDK supports a designer to debug the program flow and data structures of a ubicomp system at run-time, using a specially compiled localised version of the system. This addresses technical effectiveness but not the more user-centric issue of situational appropriateness, the latter of which is more central to the focus of this thesis.

DiaSim provides an element of traceability by enabling the designer to manually examine log files which can uncover sequences of events leading to specific situations. This is more than most tools report, however it is still insufficient for thorough examination of a system’s behaviour, particularly because DiaSim does not support automatic monitoring for problem detection. The large quantities of data that ubicomp systems can produce has the potential to make this a tedious and laborious task. AmISim aims to support forensic analysis as a method of testing, debugging and analysing ubicomp systems, however results for this are yet to be published.

3.8

Chapter Conclusions and Summary

This chapter began by presenting a survey of modelling approaches appropriate for supporting a situation-based testing approach. The chapter also presented a survey of ubicomp tools that have made contributions to the area of testing and evaluating

action-based ubicomp systems. A comparison framework, based on the challenges identified in Chapter 2, was devised as an instrument to compare the surveyed tools.

Based on the review of tools, there is promise in simulation-based approaches and in the use of iterative and incremental development cycles. It is apparent however, that although much work has been conducted towards addressing the prototyping phase of an iterative development cycle, there has been less emphasis on the testing and evaluation stages required to complete each cycle, particularly for lower investment testing. There is a need for methods and approaches to testing that address the novel issues introduced by ubiquitous computing. In particular gaps and open issues remain surrounding (i) monitoring spatio-temporal relationships throughout large scale environments, (ii) assessing the appropriateness of situational outcomes that result from the implementation of a particular design, (iii) traceability of causal factors across the physical-digital boundary, and (iv) provision of structured feedback to the designer to inform subsequent iterations in the design cycle.