5.5 Plugins
5.5.10 End-User Plugins
These “End-User” plugins were described from the user view in Chapter4. Brief descriptions and cross-references are provided here for completeness, with reference to Table5.4. Browsing Images and Folders (browser) This plugin provides the hierarchical and
clustered browsers that are critical for photo sharing with authentic collections of digital photographs, used in the PhoTable application. Common parts are a plugin library (§5.4.5), Browsers are discussed briefly in §4.3.7 and the clustered browsers are explored in the context of photo sharing in §4.5.5.
Storage Bins (storagebin) Storage bins also leverage common parts of the BrowseContainer implementation (§5.4.5). Storage bins are described in §4.3.9.
Video Loader The video loader plugin provides a means to load an initial set of Video images (§5.4.3) into the interface. Video images are described further in §4.4.2.
5.6. Feature Highlights CHAPTER 5. DESIGN
5.6
Feature Highlights
Here, the focus is on features newly implemented in Cruiser, rather than those provided by software libraries external to Cruiser (specified in italics with the name of the library in bold italics). Also, the focus is features of the Cruiser framework – features from the user view (i.e. application features) are discussed in Chapter4 and those most relevant to the PhoTable application were discussed in Section4.5. The features listed will be elaborated in the System Overview (§5.1), here is provided a summary of the features considered when designing the Cruiser framework:
• Highly concurrent and thread-safe operation.
• A responsive main(), drawing and input-processing thread able to maintain 60fps (save some exceptional circumstances), with other processing managed in separate
threads, involving:
– caching of transformation matrices;
– separate threads for intensive (or potentially blocking) tasks such as: ∗ loading and decoding images from disk,
∗ pre-processing images into mipmappedtextures, ∗ saving (or streaming) sound recordings to disk. • Efficient layout of on-screen collections of objects, involving:
– transformation inheritance for dependent objects;
– layouts provided for fixed, relative, “flow”, and space-filling layouts;
– a framework whereby plugins may implement their own layout, and hooks to apply this layout to managed collections of objects.
• Efficient input handling, involving:
– a versatile framework for interfacing with arbitrary input hardware (via a plugin);
– coordinate system conversion and calibration, supporting arbitrary scaling, keystone, shear, offsets and other warping;
– ability to receive input over TCP;
– a flexible gesture framework for interpreting input as manipulations;
– support for multi-touch, user-identification, and multiple simultaneous inputs; – conversion of mouse cursor inputs from the windowing system into gestures (e.g.
for debugging).
• A versatile audio framework, with support for:
– loading and saving various sound formats for prompts and annotations (e.g. formats encoded for efficient use of space when recording voice);
– streaming audio to or from disk (encoding or decoding on-the-fly); – sample rate conversion of loaded audio where necessary;
– managing multiple simultaneous recordings; – mixing multiple simultaneous sound samples;
– providing feedback for play and record levels of any currently playing sample (or the record level);
– bindings for Python, Ruby, and other languages,
∗ this allowed album creation to be implemented in Python, for example. • A range of media support:
– Image formats via SDL_Image38 (JPEG, PNG, GIF, TIFF, MP, LBM, PCX, PNM, TGA, XCF, XPM, XV):
∗ loaded from disk or network socket as an Image Resource in Cruiser that may be manipulated,
∗ also support for saving Frame captures and streaming screenshots (for video) in JPEG format;
– Sound formats via libsndfile39 (WAV, AIFF, AU, PAF, IFF, VOC, MAT, PVF, CAF, OGG, FLAC, others):
∗ also various encodings via libsndfile (PCM, ADPCM, µ-law, A-law, Vorbis,
others),
∗ loaded from disk or network socket as an AudioCircle object that may be manipulated and attached to objects in Cruiser,
∗ also support for saving explicit or background recordings (e.g. from a microphone) as PCM or ADPCM WAV in Cruiser;
– Video formats and encodings via mplayer40(AVI, MPEG, MOV, WMV, OGV,
MKV, VOB, others):
∗ also streaming formats such as RTSP, TV (via a tuner card), or a webcam, ∗ loaded into a Video Resource that may be manipulated in Cruiser via a
named socketcommunicating with mplayer;
– Remote framebuffer encodings via built-inVNCClient: ∗ 3DES password authentication via librfb library,
∗ RAW, RRE, CoRRE and copyrect encodings are implemented in Cruiser’s “rfb” plugin,
∗ remote desktop is updated into amipmapped texture for rapid rendering in OpenGL;
– Exif metadata via libexif41 library:
∗ JPEGExifthumbnail is extracted for rapid thumbnail loading when browsing photos,
∗ photo capture time (and other metadata) is read and saved to an SQLite database cache for accurate sorting and clustering.
5.7
Chapter Summary
The Cruiser framework is decoupled in three main parts, which have been discussed in this chapter. The core provides key tabletop interaction functionality and the framework for allowing new functionality to be added while maximising reuse of existing parts. Plugin
38 http://www.libsdl.org/projects/SDL_image/ 39 http://www.mega-nerd.com/libsndfile/ 40 http://www.mplayerhq.hu/design7/info.html 41http://libexif.sourceforge.net/
5.7. Chapter Summary CHAPTER 5. DESIGN
libraries introduce new functionality, not needed by all tabletop applications, in a way
that allows it to be shared amongst multiple plugins. Plugins introduce new features and interface elements that build on the core and plugin libraries to realise new interfaces that leverage the Cruiser framework to provide a mostly complete application. Another layer – utility libraries – encapsulates reusable, general purpose tools that were created specifically for Cruiser. Throughout development, the drivers for design (§3.5) influenced the implementation approach, with a focus on stability, flexibility, cohesion, platform and input hardware independence, and performance.
Chapter
6
System Evaluation
Because of the dual goals of this thesis, there are necessarily two components to the evaluation. For the framework design, this chapter presents the evaluation of the Cruiser framework; that is, how Cruiser provides an effective architecture for the development of highly immersive and pervasive interactive tabletop applications. The following chapter will present our evaluations of the user interface – its learnability and usability – and how PhoTable is able to facilitate natural storytelling around personal digital photographs and the automatic capture of those stories to create an annotated digital photo album. These two components are related – the effectiveness of Cruiser is critical as the means to create a useful PhoTable.
One aspect of evaluation is usability, but usability evaluations are not always appropriate, and they cannot be used to evaluate all components of the system. This thesis addresses usability as part of the interface evaluation in Chapter7. For this chapter, Greenberg and Buxton [2008] offer some alternatives for how a novel system should be evaluated, where a usability evaluation might not be appropriate. Examples include: a design rationale, a vision of what could be, expected scenarios of use, reflections, case studies and design critique. In this thesis, these are not subjects for a single chapter, but are instead drawn from all parts of this thesis.
Chapter 1.2 presented the photo sharing vision and the scenario of use within which we framed the problem, but also introduced the necessity of a framework for realising this vision. In Chapter3 we gave our design rationale, based on guidelines derived from best practices in user interface research and identified usability problems that might arise. Chapter 4 presented the user view with a number of case studies, and reflected upon our design choices, including several avenues that were explored but were not included in the final version of PhoTable. In addition, Chapter 7, to follow, explores usability and reflects upon some other design choices that were discontinued. The focus of this chapter will be a systems-oriented evaluation of the Cruiser framework.
6.1
Introduction
In this thesis, the Cruiser framework is key to address the research problem outlined in Section 1.2. Cruiser is a system for building applications, so evaluation has been designed following best practice for systems. Olsen [2007] has recommended that systems evaluation should be based upon “study of use,” meaning a study of the system as people would actually use it. This is the approach we follow.
Importantly, Olsen discusses a number of evaluation errors that may cause damage to this research field. One of these is the usability trap. Usability metrics such as task completion time, learnability and error minimisation are important for interactive systems.
6.1. Introduction CHAPTER 6. SYSTEM EVALUATION
PhoTable is an interactive system with a design that has been driven in part by a goal of optimising these measures. However, being a new system, there is no suitable system that we may compare with PhoTable. Any attempt to find a comparative system would detract from the novelty that PhoTable provides. Being, also, a complex system, these usability metrics are difficult to measure, since authentic use involves significant creative user input and social interaction as they tell stories. This is the usability trap: research should not be limited to what is trivially measurable, in order that we may effect significant change.
Olsen [2007] suggests alternative methods to evaluate systems, and to demonstrate evid- ence of the claims made about the system. This includes putting the system in appropriate context, in terms ofSituations, Tasks and Users (STU); demonstrating importance; showing (e.g. for a tool) that it addresses a problem not previously solved; showing generality; and ensuring that it scales up to realistic problems. Another important attribute is reduction of the effort required to iterate on many possible solutions, or to reduce solution viscosity which can be done in terms of flexibility, expressive leverage and expressive match.
Olsen also discusses effectiveness in terms of power in combination, which includes
inductive combination (e.g. combining primitive design components into more complex
designs), simplifying interconnection (e.g. by implementing a standard interface to allow integration with all existing components), and ease of combination (i.e. making interconnec- tion simple and straightforward). Context, importance and novelty were initially covered in Sections 1.2and2.3, with details in Chapter 4; Cruiser’s framework approach (outlined in Chapter 5) is key to satisfying many of the other requirements, and will be discussed further in this chapter.
Greenberg and Buxton [2008] have also observed that, some of the time, usability evaluation can be considered harmful. If applied incorrectly, usability evaluation can “quash potentially valuable ideas early in the design process, incorrectly promote poor ideas, misdirect developers into solving minor vs. major problems, or ignore (or incorrectly suggest) how a design would be adopted and used in everyday practice.” For example, an existence proof could be used to show that PhoTable allows people to share their digital photographs faster or with fewer errors than some other state-of-the-art system. But this would be misguided. Not only is efficiency meaningless for such a task (where increased time may even be an indication of enjoyment and engagement), but there exists no existing system in the state of the art that could be used for a meaningful comparison. Furthermore, the task would have to be artificially constrained in order to be evaluated. We would, in effect, be designing our interface to suit the evaluation, rather than addressing the research problem; that of sharing photographs (§1.2).
Usability evaluations of the user interface are still important and this thesis explores them in Chapter 7. Part of what this chapter explores might be described as a usefulness evaluation in relation to the Cruiser framework. Indeed, it has been said that the trouble
with computers is not that they are too hard to use, but they do too little that is useful
[Landauer, 1995]. To show that the Cruiser framework is useful, we show how it has been used by other system builders to create new tabletop applications. To provide effective interaction, the Cruiser framework must also have sufficient performance to be responsive for the kinds of tasks for which it was designed – we will show that it does.
In summary, this chapter evaluates the Cruiser framework from a systems perspective. To do this, we first report its use, by others, demonstrating its power and flexibility in terms of the ease of creating new plugins. We then discuss performance of the framework in terms of rendering efficiency, resource usage and scalability. The summary revisits evidence for the usefulness of the Cruiser framework, which is drawn from many chapters of this thesis.
Plugin Name Code size in SLOC Author(s) Time Cost in man-months Context
fsviewer 926 Anthony Collins
(final year undergraduate)
4* Undergraduate Honours’ thesis project, 1 semester load, includes writing thesis and evaluation studies
(July 2006)
blackjack 3 171 Rainer Wasinger
(postdoctoral researcher), Anthony Collins
3 Industry research project, 3 months including some other duties, client interaction
(July 2007) map tagger 553 Glen Whitaker (final year undergraduate) 4 Undergraduate Honours’
thesis project (as above) (July 2006)
brainstorm 601 Gregory Darke, James Bunton
(3rd-year
undergraduates)
3 Undergraduate summer re- search scholarship, 2 months on multiple projects (Janu-
ary 2008) menu system 877 Hugh MacReady (recent Bachelors’ graduate)
2 Industry research, 2 month summer employment (Febru-
ary 2005)
*Time to develop the “OnTop” version of the fsviewer plugin, which is continuing into postgraduate study as “Focus”
Table 6.1: Summary of Contributed Plugins, built using Cruiser
6.2
Contributed Plugins
All elements of Cruiser presented in Chapters 4 and5 have been implemented by myself. However, through the plugin application framework, Cruiser’s design lends itself well to experimentation by others. The flexibleAPIprovides a high degree of utility to create new tools to realise novel tabletop “applications”, implemented through the use of plugins. In this way, as an application framework, Cruiser assumes much of the burden of application development. To experiment with new functionality, developers need only implement a plugin, rather than a new application.
This section will describe plugins contributed by other developers. Table 6.1 is a summary of the plugins that will be presented, showing the size in Source Lines Of Code (SLOC), the authors involved, an estimate of the development cost in man-months, and the development context. The author and context columns are provided to give a sense of the expertise and time-commitment of the plugin authors.
Assistance was provided by myself, mostly in the early stages, to familiarise the author with the framework – typically by using a simple plugin (such as the storage bin) as a worked example and starting point for the new plugin skeleton. After this initial familiarisation, my own role in the development of each plugin was minimal – mostly limited to answering questions and giving advice on how to best leverage the functionality already present in the Cruiser framework (where my role differed, details will be given in the relevant subsection).
Each subsection below presents a plugin and its application context and includes the following:
6.2. Contributed Plugins CHAPTER 6. SYSTEM EVALUATION
Figure 6.1: Screenshot of the OnTop associative search Cruiser application [Collins et al., 2007]
• an overview of added functionality and a discussion of the evaluation or usage scenario of the new application,
• the main aspects provided by the Cruiser framework that were leveraged to provide this functionality,
• a description of the new code and how it extended upon components in Cruiser, • evaluation of ease of creation in terms of effort (lines of code) to implement the
application,
• evaluation of ease of creation in terms of developer time taken to implement the application, and
• a summary of what each plugin demonstrates about the power, flexibility and extensibility of Cruiser.