In this chapter, we discussed research into development environments from a historical perspective, and then categorized these environments into end-user programming environments and text-based programming environments. We also covered tools for programmers that support the development of programs with real-world I/O.
Text-based development environments have long been conservative in their development, and typically employ a text-based code editor and an integrated text-based debugger. Typical visual programming environments are designed for end-users and eliminate the need to manually write code (i.e., text). Existing IDEs developed using these approaches assume that the development and the runtime environments use the same input and output devices. However, a grow-ing number of programs involve real-world I/O, and where development suffers
from a gap between the development environment and the runtime environment.
Programming by example systems may eliminate this gap, since they use the same device for the development and execution of the programs. However, most of them are designed for end-users and do not provide precise control over the program logic. Tools for the programmer address this issue by providing partial support on the programmer’s workflow.
Given this background, the aims of the work described in this dissertation are to integrate graphical representations into text-based IDEs and benefit from the unique advantages of both the comprehensibility of graphical representations and the expressivity of textual programming. The graphical representations are of example data retrieved from the real world, bridging the gap between the development environment and the runtime environment. The resulting IDEs have both visual and textual components, and provide total support for the programmer’s workflow.
Chapter 3
Integrated Graphical Representations
In this chapter we define the scope of this work by comparing it with existing approaches. In Section 3.1 the characteristics of programs that deal with real-world input and output (real-real-world I/O) are highlighted by comparing them with conventional input and output modalities. In Section 3.2 the workflow of the development of such programs is described, and the model describing the programs is introduced. In Section 3.3 our approach of integrating graphical representations is described, along with the expected advantages, which will be examined in the subsequent chapters.
3.1 Real-world Input and Output
We aim to address difficulties in the development of programs that use real-world I/O. Real-real-world I/O is data that is inputted or outputted between the real world and the computer. This includes visual data, such as photos and videos, the properties of objects, such as color, shape, and locations, as well as more structured information, such as the posture of humans or robots. We focus on these visual data; however, real-world I/O also includes data describing sound, haptic technology, smell and taste. While processing such information is challenging, it may facilitate a new paradigm of user interfaces, such as organic user interfaces [130].
The mouse, pen, keyboard and display are all physical devices in the real world that function as interfaces between the user and computer; therefore, it appears reasonable to consider that they may also provide real-world I/O. However, we exclude these conventional forms of I/O from our definition. Conventional I/O is designed to map well to the metaphorical concepts used in the computer, such as the WIMP environment and modern graphical user interfaces. Therefore, conventional I/O devices are designed to provide discrete values by nature, such as keystrokes (e.g., a, b, c...), which map to character codes, or mouse movements that correspond to two-dimensional coordinates (i.e., numbers) describing pixels on the display.
Real-world I/O provides data that cannot be represented using symbols or constants. Real-world I/O contains information that exists in the real world regardless of the existence of the computer. All data for real-world I/O is sampled and discretized using physical devices such as image sensors and rotary encoders, and subsequently processed by the computer; however, it has continuous values both temporally and spatially.
One particular paradigm that has significant overlap with real-world I/O is recognition-based interfaces. This includes voice recognition and gesture recogni-tion. These input technologies create input with some ambiguity, which should be
handled as probabilistic events rather than conventional events with discretized data. This difficulty has been addressed by existing work on user interface toolk-its [100, 134]. Compared with such a toolkit approach, the work described in this dissertation aims to provide integrated support to the programmer. While such toolkits address difficulties in packaging common features and exploiting useful application programming interfaces (APIs), difficulties in programming using a text-based IDE remain. Instead, in this work, we investigate ways to enhance existing text-based IDEs using a combination of multiple tools for programmers, such as editors, debuggers and APIs. We aim to support the whole workflow of programming.
Our approach is strongly inspired by Patel’s work on providing an IDE for machine learning [125]. To create a machine learning application, which may be a recognition-based interface, the programmer must collect many example data and create a classifier by training and testing the classification pipeline. The IDE is specifically designed to support the development of such programs. As with our approach, Patel’s IDE does not focus on providing specific features that eliminate the requirement for implementation, rather focuses on providing the necessary development support. The main distinction comes from differences in the application domain: Patel’s work focused on machine learning applications and our work focuses on programs with real-world I/O.
The development of programs with conventional I/O uses the same environ-ment for developenviron-ment and execution; however, the developenviron-ment of programs with real-world I/O makes use of different devices for development and execution.
This widens the gulf of execution and evaluation for the programmer, making it difficult to develop such programs using conventional text-based IDEs. While toolkits for programmers typically cover partial workflow of the programming ac-tivity, IDEs are designed to cover the entire workflow. We aim to cover the whole workflow and bridge this gulf using a text-based IDE that includes graphical representations of real-world data.