Discussion

The overall design of the GUI testing framework is partitioned on a host and a target with the Robot Framework as the front-end on the host-side and TouchGFX as the run-time environment on the target-side. With the use of Robot Framework follows actionword- and keyword-scripting as the primary approach to test specification, and the partitioned architecture creates the need for a communication protocol and mechanisms between the host- and target solutions.

3.5.1 Comparison to related work

The partitioned architecture of our framework is based on the work by Jacobs et al. described in Section 3.3.1. Their solution is not directed specifically at GUI testing, but the challenges of constrained resources, complex software, and heterogeneous hardware platforms are the same. The TestFrame engine developed by Jacobs et al. corresponds to our use of the Robot Framework. The use of the Robot Framework does however, provide greater extensibility due to its use of test libraries for keyword implementation. As its back-end, TestFrame has a test navigation layer con- sisting of host- and target-side interpreters and a communication tool. The tool, named ActiveLink, abstracts the specific method of communication from the TestFrame engine and the interpreters. This abstraction is important for heterogeneous platform support and is achieved in our solution through interface- and component-based development.

The solution presented by Lin et al., described in Section 3.3.2, is directed at GUI testing on smart- phones. As with a partitioned architecture, such as the one used in our framework, most of the processing is performed on a host machine. However, unlike our framework, GUI actions are not transmitted to a target platform but executed on a host-based representation of the AUT. Validation of the GUI state is also achieved through this representation of the AUT by the image process- ing of screenshots taken directly on the host. The tool used to create the remote representation is designed for Android smartphones and while smartphones come in various shapes and sizes, the general hardware and software architecture is a standardized platform. Their solution does not suffer from the challenges of heterogeneous environments faced by embedded test automation frameworks. Their use of Computer Vision seems promising, but without the host-based repre- sentation of the AUT our framework would spend a significant amount of resources frequently sending screenshots from the target to the host.

The Smart Wait method also proposed by Lin et al. is a simple approach to synchronization be- tween the host and target. Its primitive nature makes it a lightweight and versatile solution, but for our work it has been disregarded in favor of a solution on the target, described in Section 4.4. Due to the use of Robot Framework our solution is based on actionword and keyword-scripting. The work by Wu et al., presented in Section 3.3.3, also promotes this approach due to the high level of reusability and maintainability it provides. While their solution is also directed at smart- phone GUI testing, their use of actionword and keyword-scripting is generally applicable. As with the Robot Framework, their solution, called AKDT, uses test libraries. These libraries include application-specific functions that rely on the Android test automation framework Robotium, used to control and communicate with an Android smartphone. AKDT is thus very similar to our GUI testing framework, where Robotium is replaced by our back-end, TestGFX.

3.5.2 Comparison to thesis goals

In terms of the five quality properties presented in Section 1.3.1, our primary objective is to provide a solution to embedded GUI testing that provides the functionality expected, is resource-effective and is easy to maintain. The solution should be compatible with existing software, in the sense

Chapter 3. An Automated GUI Testing Framework

that contamination of TouchGFX and the AUT should be minimized, and it should be easy to transfer to other platforms.

The overall design and host-side solution presented in this chapter utilizes partitioning in order to reduce resource consumption in the target environment. Another benefit of running most of the solution on the host is reducing the amount of code that could be platform-dependent, increasing transferability and maintainability. Maintainability is further increased by the use of interface- and component-based design as well as the Command Pattern, enabling easy extension of the frame- work. Furthermore, the heuristics-based GUI object recognition provides robustness against trivial changes to the GUI, compared to a bitmap-based approach relying on image processing [33]. The use of the Robot Framework provides actionword and keyword-scripting with support for data-driven scripts as well. While this approach to GUI testing may provide a steep learning curve for testers without programming experience, the hierarchies of reusable keywords make the test scripts easy to maintain. With the Robot Framework and its reusable test libraries that can contain utility- or application-specific functions, our solution is capable of providing all of the functionalities necessary of a test automation framework with the addition of easy extension.

CHAPTER

4

Test Automation on Low-Resource

Embedded Systems

This chapter describes the target design of the automated GUI testing framework presented in Chapter 3. The target design enables remote execution of GUI tests on multiple, heterogeneous targets; both physical and simulated. Apart from describing the core target design and integration with the TouchGFX framework, this chapter will present the protocol used for communication between the host framework presented in Chapter 3 and the target framework presented here.

4.1.

Overview

The target design presented in this chapter enables automated execution of GUI tests on TouchGFX- based applications running on remote heterogeneous targets. Following the overall design, the target design completes the GUI testing framework in order to fulfill the thesis goals presented in Chapter 1.

The design presented here is influenced by certain characteristics of the TouchGFX framework, described in Section 2.4. These characteristics are presented in this chapter and used as assump- tions for the enabling GUI framework of the AUT.

The assumptions made for the design are presented in Section 4.2 below. Related work relevant to the target design is presented in Section 4.3. The target design itself is presented in Section 4.4 and subsequently discussed in Section 4.5.

4.2.

Assumptions

The target design presented in Section 4.4 is influenced by the constraints and the design of the GUI framework TouchGFX described in Section 2.4. This has led to the assumptions presented below. The context referred to by the assumptions relates to a runtime context where public meth- ods of the GUI framework can be accessed.

While interfacing to the AUT it is assumed that:

• a public abstraction of the Application instance is available to any procedure running in the same context. This abstraction contains the means to invoke GUI events such as click and drag events on the AUT.