The studies that are described in the following have taken into consideration a set of testing tools that can be used for testing Android applications. 2nd and 3rdgeneration GUI testing tools were taken into account. The main inclusion criteria for testing tools to be considered was their open-source nature, and, for Layout-based testing ones, the possibility of generating test scripts in a common language, to make the application of size metrics on produced test code possible. Java has been chosen as the common script generation language for the selected tools.
3.2.1
Selected Layout-based testing tools
Regarding the Layout-based testing technique, six testing tools have been selected, in order to cover all the possible features offered by such generation of GUI testing. Information about the six selected Layout-based testing tools, previously reported in [32], is given in the following.
A classification of the main peculiarities of the tools is given in table 3.4. In particular, the table reports the type of testing that can be performed using a given tool (either white box or black box testing); the presence of multi-platform capabilities; the possibility of using the tool for other forms of testing rather than GUI testing; the support to other ways of generating test cases (possibly with add-ons of the basic tool) in addition to manual scripting (i.e., through Capture and Replay and/or Image Recognition); the possibility of testing multiple applications at the same time or to span the GUI of the Android operating system in addition to the application activities.
Based on the characteristics listed in table 3.4, and on similar investigations available in literature [68], the selected sample of six tools is evaluated as represen- tative of the category of Layout-based testing tools. Several other frameworks are
3.2 Selected testing tools for the studies 31
based on the six considered testing tools, and hence can be represented by them: RaceDriver [95] or Barista [39] are based on Espresso; T+ [66] or Fusion [81] build their testing approaches over the UI Automator APIs; Segen [84] (as Appium) works on the Selenium WebDriver framework; many advanced tools for ripping interactions with the GUI of Android apps and recording Capture & Replay tests are based on Robotium (examples are A2T2 [11] and AndroidRipper [12]).
Espresso
The Espresso automation framework is part of the Android Instrumentation Frame- work, and it is the tool officially supported by Android for testing the GUI of a single application, without taking into consideration the host OS. The used approach for the generation of test cases is defined gray-box, meaning that the tester/developer can develop test scripts without knowing the internal implementation of the activities, but needs access at least to the definition of the appearance of the activities, and to the declaration of the internal disposition of the elements of the activities. Coding test scripts of Espresso is typically carried in the Android Studio IDE, hence having full access to the layout and resource files of the application. An expansion of Espresso, namely Espresso Test Recorder, allows the definition of test scripts through the Capture & Replay technique, executing the desired sequence of operations on an Android Virtual Device. Espresso can be used to test native and hybrid applications, leveraging the EspressoWeb extension1.
UI Automator
The UI Automator framework is part of the Android Instrumentation Framework, and it is the tool officially suggested for testing multiple applications, also spanning the GUI of the Android operating system. The tool can also perform operations on the GUI of the operating system, and operate on the system and display settings (e.g., enabling the WiFi, working on the settings for fonts and colors of the screen). As for Espresso, support for the hybrid applications is given, through the possibility of automating WebViews. Tests written with UI Automator are based on the elements of the app that are exposed through its user interface, hence the tool is considered as a black-box testing tool.
Robolectric
Robolectric is a white-box unit testing tool for Android, that allows testing on the Java Virtual Machine directly, without the need (as for all the other testing tools considered) of a real or emulated Android device. The checks and assertions used by Robolectric are mostly at code-level, so it cannot be used for testing the actual appearance of the user interface of an Android app, but at most its definition and instantiation. The optional emulation of a device can be enabled to test interaction with a full Android environment.
Robotium
Robotium is an extension of the JUnit framework for the definition of unit tests of Android apps; it has been very popular among Android developers before the release of Espresso and UI Automator. Tests created with Robotium can either be black-box or white-box, and can be deployed on any kind of Android app (either native, web-based or hybrid), with the limitation of testing a single application at a time and without the possibility of exercising the operating system user interface. An extension, namely Robotium Recorder, allows the creation of test scripts with the Capture & Replay technique.
Selendroid
Selendroid2 is conceived as the Android counterpart of Selenium, a very popular tool for automated testing of web applications. The tool allows the execution of automated test scripts on native, hybrid and web-based applications (thanks to the integration with Selenium WebDriver). The generated test scripts are black box, since the widgets of the application are retrieved without having any access to the source code; the AUT is instrumented through the use of the Android Instrumentation Framework.
3.2 Selected testing tools for the studies 33
Appium
Appium is a testing tool based on Selenium WebDriver and Selendroid, to create black-box tests for both Android and iOS. The test cases can be created through manual scripting, and exported in a series of scripting language (ranging from Python to C# and Java), or with the use of an Image Recognition extension based on the SikuliX libraries.
3.2.2
Selected Visual GUI testing tools
Two different Visual testing tools have been considered for the further steps of this doctoral work: Sikuli and EyeAutomate. Both the considered tools are not specifically developed for working with Android applications, hence they have been used on an emulated device on the screen of the desktop PC. The AVD provided by Android3or the Vysor4tool have been used for that purpose. The main characteristics of the tools are described in the following.
Sikuli
Sikuli5is an open-source image recognition tool, presented originally by Yeh et al. [100] and then no longer maintained in its original branch. The version of the testing tool using for the following work is Raiman’s SikuliX6. The tool is powered by OpenCV for its image recognition functions, and it can run on any desktop computer platform. The supported languages, for the generation of test scripts, are Python, RobotFramework, Ruby, JavaScript and any Java aware programming and scripting language. The tool contains modules for OCR (i.e., Optical Character Recognition) powered by the Tesseract library, and is paired with an IDE that makes a set of basic commands easily reachable for the tester/developer.
3https://developer.android.com/studio/run/managing-avds 4https://www.vysor.io/
5http://www.sikuli.org/ 6http://sikulix.com/
EyeAutomate
EyeAutomate7 is an open-source image-recognition library, developed in Java by Auqtus and subject of several studies in literature about the advantages of Visual testing with respect to Layout based testing techniques, and the feasibility of a Visual testing set up on industrial settings. It comes paired with an IDE, EyeStudio, which has embedded many simple to complex commands that can be performed on the screen to emulate the mouse and keyboard of a desktop pc. The EyeStudio editor can record and save test scripts in plain text format. The pairing algorithms for the provided screenshot is based on pixel and vector-based image recognition, and AI-based features are provided by some of its components.