Motivating Example: a test script for K-9 Mail

A motivating example of the possibility of repairing layout-based or visual fragilities of an existing test suite through translation has been created with K-9 Mail (version V5.500-Snapshot).

In the following discussion, the original version of the application – as cloned from the GitHub repository – is called v1. A sample test case has been developed, to exercise the authentication feature of the application with two tools pertaining to different generations: Espresso for the generation of Layout-based testing tools, and EyeAutomate for the generation of Visual testing tools. All the tests have been run on an Android Virtual Device (AVD), namely a Nexus 5 with Android API 24 installed, with enabled device frame and hardware keyboard inputs.

The considered use case is the wizard that is started at the first launch of K-9 Mail. Table 9.1 shows the steps of the use case, whilst figure 9.1 shows the four screens that are traversed by the use case, with the respective Activity names indicated in the captions. The use case traverses a first WelcomeMessage Activity; in the second Activity encountered, AccountSetupBasics, the user inputs his/her e-mail address and

Table 9.1 TOGGLE motivating example: Steps for the Authentication use case of K-9 mail

Step Screen Widget Description Operation

1 s1 Next Button Click

2 s2 Email account Form Type test account email

3 s2 Password Form Type test password

4 s2 Next Button Click

5 s3 Account description Form Type test account description 6 s3 Account name Form Type test account name

7 s3 Done Button Click

8 s4 Activity Title Check that "Accounts" is shown 9 s4 Account List Item Check that test name is shown

Table 9.2 TOGGLE motivating example: Retrieved IDs for Layout-based test case

Step Object Description Object ID

1 Next Button next

2 Email account Form account_email

3 Password Form account_password

4 Next Button next

5 Account description Form account_description

6 Account name Form account_name

7 Done Button done

8 Activity Title action_bar_title_first 9 Account List Item description

9.1 Motivation 123 (a) WelcomeMessage Activity (b) AccountSetupBa- sics Activity (c) AccountSetup- Namesb Activity (d) Messages Activ- ity

Fig. 9.1 TOGGLE motivating example: Screens and Activities traversed by the authentication use case

password; after going (through the use of the Next button) to the AccountSetupName Activity, the user inputs his/her desired account name and nickname for managing the e-mail account in the K-9 mail client; finally, the app shows the Messages activity (which is initially empty, if no message has not been downloaded yet by the client).

Test scripts definition for the original release

The Layout-based test script for the Authentication test script has been developed using the Android Studio IDE, and inside the K-9 Mail application project, having full access to the AUT production code and the .xml layout files describing its user interface. The resource IDs used in the Layout-based test script have been collected by launching the application and using the UI Automator Viewer tool, retrieving the "resource-id" field for any of the widgets that had to be interacted. Table 9.2 shows the retrieved ids for all the elements interacted throughout the execution of the test case. The test script, when executed on v1, runs to completion.

The test script has been developed also leveraging the Visual testing technique, with the EyeAutomate testing tool. Table 9.3 shows the retrieved images that were used as visual locator for executing the same test case on the app. All the reference images were gathered from a first manual execution of the use case on the emulated application, leveraging the image capturing tool embedded in the EyeStudio suite. The interaction points inside the visual locators were fixed to the center of the images

Table 9.3 TOGGLE motivating example: Retrieved images for the EyeAutomate test script

Layout excerpt from v1

Layout excerpt from v2a

Fig. 9.2 TOGGLE motivating example: Modification in the layout file between v1 and v2a

(the default location of the interaction for each screen taken with the image capturing tool). Sleep instructions have been added at every screen transition to prevent the visual testing tool to search for elements that were not yet been rendered on screen. Executed on v1, the Visual test runs to completion.

Layout-based fragility induction

To highlight the fragility of layout-based test scripts to changes in the properties of the interacted widgets, a simple modification on the definition of a layout of v1 of the considered app has been performed. This way, the version that was called v2a is obtained. In particular, the resource ID associated with the Done button is changed

9.1 Motivation 125

from ”done” to ”completed” in the layout file (namely, wizard_done.xml) where the button is declared. The modification performed in the layout file is shown in figure 9.2.

In the considered Layout-based test scripts, the Resource IDs are inserted as constants inside the test script. If they are changed externally by a modification of the layout file (and no automated refactoring tools are applied to the application project) manual effort is required to fix the changed IDs in the test script. The same applies to any change in the textual properties of the widgets, like the contained text or the textual description of a button.

In all transitions like the one from v1 to v2a, containing only modifications in the widget definition and properties, all test cases leveraging Layout-based properties and locators will fail and require maintenance from testers/developers. In the provided example, according to the taxonomy of modifications provided in the previous chapters, the test case fails due to an ID change fragility.

On the other hand, the Visual test runs to completion without errors, since no graphical property has changed for any widget in the screens traversed by the tested use case. Using the translational approach, the Visual test script, which is still valid, can be used to retrieve the actual (changed) id of the Done button when it is clicked, generating a valid companion layout-based test case.

Visual fragility induction

To cause fragility in the visual test script, we performed a couple of graphic modifi- cations on the original version v1, leading to the version called v2b. In particular, the appearance of Screen 3 was changed, modifying the background color and the text of the "Done" button. Each of the two modifications performed would be sufficient, if applied alone, to invalidate a visual test case. The changed appearance of the screen between v1 and v2b is shown in figure 6.3.

Without any modification in layout and widget definitions, there is no impact in scripted test cases, since the button to be clicked is unambiguously retrieved by its unchanged ID. On the other hand, the visual recognition test case that we developed experiences a failure, due to the inability to identify an element with the appearance of the Done button, still linked in the test script to the screen capture shown in table 9.3.

(a) Screen 3 in v1

(b) Screen 3 in v2b

Fig. 9.3 TOGGLE motivating example: Graphic changes in Screen 3 between v1 and v2b

In the envisioned translational approach, using the coordinates of the interaction and possibly additional information extracted from layout files (e.g., size and padding of the interacted widget), a picture of the new appearance of the interacted widget can be cut from the capture of the whole current screen. Hence, a new working Visual test script can be obtained.