4.2 Results
4.2.2 Changes in Test Suite
A summary of the need for maintenance of scripted (Espresso, UI Automator, Selen- droid) test cases in the transitions between the considered releases is shown in table 4.2. The same is done for Sikuli test cases in table 4.5.
Table 4.3 shows the percentage of tests that, for each release of the application, could be maintained as they were developed for the previously considered release. The same is done for Sikuli test cases in table 4.6.
As it was expected, in correspondence with tangible changes in the GUI (as it happens between the third and fourth major release), the majority of test cases had to be rewritten. On the other hand, in the transition towards the latest releases – which mainly corrected bugs without performing any intervention in the app appearance – no test cases were broken.
This first case study proved that, even for a very small sample test suite for a single application, the occurrence of fragility may require significant intervention (re-writing up to the 75% of scripted test cases of a test suite, and up to the entirety of a visual test suite) in existing test suites. The evolution of the K-9 Mail application was characterized by fragilities in both the definition of the user interface and its appearance. Table 4.4 summarizes the main causes of the fragilities found for the Espresso, UI Automator and Selendroid test suites, with individual test cases that
Table 4.5 Test suite implementation on various versions of K-9 Mail, with Sikuli. Test case v2.102 v2.995 v3.309 v3.993 v4.804 v5.010 Authentication n o o o x o Send a message n o o o x o Reply to a message n o o o x o Delete a message n o o o x o Add user account n x x o x o Delete user account n o o o x o Delete account data n o o x x o Restore account data - n o x x o Export account settings - - - - n o Import account settings - - - - n o
’-’ feature not supported, ’x’ test had to be modified, ’n’ new test written, ’o’ previous version of test still working
Table 4.6 Tests compatible with previous versions, with Sikuli. v2.995 v3.309 v.3993 v.4804 v5.010 Number of unbroken tests 6/7 7/8 6/8 0/8 10/10 Percentage of unbroken tests 85% 87% 75% 0% 100%
could be weakened by multiple different causes. The encountered fragilities were due to the following reasons:
• Changed identifiers: modification of the identifiers that were used by test cases for the identification of interface widgets.
• Changed text: modification of the textual content, when such text is used to identify widgets in test cases (in absence of unique identifiers).
• Deletion or relocation: changes in the arrangements of the widgets in the layouts of the app activities.
• Usage of physical buttons: older Android apps made use of physical buttons, deprecated since Android 4.0 and replaced with the use of Action Bars. New releases of applications after that version of the o.s. needed to change the in- teraction paradigm used accordingly, thus invalidating all test cases developed beforehand with the use of physical buttons.
Chapter 5
Study 1: Survey with mobile
developers from the industry
In order to perform a first investigation about the adoption of automated GUI testing techniques among IT professionals, and to have insights about the testing practices performed by IT professionals having mobile and web apps in their portfolio, a set of semi-structured interviews were conducted with representatives of seven medium- and large-sized companies of the Turin area.
This study allowed answering to the first research question of the study: RQ1 - What is the perception of GUI testing for Android apps among practitioners from the
industry?
5.1
Study design
The interviewed representatives are described in table 5.1. The companies were selected based on the location of the interviewers at the moment of the conduction of the experiment. All the interviewed representatives were involved in the development of mobile or web applications. The interviews have been conducted between June and December 2017.
RQ1 can also be split into three sub-questions, related each to a different topic covered by the whole survey subministrated to the developers. First, to understand the testing practices adopted by developers, insights were gathered about the most
Table 5.1 Interviewed developers from the industry
Interview ID No. of representatives Company and project
A 1 Distributor of testing tools for various typologies of applicatives. B 2 Test factory for third party applications and test consulting.
C 1 Insurance company: web and mobile apps for insurers and customers. D 2 Insurance company: platform for insurance management.
E 1 Test factory for third party applications and test consulting.
F 1 Full-stack development of mobile applications for multiple platforms.
G 2 Test factory for consulting of test and test management for banking applications.
common tools and techniques they adopted, and the typologies and levels of testing performed. Hence, RQ1.1 could be formulated as:
RQ1.1 : Are mobile applications tested by the interviewed sample of industry practitioners? How? To what extent?
As discussed in the introduction section, mobile apps showcase a set of peculiar aspects when it comes to testing them, that must be taken into account by testers/de- velopers wanting to design and execute automated test cases. The second section of the survey aimed at understanding which aspects of mobile applications were considered crucial by professionals adopting automated testing, and at the same time which peculiarities were seen as a deterrent from the adoption of automated testing techniques. Hence, RQ1.2 could be formulated as:
RQ1.2 : What are the most peculiar properties to test in mobile appli- cations according to the interviewed sample of industry practitioners? What aspects of mobile apps discourage them from adopting automated testing?
Finally, questions were asked to characterize the interest in emerging testing techniques, and to summarize the principal difficulties felt by developers, along with the amount of perceived fragility of developed test cases and the needed human labour to maintain existing test suites. Hints were gathered for possible research directions to aid developers and testers from the industry in overcoming those difficulties. Hence, RQ1.3 could be formulated as:
5.1 Study design 49
Table 5.2 Structure of the survey to developers from the industry
RQ Number Question
RQ1.1 1 Do you use to test your application code?
2 If you test your apps, do you take advantage of manual or automated testing techniques? 3 If you use automated testing techniques, which kind of technique do you prefer? Why? 4 What are the typologies of testing you perform?
RQ1.2 5 In your experience, what are the differences that you have found between testing mobile applications and web/desktop applications?
6 Which aspects of mobile applications encourage you to adopt specific testing techniques? 7 Which aspects of mobile applications discourage you from testing them?
RQ1.3 8 What are the main difficulties you encounter in testing mobile applications?
9 Have you ever needed additional effort to keep test suites up to date with the evolution of the application GUI?
10 What is your attitude towards new generations of testing like model-based testing and visual recognition?
11 Which directions should take the academic research on mobile testing?
RQ1.3 : What are the main challenges felt by developers from the in- dustry performing automated testing, and the directions research should take according to them?
To gather answers to the survey, a set of questions arranged in three different groups (each pertaining to one of the three subquestions of RQ1) were subministrated to the interviewed developers/testers. The interview sessions lasted around 30 minutes each, and a transcript was obtained at the end of every session based on the minutes taken during the interview. The findings were cataloged and organized, to answer the individual questions of the survey. All the questions of the survey, detailed in table 5.2, were open. In each interview, the motivation of the study was clearly stated at the beginning, and the definition of testing fragility for mobile applications was provided, according to the formulation defined in the introduction section. Hypotheses were not stated, neither implicitly nor explicitly, at the beginning or during the interviews, in order to avoid any bias.