Mechanisms to support automated testing of mobile
applications
Guilherme de Cleva Farto1,2,3, André Takeshi Endo1 1Universidade Tecnológica Federal do Paraná (UTFPR)
Avenida Alberto Carazai, 1640 – 86300-000 – Cornélio Procópio, PR – Brasil 2Fundação Educacional do Município de Assis (FEMA)
Avenida Getúlio Vargas, 1200 – 19807-635 – Assis, SP – Brasil 3TOTVS Agroindústria
Rua Prudente de Moraes, 654 – 19806-160 – Assis, SP – Brasil
guilherme.farto@gmail.com, andreendo@utfpr.edu.br
Abstract. Due to the high number and diversity of users, new testing approaches are necessary to reduce the occurrence of faults and ensure better quality in mobile applications. The major objective of this project is to propose mechanisms to support automated testing of mobile applications with an emphasis on solutions developed for the Android platform. Specifically, we will investigate the application of the Model-Based Testing (MBT) in order to validate and improve the reliability of Android applications. The proposed approach and tool will be evaluated in an industrial configuration with professional developers of mobile applications.
1.
Problem Characterization
Currently, there is a rapid growth in popularity of mobile devices, such as tablets, smartphones, and e-readers, expanding the variety of applications that are developed with the support of mobile computing technologies. A 2013 survey on sales of mobile devices reports that the Android platform had an increase of 127% over the previous year with approximately 121 million units sold [Gartner 2014]. Thus, Android took the lead, with 62% of systems and development environments for mobile applications, well ahead of Apple’s iOS and Microsoft’s Windows Phone.
While mobile applications were initially developed for the entertainment industry, there is a more widespread adoption in critical areas (e.g., financial systems, health care, and industries) [Muccini et al. 2012]. As a large number and diversity of users as well as critical systems have benefited from the mobility provided by these applications, the occurrence of faults can result in human and economic loss. In this context, software testing has been applied during the development process to minimize the occurrence of faults. The activity of testing consists of designing test cases, executing the software with these test cases, and examining the results with the central goal of detecting faults [Harrold 2000].
The mobile application testing provides many challenges to be overcome and has received special attention from Software Engineering and Mobile Computing community [Delamaro et al. 2006, Bo et al. 2007, Maji et al. 2010, Hu and Neamtiu 2011, Muccini et al. 2012, Pathak et al. 2012, Yang et al. 2013, Liu et al. 2014]. Muccini et al. [2012]
argue that there is a need for approaches specialized in testing of mobile applications. They identified characteristics of mobile applications that influence the testing activity, such as connectivity, limited resources, autonomy, user interface, context awareness, adaptation, new programming languages and operating systems, diversity of settings, and touch screens.
The dynamics of mobile applications and their use in critical environments demand more accurate and repeatable tests. These characteristics can be obtained by the application of formal approaches and automated tests. Formal testing is characterized by the adoption of mathematical models to support the testing activity. Examples of formal models are Finite State Machines (FSMs), Labelled Transition Systems (LTS), and Event Sequence Graphs (ESGs) [Belli et al. 2006]. According to Hierons et al. [2009], the presence of formal specifications and designs can lead to more efficient and effective testing. These models also provide benefits to automation since their syntax and semantics are well-defined and allow the development of tools to manipulate such models. In this context, this project aims to investigate mechanisms to support automated tests for mobile applications. Specifically, we plan the development of an approach and a supporting tool based on concepts of model-based testing and on the Android platform.
2.
Background
Several techniques can be applied during the development process in order to reveal faults in the software artifacts. One of these techniques proposes the automatic generation of test cases through a behavioral or structural model, named test model, of the software under test (SUT); this approach is known as Model-Based Testing (MBT). The MBT process can be more efficient because the tester can update the model and regenerate the test suite, avoiding manual and error-prone changes [Utting and Legeard 2006]. The literature of MBT reports a set of benefits resulting from its appropriate adoption, such as high fault detection rate, reduced cost and time for testing, supporting of the requirements’ evolution, and a high level of automation [Grieskamp et al. 2011].
Pieces of research on the MBT process suggest four main steps: (i) modeling, (ii)
test generation, (iii) concretization, and (iv) test execution [Pretschner and Philipps 2004, El-Far and Whittaker 2001, Bouquet et al. 2006]. In modeling, the tester uses her/his understanding of the software to design a model focused on testing. It is advisable the use of requirements as the main source of information in order to maximize the independence between the model and the SUT [Utting and Legeard 2006]. In test generation, the algorithm to derive test cases from the model depends on the modeling technique adopted. According to El-Far e Whittaker [2001], modeling techniques should have properties that make both less costly test generation and easier automation. This step requires a tool to receive the test model as input and automatically generate a set of test cases as output. The generated test cases are abstract and not executable because they are in a different level of abstraction of the SUT. In concretization, the tester provides means to transform abstract test cases into ones executable in the SUT. To do so, abstract test cases could be implemented using adapters [Pretschner and Philipps 2004]. In MBT, an adapter is a software component that enables the execution of abstract test cases. In test execution, the concretized test cases are executed in the SUT. The results are analyzed and corrective actions can be taken if faults are revealed.
Event Sequence Graph (ESG). In this project, the expected behavior of the SUT is going to be modeled by Event Sequence Graphs (ESGs). The choice of ESG as the test
modeling technique is backed up by ease of use and successful experience on MBT. According to Belli et al. [2006], an ESG is a directed graph used to model possible interactions between the events of the SUT and is formed by nodes that represent events and by edges that represent valid sequences between these events. The ESG technique can be learned in a short period, requires little manual work, and is supported by specific tools [Belli et al. 2006]. Figure 1 illustrates an ESG model for the “cut-copy-paste” procedure. The brackets represent the beginning and end of event sequences.
Figure 1. ESG for a “cut-copy-paste” procedure – adapted from [Endo 2013].
3.
Related Work
The emergence of technologies and platforms for software expands or creates new fields of study, despite the existing researches on software testing [Muccini et al. 2012]. Thus, the mobile applications introduce new testing challenges that must be overcome and, as a consequence, have been investigated by researchers of software engineering and mobility. These studies can be divided into two lines. In the first line, traditional testing techniques have been adapted to mobile applications. Delamaro et al. [2006] describe a strategy to support structural testing of mobile applications and enable test execution through emulators and physical devices in the Java Micro Edition (JME) platform. In functional testing, Bo et al. [2007] implement a tool, named MobileTest, to automate the black-box testing from an event-based approach to simplify and improve the design of test cases. Its functionalities include the record of sensitive events, a schedule mechanism for regression testing, and adapters for future devices to be added.
The second line investigates faults characteristic of mobile applications and, based on them, new testing strategies are proposed. Maji et al. [2010] evaluate the reported failures in Symbian and Android platforms resulting in: a detailed analysis of faults found, a characterization of corrections made, and a comparison between the two operating systems. Neamtiu and Hu [2011] describe an approach to test Android applications, emphasizing the user interface’s faults. Random testing, instrumentation of virtual machine, and log analysis were employed. Pathak et al. [2012] investigate software faults related to excessive energy consumption on smartphones running Android and provide an automatic solution to detect these problems using a data flow analysis algorithm. Yang et al. [2013] propose a testing technique to identify and quantify faults related to excessive waiting times for certain events in Android applications. For this, the authors rely on the artificial insertion of delay instructions in typical problematic operations. Finally, Liu et al. [2014] characterize a set of performance faults commonly identified in Android mobile applications. The paper also presents a source code analyzer to detect the identified performance fault patterns in Android applications.
Notice that several researchers have investigated techniques for mobile application testing, highlighting the most notorious adoption of the Google Android platform. So far, few efforts have been spent to evaluate mechanisms for test automation in mobile applications. Moreover, most of the papers are academic and there is a lack of research on applying automated testing in industrial configurations. As mobile applications pose challenges to software testing, many topics can be further explored.
4.
Project Planning and its Current State
This section shows the activities planned to achieve the research project’s goal; its current state is also presented. The activities are briefly described as follows.
1. Study of the Android platform: this activity aims to study the Android operating system and how applications are developed for the given platform. In particular, we will emphasize the tool named Android Studio. Android Studio is an integrated development environment that provides mechanisms to design, code, debug and test Android applications using emulators and different devices. Existing open source applications will also be studied and collected as examples or case studies for future evaluations.
2. Study of the mobile application testing: this activity aims to understand the challenges and limitations faced by testers during the development of mobile applications. Another point is to identify tools that automate the testing of mobile applications, specifically for Android. A starting point is the testing capabilities already provided by Android Studio. Another interesting tool is Robotium [2014], which is a framework for automating functional GUI testing in Android applications. Native capabilities for Android test automation as Instrumentation and MonkeyRunner may also be studied [Android Testing Fundamentals 2014]. Finally, a detailed literature review about mobile application testing will be performed.
3. Conduction of an exploratory study: based on the elicited knowledge in previous activities, we will conduct a first exploratory study to investigate the application of MBT in mobile applications for the Android platform. This study aims to identify tools, lessons learned, and limitations to be considered in the next activities.
4. Proposal of a testing approach: a model-based testing approach will be proposed to verify mobile applications, taking into account the results obtained in previous activities. The purposes of the approach are to maximize the fault detection rate, minimize the testing rework, and reduce the test execution time. It is expected to maximize the fault detection rate because it becomes possible to create and reuse test models to evaluate the SUT on different configurations of mobile devices. The testing rework will be minimized because the models can be updated and the test suite regenerated.
5. Development of supporting tools: this activity aims to develop supporting tools for the proposed approach. Existing tools can also be modified to integrate with the developed tool. A possible candidate would be the Astah tool; plug-ins could be developed to simplify the modeling and test generation steps.
6. Case studies in an industrial setting: the evaluation of software testing approaches basically involves two aspects: cost and effectiveness. We should consider the most appropriate metrics to evaluate these aspects, e.g., the time elapsed to design and implement the tests. The guidelines proposed in the literature [Wohlin et al. 2000, Kitchenham et al. 2002] will guide the planning, execution, and analysis of experimental
studies performed. Initially, we plan to evaluate our proposal by conducting case studies in industrial settings with real-world applications and experienced professionals.
7. Elaboration of scientific papers: this activity aims to report the findings of this master’s project as scientific papers that will be submitted to journals and conferences of the software engineering and mobility areas.
Current State of the Project. So far, we have worked on Activities 1, 2, and 3, described previously. A summary of Activity 2 is shown in Section 3. As Activities 1 and 2 involve literature review, future updates along the project are still necessary. Activity 3 was also performed. In summary, we evaluated the adoption of MBT concepts and modeling with ESG in the testing of Android applications. The preliminary results give evidences of the feasibility of TBM to verify mobility solutions with Android, observing automatic generation and execution of test cases, ability to detect faults, and reduced time for regression tests. This study and obtained results have been compiled in a submitted paper. Currently, the master’s candidate is working on Activities 4 and 5. Particularly, the result of Activities 1-4 is planned to be presented as a master’s proposal in December 2014.
5.
Expected Results
This section presents the results and benefits expected from this master’s project. The desired results are described as follows: (i)Mechanisms to support automated testing of mobile applications on Android: this result will be an approach based on MBT to verify mobile applications developed on Android platform. (ii) Supporting tools: testing tools will be developed and/or adapted to support the proposed approach. Furthermore, training materials will be prepared to use the tools. (iii)Experimental evaluation: case studies will be conducted in order to evaluate the applicability of the proposed approach, providing evidence on cost and effectiveness, relevant to industrial adoption. (iv) Technology transfer: we expect to transfer the technology developed to software companies that develop Android applications.
We expect that the results bring the following benefits: (i) Strengthen the software engineering area in the master’s program, encouraging the transfer of technology to industry and disseminating the acquired knowledge through papers; (ii) collaboration with other research centers; and (iii) cooperation with technology incubators to implement software testing best practices in incubated companies that develop mobile applications.
References
Android Testing Fundamentals (2014) “Android Testing Framework”, available at
http://developer.android.com/tools/testing/testing_android.html, June.
Belli, F., Budnik, C. J. and White, L. (2006) “Event-based modelling, analysis and testing of user interactions: approach and case study”, Software Testing, Verification & Reliability, v. 16, n. 1, pages 3–32.
Bo, J., Xiang, L. and Xiaopeng, G. (2007) “MobileTest: A Tool Supporting Automatic Black Box Test for Software on Smart Mobile Devices”, In: Proc. of the Second Int. Workshop on Automation of Software Test (AST).
Bouquet, F., Debricon, S., Legeard, B. and Nicolet, J. B. (2006) “Extending the unified process with model-based testing”, In: 3rd Int. Workshop on Model Development, Validation and Verification (MoDeVa), Genova, Italy, pages 2-15.
Delamaro, M. E., Vincenzi, A. M. R. and Maldonado J. C. (2006) “A strategy to perform coverage testing of mobile applications”, In: Proc. of the 2006 Int. Workshop on Automation of Software Test (AST), pages 118–124.
El-Far, I. K. and Whittaker, J. A. (2001) “Model-based software testing”, In: Encyclopedia on Software Engineering, Wiley, pages 825-837.
Endo, A. T. (2013) “Model-based testing of service oriented applications”, PhD dissertation,
ICMC/USP, São Carlos, SP, available at
http://www.teses.usp.br/teses/disponiveis/55/55134/tde-20062013-140259.
Gartner, Inc. Press release (2014), available at: http://www.gartner.com/newsroom/id/2674215. Grieskamp, W., Kicillof, N., Stobie, K. and Braberman, V. A. (2011) “Model-based quality
assurance of protocol documentation: tools and methodology”, Software Testing, Verification and Reliability, v. 21, n. 1, pages 55-71.
Harrold, M. J. (2000) “Testing: A Roadmap”, In: Proceedings of the Conference on The Future of Software Engineering (ICSE), pages 61-72.
Hierons, R. M., Bogdanov, K., Bowen, J. P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Harman, M., Kapoor, K., Krause, P., Lüttgen, G., Simons, A. J. H., Vilkomr, S., Woodward, M. R. and Zedan, H. (2009) “Using formal specifications to support testing”, ACM Computing Surveys (CSUR), v. 41, n. 2, pages 1-76.
Hu, C. and Neamtiu, I. (2011) “Automating GUI Testing for Android Applications”, In: Proc. of the 6th Int. Workshop on Automation of Software Test (AST), pages 77–83.
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., Emam, K. E. and Rosenberg, J. (2002) “Preliminary guidelines for empirical research in software engineering”, IEEE Transactions on Software Engineering, v.28, 721–734.
Liu, Y., Xu, C. and Cheung, S.C. (2014) “Characterizing and Detecting Performance Bugs for Smartphone Applications”, In: The 36th International Conference on Software Engineering (ICSE), Hyderabad, India.
Maji, A. K., Hao, K., Sultana, S. and Bagchi, S. (2010) “Characterizing Failures in Mobile OSes: A Case Study with Android and Symbian”, In: The International Symposium on Software Reliability Engineering (ISSRE), pages 249–258.
Muccini, H., Di Francesco, A. and Esposito, P. (2012) “Software testing of mobile applications: Challenges and future research directions”, In: The 7th International Workshop on Automation of Software Test (AST), IEEE.
Pathak, A., Jindal, A., Hu, Y. C. and Midkiff, S.P. (2012) “What is keeping my phone awake? Characterizing and detecting no-sleep energy bugs in smartphone apps”, In: The Int’l Conf. Mobile Systems, App’s, and Services (MobiSys), pages 267-280.
Pretschner, A. and Philipps, J. (2004) “Methodological issues in model-based testing”, In: Model-Based Testing of Reactive Systems, LNCS, pages 281-291.
Robotium (2014) “Robotium – The world’s leading Android test automation framework”, available at https://code.google.com/p/robotium/, April.
Utting, M. and Legeard, B. (2006) “Practical model-based testing: A tools approach”, San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.
Wohlin, C., Runeson, P., Hoest, M., Ohlsson, M. C., Regnell, B. and Wesslon, A. (2000) “Experimentation in Software Engineering: an Introduction”, Kluwer.
Yang, S., Yan, D. and Routev, A. (2013) “Testing for poor responsiveness in Android applications”, In: Proc. Int’l Workshop on the Engineering of Mobile-Enabled Systems (MOBS), pages 10-20.
![Figure 1. ESG for a “cut-copy-paste” procedure – adapted from [Endo 2013].](https://thumb-us.123doks.com/thumbv2/123dok_us/397052.2544417/3.893.250.640.299.552/figure-esg-cut-copy-paste-procedure-adapted-endo.webp)
