Modeling Mobile Application Test Platform and Environment: Testing Criteria and Complexity Analysis

(1)

Modeling Mobile Application Test Platform and

Environment: Testing Criteria and Complexity Analysis

Chuanqi Tao

School of Computer Science and Engineering Nanjing University of Science and Technology

Nanjing, Jiangsu, China

Jerry Gao

School of Computer Engineering San Jose State University San Jose, CA 95112, USA

ABSTRACT

With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mo-bile subscriptions. This drives a strong demand on momo-bile application testing on mobile devices. Since mobile APPs are native to mobile devices, an underlying mobile platfor-m becoplatfor-mes the basic foundation of their test environplatfor-ments. To achieve effective test automation, test solutions must be compatible, deployable, and executable on different mobile platforms, devices, network, and appliance APIs. This pa-per is written to provide an approach to modeling mobile test environments based on a Mobile Test Environment Se-mantic Tree (M T EST). Based on this model, the paper discusses test complexity evaluation methods for test envi-ronment. Furthermore, some case study results are reported to demonstrate and analyze the proposed testing models.

Categories and Subject Descriptors

D.2.5 [Software Engineering]: Testing and Debugging

General Terms

Languages, Experimentation

Keywords

test modeling and analysis, mobile testing, mobile test en-vironment, mobile APP testing

1. INTRODUCTION

With the recent fast increase in the number of mobile users, more mobile devices are shipped daily and more mo-bile APPs and applications are deployed on momo-bile devices to meet their needs. According to a Clearwater Technology Team Report in 2011 [1], the mobile computing industry is expected to be worth almost US $330 billion by 2015. Ac-cording to ABI research, the smartphone market is expected to grow at a CAGR of 24% for the period 2011-15, mainly

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

due to rising demand from emerging markets in Asia Pacific and Latin America [2]. The fast growing market and ex-pected increasing revenues drives new a strong demand on developing and testing more mobile APPs and mobile web applications. Up to now, most published research papers primarily focus on the specific technical issues and solution-s solution-such asolution-s white-box and unit tesolution-sting for mobile programsolution-s, black-box testing and GUI testing mobile applications, mo-bile application QoS requirements, momo-bile usability testing, mobile test automation and frameworks, and testing intelli-gent mobile terminals.

Nowadays, diverse mobile test environments and technolo-gy tools cause tedious operations and higher costs in mobile test environment set-up and test automation control. Ac-cording to our survey, the existing test models seldom ad-dress the test modeling and analysis for mobile environment contexts (such as mobile platforms, web browsers, mobile technologies, different native APIs, and device-specific ges-ture, and related configuration on different devices), diverse network connectivity and related contexts, scalability and mobility, usability and security. Thus, engineers lack well-defined test models and criteria to address the special fea-tures in mobile APPs and mobile web applications, as well as test automation solutions for a variety of mobile devices, mobile functions, and mobile environment set-up, i.e., there is a lack of well-defined test models to address distinct needs in mobile testing.

This paper focuses on those needs above. The paper uses a model-based approach to address testing issues. A new

model, known as Mobile Test Environment Semantic

Tree(M T EST), is used to assist engineers to perform test modeling and analysis for mobile test environments. Based on the given model, a test complexity evaluation method is provided. The paper has three primary contributions in mobile environment testing. Firstly, it uses a model-based approach to modeling, presenting and analysis of diverse mo-bile test environments. Secondly, it provides a systematic method to evaluate test complexity of diverse mobile envi-ronment deployments. Thirdly, Two realistic mobile apps are studied using the proposed models.

The paper is structured as follows. The next section presents the model-based approach to mobile test environ-ment. Test complexity analysis is discussed in Section 3. Section 4 reports the results of case studies. The related work is provided in Section 5. Conclusion and future work are summarized in Section 6.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

(2)

Figure 1: Mobile Test Environments

2. MODELING TEST ENVIRONMENT IN

MOBILE TESTING

Since mobile applications are expected to be deployed and executed on diverse mobile platforms, they must be validat-ed on different mobile platforms and devices. In addition, most mobile devices today support diverse wireless network connectivity (such as 2G/3G/4G/Wi-Fi/Wi-Max), mobile applications must be validated under different network con-nectivity and related contexts. Windows, Linux, and Mac are the three popular mobile platforms for mobile testing tools. At the current point, only a few of them have its limi-tation on their execution platforms. For example, Keynote’s MITE only supports Window platform. As shown in Fig-ure 1, there are four different mobile testing approaches and supporting environments [7]. They are such as emulation-based testing, simulation-emulation-based testing, device-emulation-based test-ing, remote device-based testing.

According to the recent feedback from test engineers, there are some major issues and needs in mobile test environments [7]. For example, there is a lack of well-defined test models and coverage criteria to address distinct features and needs in mobile application testing. Since diversity of mobile app operation environments on mobile devices requires a spe-cial test model to address the test coverage for complicated mobile app (or mobile web app) operation contexts. The similar needs can be found in testing mobile scalability, mo-bility, mobile usability and security. Therefore, well-defined test models to address the special features are needed in mobile APPs and mobile web applications. However, mo-bile test environment brings a lot of issues and challenges, such as how to solve the high cost and complexity problem in building mobile test environment, how to meet the demand of diversity of mobile platforms with different device-based gestures, and how to cope with the incompatible mobile plat-forms with limited computing resources. In this paper, we only focus on modeling test environment.

Although there are numerous useful models in software testing, very few are suitable to model and present diverse

Figure 2: Test Space for Mobile Applications

Table 1: The Notations of Semantic Relations in the

M T E−ST

Relations Semantic descriptions

(P is a parent node, andCiis its child node) EOR(P,hC1, C2i) P-Node must be deployed with

two child nodes C1 and C2 exclusively

AN D(P,hC1, ..., Cni) P-Node must be deployed with all of its child nodesC1,...,Cn SELECT−1 P-Node must be deployed with one (P,hC1, ..., Cni) of its selective child nodesC1,...,Cn

SELECT−M P-Node must be deployed with M selective (P,hC1, ..., Cni) nodes from its child nodesC1,...,Cn

deployments or configurations and their mappings in the 3-dimension configuration space for mobile test environment. As shown in Figure 2, we need a well-defined test model to assist engineers to analyze and present each device (sayDj) under a specified configurable environment (sayEk) as well as the corresponding configuration function (sayFi), so that test generation methods, test complexity analysis techniques can be developed.

2.1 Test Model

In previous work [8], we introduced a sematic tree model to test configurable component-based software. Here, the semantics tree model is used as a basis to model to address the needs of the mobile test environment in the 3-dimension space. The tree nodes present configurable parts (or ele-ments), such as deployable platform, network, or APIs. The links present different semantic relations between nodes.

A mobile test environment semantic tree modelM T EST can be formally defined as 3-tuple = (N, E, R), where

• N is a set of tree nodes. There are three types of nodes: a) a single root node, b) intermediate nodes (or parent nodes), and c) leaf nodes.

• E is a set of links between nodes. Each link connects a parent node and one of its child nodes in a tree. Each link show a part of a semantic relation between a parent node and its child nodes.

• R is a set of relations, and each item in R has a se-mantic label that presents a sese-mantic relation between a parent node and its child nodes. There are four types of semantic relations with labels: EOR, AND, SELECT-1, and SELECT-M. Their detailed semantics are given in Table I.

To support the model-based analysis, we introduced a con-cept of semantic spanning trees based on the semantic tree model to present the various configurations. A semantic s-panning treeM T ESP T is a sub-tree of a given semantic tree

(3)

Figure 3: A Sample Semantic Tree and Its Selected Semantic Spanning Trees

M T EST. Unlike regular spanning trees, a semantic span-ning treeM T ESP T forM T EST only can be derived based on the given configuration semantic properties.

Semantic Spanning Tree: A semantic spanning tree

M T ESP T is a sub-tree of a given semantic tree M T EST. Unlike common spanning trees, a semantic spanning tree

M T ESP T for M T EST only can be derived based on the following properties:

-For each parent node (Np) with an AND relation in

M T ESP T, it must include all of its child nodes and its links. -For Np with an EOR relation in v, it must include only one of its child nodes and the corresponding link.

-For Np with aSELECT−1 relation inM T ESP T, it must include only one of its child nodes and the corresponding link.

-For Np with a SELECT −M relation inM T ESP T, it must include only M child nodes and the related links.

As shown in Figure 3, (a) is a semantic tree model, and its two sample spanning trees are shown in (b) and (c). The detailed Algorithm for generating Spanning Tree can be ref-erenced in [8].

Compared to traditional Classification Tree Method (CTE) by Grimm and Grochtmann [9], the proposedM T EST has the following new features in modeling mobile app test en-vironment.

a)M T EST presents semantic relations such as OR, AND, and SELECT-N between parent nodes and its child nodes;

b) A semantic sub-tree M T ESP T can be derived from

M T EST based on the defined semantic properties;

c) Test complexity and criteria can be analyzed effectively according toM T EST.

Therefore, the proposed semantic tree is more suitable for configuration testing in mobile app or component-based software due to the rich semantic relation. In addition, the semantic tree can be utilized in modeling test environment, function, or architecture while the traditional CTE approach primarily focuses on feature-wised function partition testing. Furthermore, we proposed the spanning tree concept, which can present a possible configuration in practice.

2.2 Model Identification and Generation for

Mobile Test Environment

All commercial mobile applications must be executed in

a certain operation environment. Figure 4 shows a sim-ple semantic model examsim-ple which presents different config-urable operating environments for a smartphone A. In the real world, we can use this model to consider all required configurable hardware and software elements (or entities) in a producta´_,rs operation environment. They include different configuration selections in network protocols, device driver-s, diverse operating systems and their versiondriver-s, multimedia and third-party dependent technologies. Mobile app test en-gineers need a semantic tree model to perform test modeling and test complexity of these diverse configuration environ-ments. Since each configurable environment usually requires a set of environment-oriented test scripts to set up so that system function and performance testing can be conducted properly.

It is important to have some systematic way to specify diverse configurations to support test modeling, complexity and coverage analysis. Although there are well-established software analysis and design models, such as UML, they are not suitable to present the diverse configurations in mo-bile applications in terms of environments. The proposed semantic tree model provides an effective modeling tool to support engineers to perform software configuration analysis and specification for configurable mobile applications . The first approach is a static specification-based approach, in which engineers use the semantic tree model to specify and model the configurations. Clearly, when the given software supports complicated configurations, this approach becomes tedious. Therefore, the second approach is more dynamic and systematic one, in which some built-in dynamic con-figuration discovery and tracking capability will be provid-ed in configurable software. With this capability, dynamic configuration decisions in environments can be tracked and analyzed for the purpose of test modeling, test complexity analysis, and test coverage measurement.

3. TEST COMPLEXITY ANALYSIS FOR

MOBILE TEST ENVIRONMENT

The existing research indicates that complexity can be used to estimate the cost or effort required to design, code, test, and maintain software, as well as predict errors or faults that might be encountered during testing [12]. In addition, complexity measurement provides a guideline and cost indi-cator for software maintenance.

High cost and complexity exits in building a mobile test environment for mobile APPs. Fast upgrading mobile plat-forms and diverse native appliances brings higher costs and complexity on building and set-up a desirable test environ-ment for mobile APPs due to the diversity of mobile plat-forms and native APP interfaces with different device-based gestures, the incompatible mobile platforms with limited computing resources, and the lack of reusable test tools for mobile APPs on different platforms.

For any configurable node NCi inNS of M T EST in se-mantic tree model, its test complexity can be computed based on its semantic relation with child nodes. Let T −

complexity(NCi) be the configuration complexity for its ar-chitectures. To support the evaluation of test complexity of diverse configurable environments, we provide a detailed computation method for test complexity below.

• Suppose the nodeNCihaving EOR semantic relation with its child nodes, then it has two different

(4)

architec-Figure 4: Mobile Test Environment Semantic Tree Model

tures, thus, its configuration complexity will be 2.

T−complexity(NCi) = 2 (1)

• Suppose the nodeNCihaving AND semantic relation with its child nodes, then its configuration complexity will be 1.

T−complexity(NCi) = 1 (2)

• Suppose the node NCi having a SELECT-1 seman-tic relation with its child nodes, then its configuration complexity will be n if we assume that a total number of environments n are allowed to be selected.

T−complexity(NCi) =n (3)

• Suppose the nodeNCihaving a SELECT-M semantic relation with its child nodes. Assuming each environ-ment can be configured with selected m nodes from a total of n nodes, its configuration complexity will be n!/(m!(n-m)!) .

T−complexity(NCi) =n!/(m!(n−m)!) (4)

4. CASE STUDY

We report our case study by applying the proposed test-ing approach and complexity analysis into several realistic mobile applications. we have selected the yelp application and justWink for the mobile testing. Yelp is an online guide that searches businesses near you. The current release of the yelp is the hybrid application for mobiles. Hybrid ap-plication can be defined as the combination of the native and the mobile web application. In this study we will select some deployable environments for the yelp application and test those features to verify their correctness.

Configuration testing of the native mobile application YELP here involves testing the application on various operating systems of the mobile with various connectivity. The oper-ating systems on which we tested the application are such as Android, IOS, and Windows. The networks that are be-ing considered are WAN (Wide Area network). The various types of Wide Area Networks are 2G, 3G, and 4G. The Wire-less Internet is WiFi. Network connection depends on the

carrier which user has opted, In our case we are consider-ing T-Mobile Network and wifi-internet access. The third party API used by the app are android, Google maps API, Hardware Management API. The semantic tree model also provided us with an effective tool to analyze justWink ap-plication. We have considered four features in our project test environment - mobile devices, platform , connectivity and API.

We have used two software testing classes and three mas-ter project teams to conduct the related experiments in San Jose State University, California, USA. In this study, we primarily focus on the following items:

• Model mobile applications using the proposed approach based on the mobile test environment semantic tree model (M T EST).

• Identify and analyze the test complexity of test envi-ronment in mobile testing.

4.1 Study Results and Discussion

Figure 5 shows the sample semantic tree model for just-Wink mobile app. The tree model has 18 nodes, among of which there are 11 leaves. In addition, there are two d-ifferent types of semantic relations, including AND, EOR, and SELECT-1. The test complexity value is presented on top of nodes. Figure 6 presents one spanning tree from the semantic tree model shown in Figure 5. The spanning tree represents one mobile test environment configuration, which means justWink app is configured with Android 4.3 platfor-m, wifi network connectivity, and both Camera and Speech APIs.

Table 2 presents the detailed complexity of the semantic tree model of the studied moble app Yelp and justWink. For example, the total test complexity for Yelp app is 32 and 18 for justWink, which presents the total number of d-ifferent configured and deployed environments for Yelp and justWink. Hence, while validating this software, a vendor’s engineers must test its deployed instances to cover its con-figurable environments. In practice, they can achieve the adequate test criteria in an incremental approach. For ex-ample, whenever a customer is deployed one instance, its configured environment (or platform) will be recorded. The

(5)

Table 2: The Semantic Tree and Its Complexity Results

Semantic Tree for No. of No. of No. of Max No. of No. of No. of

Testing Environment Nodes Leaves Links Height AND Select-1 Spanning Trees

Yelp Application 17 12 18 4 1 6 32

justWink 18 11 17 3 2 2 18

Figure 5: AM T EST Sample for justWink and Test

Complexity Analysis

Figure 6: A Spanning Tree derived from Semantic Tree

complexity analysis enables to engineers to figure out the required number of pre-test scripts for environment config-uration and set-up. This will be useful for test planning in test cost and complexity analysis. For theses two appli-cations in the study, we found that we need to develop 32 scripts for Yelp app and 18 scripts for justWink app to set up and cover different environments so that the deployed system instance can be tested with certain adequate test set using the existing test methods.

4.2 Threats to Validity

There are several potential threats to the case study. We selected two apps in the case study, that is not adequate for large-scale empirical validation. More apps are needed to indicate the effectiveness of our approach. The proposed measurement here is not the only possible measurement for testing complexity. Measurement factors such as human cog-nitive complexity, manual analysis complexity are not

con-sidered in this paper. In addition, in real-life projects testing on all mobile platforms in all system versions is not applica-ble. Testing of an Android app on all system versions and hardware (smartphone, tables from all vendors) is very time consuming. Clearly, test complexity grows much higher due to more choices are given in the semantic tree. This suggests the mobile app test environment could be very complicat-ed. More test automation research work for mobile testing is needed.

5. RELATED WORK

Up to today, many papers have been published to address different testing areas in mobile applications.

White-Box Testing Techniques: Existing white-box test-ing methods are still applicable to mobile applications. For example, the authors in [11] present an Android applica-tion verificaapplica-tion tool built on Java Pathfinder to perform white-box mobile Java program testing so that race condi-tions and deadlocks can be detected using UML state charts and symbolic execution. Mahmood et al. [15] used a white-box approach to generate test cases based on two program-based models (Call Graph Model and Architectural Model) to achieve mobile code test coverage.

Black-Box Testing Techniques: Many black-box testing techniques are useful in mobile application testing. Random testing is one example. The scenario-based testing method is another example [13]. In addition, some papers discuss how to use GUI-based testing techniques for mobile applications. For instance, Saswat Anand et al. in [17] discussed an au-tomated concolic testing approach to validating mobile GUI event sequences in smartphone applications. Similarly, D. Amalfitano et al. in [5] presented AndroidRipper, which us-es an automated GUI-based technique to tus-est Android apps with a structured manner. In addition, some researchers fo-cused on Usability Testing [14], Testing Quality-of-Service (QoS) [16], and Wireless connectivity testing [3, 18].

Mobile Test Automation and Frameworks: Some research efforts are dedicated to develop tools (or frameworks) ad-dress some limits in current tools. For example, JPFAN-DROID in [11] is a verification tool supporting white-box testing for mobile application. And JeBUTi/ME discussed in [15] is a tool supporting white-box test coverage analysis based on a conventional CFG-based test model. A few of re-cent research papers focus on GUI-based testing using test scripts and GUI event-flow models [15, 17]. In addition, a few research tools at the system level are proposed. One of them is the integrated test automation framework [10]. With this framework, high-level test cases can be executed on different mobile platforms (such as Android and iPhone). Another example is MoViT [6], which is a distributed soft-ware suite for the emulation of mobile wireless networks.

Existing black-box and white-box test models and cover-age criteria can be used in mobile applications to address mobile program structures, dynamic behaviors, and GUI

(6)

operation flows. However, engineer still need new test mod-els to address special needs in testing mobile applications. The existing test models seldom address the test modeling and criteria for mobile environment contexts (such as mo-bile platforms, web browsers, momo-bile technologies, different native APIs, and device-specific gesture, and related config-uration on different devices), diverse network connectivity and related contexts, scalability and mobility, usability and security. Unlike the existing research, this paper provides a configuration model to present the various APP deployed platform and environments. In addition, we use a model-based approach to address the testing issues in modeling mobile applications, including test modeling and test com-plexity analysis.

6. CONCLUSIONS

According to the latest study from Juniper Research, the market for cloud-based mobile applications will grow 88% from 2009 to 2014. We believe, this brings the strong de-mand on new research results and mobile test automation solutions to cope with the discussed issues and challenges. Although there are numerous papers addressing how to con-struct configurable software and components [4, 19], few papers discussed how to test configuration features in mo-bile applications, especially for momo-bile test environment.

This paper uses model-based approach to discuss the re-lating issues, challenges, and test process. It applied a se-mantic tree model as a test model to present and analyze the diverse configurable and deployable environments in mobile applications. In addition, the detailed test criteria analy-sis and complexity computation is presented. Furthermore, some case study results are reported to demonstrate its effec-tiveness and application in test modeling and test complex-ity analysis. Currently, we are developing a test automation solution to support automatic mobile test environment de-ployment. The future extension of this research is to study how to use a model-based approach to addressing testing issues and challenges in mobile applications in cloud-based and service-based background.

7. ACKNOWLEDGMENTS

This work is supported partially by the National Natu-ral Science Foundation of China No. 61202003, partially by the Specialized Research Fund for the Doctoral Program of Higher Education No. 20113219120021, and partially by Fujitsu Labs. We also thank the students of SJSU’s CMPE 287 course who participated in our study, and the support of Computer Engineering Department in San Jose State U-niversity of California.

8. REFERENCES

[1] http://www.clearwatercf.com/documents/library. [2] https://www.abiresearch.com/press/200−million−

mobile−application−testing−market−boos. [3] T. P. Akka and M. Palola. Towards automating

testing of communicational b3g applications. In International Conference on Mobile Technology,

Applications Systems, 2006.

[4] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The aetg system: An approach to testing based on combinatorial design.IEEE Transactions on

Software Engineering, 23(7):437–444, 1997.

[5] D. Amalfitano, et al. Using gui ripping for automated testing of android applications. InIEEE International

Conference on Automated Software Engineering, 2012.

[6] E. Giordano, et al. Movit: the mobile network virtualized testbed. InACM International Workshop on Vehicular Inter-networking, Systems, and

Applications, 2012.

[7] J. Gao, X. Bai, W. T. Tsai, and T. Uehara. Mobile application testing: a tutorial.IEEE Computer Special

Issue on Software Validation, pages 26–35, 2014.

[8] J. Gao, J. Guan, A. Ma, C. Q. Tao, X. Y. Bai, and D. C. Kung. Testing configurable component-based software-configuration test modeling and complexity analysis. InInternational Conference on Software

Engineering and Knowledge, pages 495–502, 2011.

[9] G.Matthias and G. Klaus. Classification trees for partition testing.Software Testing, Verification and

Reliability, 3(2):63´lC82, 1993.

[10] H. Song, et al. An integrated test automation framework for testing on heterogeneous mobile platforms. InInternational Symposium on Software

and Network Engineering, 2011.

[11] H. V. D. Merwe, et al. Verifying android applications using java pathfinde.ACM SIGSOFT Software

Engineering Notes, 37(6):1–5, 2012.

[12] A. E. Hassan. Predicting faults using the complexity of code changes. InInternational Conference on

Software Engineering, pages 78–88, 2009.

[13] J. Bo, et al. Mobiletest: a tool supporting automatic black box test for software on smart mobile devices. In International Workshop on Automation of Software Test, 2007.

[14] T. Kallio and A. Kaikkonen. Usability testing of mobile applications: A comparison between laboratory and field testing.Journal of Usability studies,

1(1):4–16, 2005.

[15] R. Mahmood, et al. A white-box approach for automated security testing of android applications on the cloud. InInternational Workshop on Automation

of Software Test, 2012.

[16] R. Mizouni, et al. Performance evaluation of mobile web services. InIEEE European Conference on Web

Service, 2011.

[17] S. Anand, et al. Automated concolic testing of smartphone apps. InACM SIGSOFT International Symposium on the Foundations of Software

Engineering, 2012.

[18] I. Satoh. Software testing for wireless mobile computing.IEEE Wireless Communications, 11(5):58–64, 2004.

[19] D. B. Stewart, R. A. Volpe, and P. K. Khosla. Design of dynamically reconfigurable real-time software using port-based objects.IEEE Transactions on Software