Mobile Testing-as-a-Service (Mobile TaaS or MTaaS)

55  Download (0)

Full text

(1)

Mobile Testing-as-a-Service

(Mobile TaaS or MTaaS)

Jerry Gao, Ph.D. Professor

Computer Engineering Department

San Jose State University

W. T. Tsai, Ph.D. Professor

Sch Compt Infor & Dec Sys Engr

(2)

Table of Contents

Introduction

What is Mobile TaaS?

Why is Mobile TaaS Important?

Mobile Testing Process

Major Players and Tools

Mobile Testing Approaches, Infrastructures, and Comparisons

Mobile TaaS Approaches, Infrastructures, and Comparisons

Expected Benefits and Features

Test Models and Test Coverage Criteria

User View of TaaS

Cloud Computing View of TaaS

(3)

According to the latest study from Juniper Research, the market for cloud-based mobile applications will

grow

88%

from 2009 to 2014

. The market was just over $400 million this past year, says Juniper, but by 2014 it will reach

$9.5 billion

. Driving this growth will be the adoption of the new web standard HTML5, increased mobile broadband coverage and the need for always-on collaborative services for the enterprise.

(4)
(5)

Knowledge Background for Mobile TaaS Research

Cloud

Computing

& SaaS

Engineering

Mobile

Computing

Software

Testing and

Test

Automation

(6)

What Is Mobile TaaS?

No existing definition for Mobile TaaS.

Our

first definition

: (Mobile TaaS in a cloud infrastructure):

“Mobile Testing as a Service (known as Mobile TaaS) provides on-demand testing

services for mobile applications and SaaS to support software validation and

quality engineering processes by leveraging a cloud-based scalable mobile testing

environment to assure pre-defined given service-level-agreements (SLAs).

Mobile TaaS on clouds offers a new business model for diverse mobile software

validation services using the pay-as-you-test model to achieve cost-sharing and

cost-reduction in mobile computing resources, networks, cloud computing and

storage infrastructures.”

(7)

Mobile SaaS Features

Mobile

SaaS

On-Demand Function

Services with APIs

Searchable, Discoverable,

Deployable, and Executable

Multi-Tenancy and

Elastic Scalability

Online Distributed Mobile

Access at Anywhere & Anytime

Mobility and Connectivity

Over wireless Internet

Mobile Client Software

(Thin/Smart/Thick Client)

Pay-as-you-go

Billing Model

Customization and

Configuration

(8)

Mobile Testing Scope

Mobile

Testing

On-Demand Mobile

Test Run & Control

Compatibility,

Interoperation ability

and Usability Testing

Multi-Tenancy Testing

On Mobile Clouds

Internationalization

Testing on Mobile

Mobility and Connectivity

Testing

Function and

Behavior Testing

Mobile Test Simulation

and Virtualization

QoS Testing

(Performance, load,

Scalability..)

(9)

Mobile TaaS Requirements and Needs

Mobile

TaaS

Large-Scale On-demand

Mobile Test Services

On-Demand Scalable

Mobile Test Infrastructures

and Environments

Contracting,

Utilization Billing

and Reporting

Test Project-Oriented

Multi-Tenancy &

Customization Support

Control and Configuration

Of Mobile Test Environments

Test Tracking, Monitoring,

and Coverage Analysis

Easy Interoperation ability,

Mobility & Connectivity

Testing at Anywhere

& Anytime (365/7/24)

for Mobile App/Web App.

(10)

Why Is MTaaS Important?

Diverse mobile devices and HW appliances &APIs

Diverse mobile operation environments

(platforms, connectivity, and configurations)

Diverse wireless connectivity and configurations

High costs on mobile test infrastructures due to:

Frequent changes and upgrades on:

Mobile devices, appliances, and their APIs

Mobile applications, features, and service plans

Mobile platforms, technologies, input solutions

Complex mobile user interfaces

Diverse mobile browsers and technologies

Input approaches, display screens

Multi-language mobile contents

Complexity

Large-scale on:

On-demand mobile test service requests

Mobile test simulation and traffic loads

Virtualization of mobile devices and environments

Large-Scale

Multi-tenancy of mobile app/web applications

Functions, behaviors, and QoS REQs

Data, security features, user interfaces

(11)
(12)
(13)

Service Component Unit Testing Service Component Integration Tenant-Based System Function Testing Tenant-Based System Testing Mobile Feature Testing Tenant-Based

Continuous Testing Recovery TestingFault/Disaster Live Upgrading Testing Scenario TestingLive State Live Re-Test Testing

Multi-tenancy Testing APP Installation

& Deployment Security Testing

Usability Testing Compatibility Testing Connectivity Testing System QoS Testing Mobile Security Testing Scalability

Testing Reliability Testing Performance & Load Testing Availability Testing Function Testing GUI-Based /Scenario Testing Business Intelligence Testing Behavior Testing Black-Box

Testing White-Box Testing Performance Testing Component API Testing Internationalization Testing Interoperability Testing Mobility Feature Testing

(14)

Major Players and Tools in Mobile Testing

M IT E M on ke yT al k se eT es tM ob il e N eo lo ad Sa nd St or m M ob il eC lo u d Si ku li C al ab as h eg gP la nt M on ke yR u nn er R ob ot iu mD ol lo p Se le ni um A nd ro id W eb D ri ve r Testing Focus Functional Testing R e -Testing Performance Testing Load Testing Platform Support Linux Windows Mac Tests on Mobile O.S

Android iOS Windows Testing Mobile Web Apps

Testing ative Apps Source code based testing Tests done using Emulator

Real Device Scrip languages Java

Python Jython Ruby Perl Record & Replay

Image based Service Cost

License / Subscription Open Source

(15)

Mobile Testing Environments and Infrastructures

Emulation-Based Testing

Emulation-Based Mobile GUI Mobile App server

Simulation-Based Mobile Testing

Simulation-Based Mobile GUI Mobile App server

Device-Based Mobile Testing

Mobile App server

Wireless Internet

Mobile Device in a Lab.

Remote Device-Based Mobile Testing

Mobile App server

Wireless Internet

Remote Mobile Device in a third-party mobile Lab.

(16)

Conventional Mobile Testing Approaches

Perspectives/ Different Approaches Emulation-Based Testing Simulation-Based Testing Mobile Device Based Testing in a Lab. Remote Device Based Mobile Testing in a Lab.

System Function Testing Limitations on device-specific functions Limitations on device-specific functions Support Support End-to-End System Behavior Testing

w/o real mobile device behaviors

w/o real mobile device behaviors

Support Support

Mobile Compatibility & Interoperability Testing

Very limited Very limited Require diverse mobile devices and platforms

Require diverse mobile devices and platforms

Mobile GUI and Usability Testing

Limited w/o devices

Limited w/o devices

Good for real device experience

Good for real device experience

Load Testing Require server-oriented load testing solutions Easy to generate simulation-based system loads Costly to generate real mobile traffic loads

Costly to generate real mobile traffic loads

Wireless Connectivity Testing

Limited coverage Limited coverage Good coverage but require local

wireless network connection control

Good coverage, but require remote wireless network connection control

Operation Costs Low Low High High

Download & deployment Not support for Mobile Native App

Not support for Mobile Native App

(17)

Conventional Mobile Testing Approaches

Perspectives/ Different Approaches Emulation-Based Testing Simulation-Based Testing Mobile Device Based Testing Remote Device Based Mobile Testing

Scalability Testing Limited Support Limited Support Not Support Not Support Performance Testing Check

emulation-based mobile system performance

Check simulation-based mobile system performance

Check mobile device user and system performance

Check remote mobile device and system performance Multi-Tenancy

Testing

Limited coverage on multi-tenanted GUI features for specific devices

Limited coverage on multi-tenanted GUI features for specific devices

Supported with very high costs

Supported with very high costs Mobility and Location-based Testing Limited testing based on configured locations Limited testing based on simulated locations Limited to signed device testing Limited to signed device testing Mobile Security Testing Limited coverage on real mobile OS/ platforms and mobile clients Limited coverage on mobile OS/ platforms and mobile clients Support testing on very limited scale

Support testing on very limited scale

Test Automation Easy for GUI testing Easy for GUI testing and load testing

Required integrated mobile test beds with wireless connectivity

Required integrated mobile test beds with wireless connectivity

(18)

Different Mobile TaaS Infrastructure

Emulation-Based

Mobile Testing on Clouds

Emulation-Based

Mobile clients Mobile App Instances Wireless Internet

Simulation-Based Mobile

Testing on Clouds

Simulation-Based Mobile clients Wireless Internet Mobile App Instances

Device-Based Mobile

Testing on Clouds

Mobile App

Instances

Wireless Internet

(19)

Mobile TaaS Infrastructure for Native Mobile App

Mobile TaaS Server

Internet

WAN

G4

G3

App. Store Server

Private/Public Cloud

Intranet

Mobile Test

(20)

Mobile TaaS Infrastructures for Mobile Web App.

Public/Private Cloud

2G,3G,4G,5G

Internet

Remote Mobile Device Cloud

Remote Mobile Emulation Cloud

Mobile TaaS Server

Mobile Test

Platform

(21)

Expected Features & Benefits of Mobile TaaS

Mobile Test Automation Mobile Test Efficiency Large-Scale Test/Simulation Mobile Test Environment Costs

Mobile Device Costs

Mobile Testing Operation Costs

Manual Test Operations

Test Service Process Tim

Pay-As-You-Test Test Anywhere Crossing Diverse Wireless Networks Crossing Platforms

(22)

Different Mobile TaaS Approaches

Perspectives/ Different Approaches Emulation-Based Testing on Clouds Simulation-Based Testing On Clouds Mobile Device Based Testing on Clouds Remote Device Based Mobile Testing On Clouds

Testing Service Model Emulation-based Mobile TaaS Service Model

Simulation-based Mobile TaaS Service Model

Private Mobile TaaS Service Model

Remote Mobile TaaS Service Model

Business and Billing Model

Pay-as-you-use on device emulation could and TaaS server cloud

Pay-as-you-use on device simulation could and TaaS server cloud

Pay-as-you-use on private device could and TaaS server cloud

Pay-as-you-use on remote device could and TaaS server cloud

Testing Environment Emulation-Based Mobile TaaS Infrastructure on Clouds Simulation-Based Mobile TaaS Infrastructure on Clouds Private Mobile TaaS Infrastructure on Clouds

Remote Mobile TaaS infrastructure on Clouds

Costs No costs on mobile

devices No costs on mobile devices High-costs for mobile devices Pay-as-you-use for third-party mobile devices Mobile Usability Testing

No Support No Support Complete

Support

Complete Support Mobile End-To-End

Transaction Testing

Complete Support Complete Support Complete Support

Complete Support Mobile Function and

Behavior Testing

No coverage on device-specific GUI function & behaviors

No coverage on device-specific GUI function and behavior

Complete Support

(23)

Different Mobile TaaS Approaches

Perspectives/ Different Approaches Emulation-Based Testing on Clouds Simulation-Based Testing On Clouds Mobile Device Based Testing on Clouds Remote Device Based Mobile Testing On Clouds Scalable Function Testing

Supported Supported Supported Supported

Scalable GUI Testing Not Supported for device-specific features

Not Supported for device-specific features

Supported Supported

Scalable Behavior Testing

Lack test coverage on device-specific behaviors

Lack test coverage on device-specific behaviors

Supported Supported

Scalable Transaction Testing

Supported Supported Supported Supported

Scalable Compatibility & Interoperability Testing limited support on mobile Very limited support on mobile

Good support Good Support

Scalable QoS Testing Supported Supported Supported Supported Scalable Mobility and

Location-based Testing

Limited Support Limited Support Supported Supported Scalable Testing

Simulation and Virtualization

(24)

Proposed Testing Infrastructure for

Mobile App. and SaaS Systems

Mobile TaaS Server

Internet

WAN (G2-G5)

Under-Test

Mobile APP Server

routers

Mobile device cloud

Back-end

Test Environment

Agent

Major Objectives:

Reduce mobile testing costs and complexity

Easy to set up an integrated mobile test environment for projects

(25)

Proposed Mobile Test Stack for Mobile App. and MSaaS

Mobile TaaS Server

Mobile

Test Platform Server (s)

Mobile Device Test Bed Mobile Test Platform

Mobile Test Network Mobile Test Engine Deployed Mobile APP.

Mobile Test-Ware Agent Mobile Test Control Agent

Mobile Test Comm. Agent Mobile Test Migration Agent MTaaS Contract and Billing

MTaaS User Management Mobile TaaS Manager Mobile TaaS Big Repository

Mobile Test Instance

Mobile Test Instance

Mobile Test Instance

Mobile TaaS Cloud

Mobile Test Bed Cloud

Mobile TaaS Agent

(26)

Proposed Mobile Test Stack for Mobile App. and SaaS

Mobile

Test Server(s)

Mobile Device Test Bed Mobile Test Platform

Mobile Test Network Mobile Test Engine Deployed Mobile APP.

Mobile Test Instance

Mobile Test Instance

Mobile Test Instance

Mobile Test Bed Cloud

Defined Mobile Device Test Bed Defined Mobile Test Platform

Defined Mobile Test Network Defined Mobile Test Intelligence Defined Mobile App. Deployment.

Mobile Device Test Bed Mobile Test Platform

Mobile Test Network Mobile Test Engine Deployed Mobile APP.

Mobile Test Instance

Mobile Test Stack

(27)

Test Models and Test Criteria

Mobility Test Model & Criteria Connectivity Test Model & Criteria Mobile Envir. Test Model &

Criteria

Multi-tenancy Test Model &

Criteria Interoperability

Test Model & Criteria Mobile Security

Test Model & Criteria

Mobile Usability Test Model &

Criteria End-To-End

Transaction Test Model & Criteria

Mobile App QoS Test Model &

Criteria Mobile User Interface Test Model & Criteria

Compatibility Test

Model & Criteria Mobile Function Test Model &

Criteria

Mobile Behavior Test Model &

(28)

Mobile Apps and Mobile SaaS Test Coverage

Mobility Functions Behaviors

QoS REQs Mobile Environments

And Compatibility Connectivity &Interoperability Usability &Languages Multi-Tenancy Mobile GUIs

Mobile GUI Events GUI Forms/Contents Operation Flows Device Features Tenanted Data Tenanted Behaviors Tenanted Functions Tenanted GUIs State Paths State Transactions Dynamic States Scenarios Partitions Boundaries Decisions Throughputs Loads Performance

Scalability Appliances & APIs OS/Platforms

Mobile Devices Diverse Browsers

Mobility Features

Mobility Transactions

Mobility Data External Connections Wireless Connections

SaaS Connections Device API Connections

Look & Feel Languages Operation Flows

(29)

Modeling for Mobile Test Environments

Smartphone Tablet Web phone Select-1 Mobile device Wireless PAN (such as NFC/BlueTooth) Network Connectivity AND Platform/OS Select-1 Android iOS S40 Window RT BlackBerry Select-1 Version #1 Version #k Appliance APIs Select-1 Wireless LAN (such as WiFi..) WAN(2G/3G/4G..) Wireless internet Celluar Network (such as GSM/PCS) Smartphone A API-A API-B OR Mobile Device Mobile Environment Set-up Function Fi Dj Ek Fi (Dj, Ek)

(30)

Scalability & Performance Modeling in Clouds

i.e. System dynamics Dynamic Scalability

and Performance Models for Testing

Adaptive Dynamic Resource-Oriented Scalability Models Dynamic Models for

Scalability Costs and QoS & SLAs

Modeling dynamic behaviors of allocating computing resources at the cloud level (such as cloud infrastructures) Modeling dynamic behaviors of

MSaaS/application performance in scalable cloud environment.

Modeling scalability-based economic costs, QoS Performance and Scalability based SLA agreements for MSaaS

Above Clouds SaaS Level Cloud Level Resource Measurement Performance

Measurement Cost Measurement

Scalability Measurement

(31)

Multi-Tenancy of SaaS and Mobile SaaS

Multi-Tenants Sharing Forms or Frames GUI Events Operation Flows Mobile GUI Structure

MSaaS Business Logics App

Workflows FunctionsService DB Schema

& Service DB Tables SaaS DBMobile Mobile SaaS App Logics Mobile GUI Single Tenant Forms or Frames GUI Events Operation Flows Mobile GUI Structure

MSaaS Business Logics MSaaS App

Workflows FunctionsService DB Schema

& Service DB Tables Isolating

(32)

W1 W2 W3 W4 e12 e24 e43 e13 e21 e31 e42 e34 S0 e1 e2 e24

G

WFM Loop node Switch node

Mobile GUI forms Mobile Style & Profile

Mobile GUI Flow Model

Mobile Client Structure Model

G

CSM

SQL DB

Tenant-A

G

DT

G

WF

scheme

G

GFM

Service Workflow

Decision Table

NoSQL DB

Multi-Tenancy Test Models

State diagram

G

SD

API flow diagram

G

API

Data test model

(33)

Multi-Tenancy Testing for MSaaS

Mobile GUI Flow Model Mobile Client Structure Model

G

B CSM SQL DB Tenant-B

G

B DT

G

B WF

scheme

G

B GFM Workflow Decision Table NoSQL DB State diagram

G

B SD

API flow diagram

G

B API

G

B DM Data test model Mobile GUI Flow Model Mobile Client Structure Model

G

A CSM SQL DB Tenant-A

G

A DT

G

A WF

scheme

G

A GFM Workflow Decision Table NoSQL DB State diagram

G

A SD

API flow diagram

G

A API

G

A DM Data test model

(34)

Cloud Computing as New Computing

SaaS (software-as-a-service), PaaS (platform-as-a-service),

and IaaS (infrastructure-as-a-service) as three principal

components of cloud

SaaS ≠ software, PaaS ≠ platform and IaaS ≠ infrastructure.

In fact, SaaS

software, PaaS

platform, IaaS

infrastructure.

From

user point of view

: XaaS means online, rentable,

composable, searchable, scalable, virtualized, seemingly

unlimited computing, storage and networking

(35)

XaaS Internal Point of View

Decentralized operations: Parallel and distributed computing, P2P

operations, and MapReduce operations

Metadata-based computing: Separating metadata from data, and use

metadata to control almost all execution

Data-centric computing: Metadata and data will be retrieved often,

significant computation requires data retrieval from the associated

databases, new database model such as eventual consistency, writers over

readers

Automated redundancy and recovery: Automated triplicate writes,

redundant components, data, processors, storage, and communication

Automated scalability: Multi-level scalability architecture with

redundant load balancers, stateless service design, automated migration,

automated data partitioning, automated workload detection

(36)

SaaS Example: Salesforce.com

[3]

36

Poly-Morphic

Application

Distributed operations

Metadata-driven computing

Data-centric computing: runtime

application generator dynamically

builds applications in response to

specific user requests

Automated redundancy and recovery

management

(37)

Example Scalability Architecture (Salesforce.com)

Network Services Storage Services Backup Services Monitoring Services NA0 Pod NA1 Pod NA2 Pod NA3 Pod NA4 Pod NA5 Pod EMEA Pod APAC Pod Sandbox Pod EMEA2 Pod NA6 Pod NA7 Pod “N” Pod Threshold User Capacity = Add a POD

(38)

Redundancy and Recovery Example

(Salesforce.com)

PaaS level

: A PaaS system is often a

black box and provides chunk-level

redundancy and recovery.

SaaS-oriented PaaS

: This divides the

PaaS resources to different tanants,

such as partitioning storage and

network resources (Cisco, VmWare,

NetApp for tenants.

SaaS

-

on-PaaS

: Design SaaS R&R

mechanisms on top of a PaaS. R&R

can be at the scheduler level,

ontology

,

data

,

metadata

levels, and

other levels.

38

Si Si SiSi

(39)

TaaS from User Point of View

TaaS means testing software will be online, composable, Web-based, on demand,

searchable, scalable, virtualized and secure testing environment, unlimited

computing, storage and networking

Test case generation

: Users or crowd can develop, debug, and evaluate

test cases online using automated test case generation tools online or

TaaS. One set of users can develop test cases, a different set can evaluate

test cases, all participants can be ranked.

Test script development

: Users or crowd can develop, debug and

evaluate test scripts using TaaS tools in a collaborative manner, or use

SaaS approach where software is constructed by reusing components

Test script compilation and deployment

: can be done automatically by

TaaS or as user directed

Test script execution

: Parallel, distributed, autonomous, or MapReduce

Test result evaluation

: Database operations such as automated saving,

intelligent retrieval, concurrent, parallel, distributed and MapReduce

evaluation execution

(40)

Concurrent Testing and Analysis in clouds

40

TA Analysis

TA Analysis

Combinatorial

Testing

Combinatorial

Testing

Interaction (X, F, P, N, U)

Configuration (X, F, P, N, U)

Eliminate configurations from

testing considerations

(1) Identify pass/fault

configurations

(2) Identify fault location

Automated detection of X or F

configurations using existing test

results

AR

AETG

IPO

(41)

TaaS from Internal Point of View

Decentralized operations

: Testing tasks may be

Executed in a parallel, distributed, autonomously,

on-demand, or triggered by policies;

Testing tasks may be migrated, performed in a redundant

manner, or embedded within cloud operations.

Metadata-based computing

: Use metadata to

Specify and control testing tasks such as time, frequency,

multi-tasking, redundant testing, parallel testing

Specify test cases, scripts, and environment

Specify test results such as indexing, location,

organization, profiles

(42)

Built-in Continuous Testing

… Workflows Services … Databases … Workflows Services

(a) SOA model (b) SaaS model

Databases …

… Workflows

Services …

Continuous Testing by Trigger Rules

triggers Continuous Testing by Obtaining Input/Output

triggers

(c) Continuous Testing in SaaS

(43)

Test Cases Generation from Metadata

Test cases can be generated by examining metadata:

For example, length of customer must be 64 bits, One can

generate a collection of customers of 64 bits, another

collection with 128 bits or any other bits.

Random number from 0

~

2

64

-

1

Another set is negative numbers

Greater than 2

64

-

1

The boundary value test cases can be generated from {-2,

-1, 0, +1, +2} around boundary of the constraints,

specifies by the metadata.

The need for International standard on test metadata for

(44)

TaaS from Internal Point of View

(continued)

Data-centric testing

: Use data in the SaaS, PaaS, or TaaS

database for testing. This may include

Automated test results saving in test databases like

in-memory databases where data are classified as hot, warm,

or cold

Storing monitoring data intelligently with automated index

generation

Analysis of test data to optimize test strategies

Multi-tenancy test script composition. i.e., one test code for

all testing needs, and use different components to

customize different testing needs

Automated logging like LogStash or Splunk manner

(45)

TaaS from Internal Point of View (continued)

Automated test redundancy management and recovery

:

Testing tasks can be partitioned and sent to different

processors for parallel and redundant processing

Test results can be merged following a mathematical

principle to increase the confidence of test execution

Test and test results can be recovered in case of failures

in a processor or in a cluster due to automated

redundancy management

Recovery follows the metadata-based approach

(46)

TaaS from Internal Point of View (continued)

Automated test scalability: Need to deal with scalability of 1) applications,

2) cloud infrastructure; 3) TaaS.

Testing scalability of applications and cloud infrastructure: Need to

follow cloud scalability mechanisms (such as 2-level scalability

architecture, stateless service design, and automated migration),

testing needs to set up an environment including test input, relevant

SaaS and PaaS services, execute test, and evaluate the performance.

TaaS scalability: as the SUT scales up at runtime (with more

resources and more data generated), TaaS needs to scale up

proportionally to track and monitor the execution. TaaS scalability

mechanism may include:

Multi-level TaaS load balancers used to dispatch (stateless) testing

tasks to different processors for execution

Automated testing tasks migration to optimize test execution

(47)

Combinatorial Testing TaaS

Combinatorial Testing (CT) can detect failures triggered by

interactions of parameters with a covering array test suite

generated. Many outstanding algorithms are available, and

they focus on

test coverage

.

But cloud computing need to identify faults, not just

failures. Also need a algebra to allow to Inference of the

test results of a subset to the test results of its superset.

Allowing asynchronous merging of test results.

Allow test problem space to be decomposed.

Allow concurrent and asynchronous testing

(48)

Tenant Application Testing with Fault Detection

10 components and 10 values in each components. Thus the total number of

configurations is 10 billion.

Faults scale ( only for two-way faults ): the number of 2-way interactions is

4500.

Simulated different number of faults: 5, 50, 100, 200, 300, 500, that represent

0.1%, 1.1%, 2.2%, 3.3%, 6.67%, and 11.1% of 4,500 2-way interactions

respectively.

48 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 200 1200 2200 3200 4200 5200 6200

final faults/faults

5 faults 50 faults 100 faults 200 faults 300 faults

(49)

Five Status Indicators

Infeasible (X):

For certain interactions, it may happen that no feasible

test is permitted to contain this interaction. For example, it may be

infeasible to select two GUI components in one configuration such that

one says the wall is GREEN but the other says RED.

Faulty (F): If the interaction has been found to be faulty.

Operational (P): Among the rest, if an interaction has appeared in a test

whose execution gave an operational result, the interaction cannot be

faulty.

Irrelevant (N): For some feasible interactions, it may be the case that

certain interactions are not expected to arise, so while it is possible to run

a test containing the interaction, there is no requirement to do so.

Unknown (U): If neither of these occurs then the status of the interaction

is required but not currently known.

(50)

Multiplication Rule

The binary operation has the following properties of commutatively and

associativity. Superset of X or F are still or F.

(51)

Concurrent Testing Framework

(52)

Merging Concurrent Testing Results

The testing results of a interaction T in different servers should satisfy the following

constraints.

If V (T ) = U in one cluster, then in other clusters the V (T ) can be either F, P,

N, or U.

If V (T ) = N in one cluster, then in other clusters, the V (T ) can be either F, P,

N, or U.

If V (T ) = P in one cluster, then the V (T ) can be either P, N, or U in all

clusters;

If V (T ) = F in one cluster, then in other clusters, the V (T ) can be F, N, or U.

If V (T ) = X in one cluster, then in other clusters, the V (T ) can be X only.

If these constraints are satisfied, then the testing results can be merged. Otherwise,

there must be an error in the testing results. Addition operations

(53)

Partition Configurations

Algebraic rules allow one to partition configurations into different sets for

different servers to test, and these sets do not need to be non-overlapping.

Fifteen configurations, assuming only interaction (c, d, f) is faulty, and only

interaction (c, d, e) is infeasible, and all other interactions pass the testing.

(54)

Issues in Mobile TaaS

(a) Diversity of mobile devices and platforms

à

high complexity of mobile testing environments

à

high cost on set-up cloud-based mobile testing infrastructure

o

Diverse mobile devices with different configurations/operation environments

o

Diverse mobile appliances, drivers, and APIs from different vendors

o

Diverse wireless connectivity

o

Private/public cloud infrastructure for both mobile client and mobile SaaS

servers

(b) Mobile test automation in large-scale

à

require intelligent large-scale concurrent test automation

à

require automatic test control and handling methods for test failures

o

Lack of high-level test automation control technology (for example, test

control-oriented script language and protocol for remote control and control)

o

Lack of well-defined scripting standards, techniques and hierarchical control

solutions

(55)

(c) Lack of standards

à

higher cost in test tool integration and set-up for mobile test automaton

o

Lack of standard interfaces, languages, and connectivity protocols, and APIs

among test tools

o

Lack of well-defined QA processes and test criteria for mobile Apps and mobile

cloud services

o

Lack of standard test interfaces, APIs, and connectivity protocols to support

mobile app testing and mobile cloud service validation

(d) Lack of effective open mobile test stack/frameworks, and test tools

à

the difficulties and challenges in mobile test automation

o

Lack of an effective mobile test bed for diverse mobile apps

o

Lack of a scalable mobile test tool to support large-scale mobile traffic

loads and user accesses

o

Diverse wireless connectivity

(e) Globalization and internationalization

à

higher cost and difficulty in testing for diverse language contents

and Mobile User interfaces

Figure

Updating...

References

Related subjects :