Mobile Testing-as-a-Service (Mobile TaaS or MTaaS)

(1)

Mobile Testing-as-a-Service

(Mobile TaaS or MTaaS)

Jerry Gao, Ph.D. Professor

Computer Engineering Department

San Jose State University

W. T. Tsai, Ph.D. Professor

Sch Compt Infor & Dec Sys Engr

(2)

• Introduction

• What is Mobile TaaS?

• Why is Mobile TaaS Important?

• Mobile Testing Process

• Major Players and Tools

• Mobile Testing Approaches, Infrastructures, and Comparisons

• Mobile TaaS Approaches, Infrastructures, and Comparisons

• Expected Benefits and Features

• Test Models and Test Coverage Criteria

• User View of TaaS

• Cloud Computing View of TaaS

(3)

According to the latest study from Juniper Research, the market for cloud-based mobile applications will

grow

88%

from 2009 to 2014

. The market was just over $400 million this past year, says Juniper, but by 2014 it will reach

$9.5 billion

. Driving this growth will be the adoption of the new web standard HTML5, increased mobile broadband coverage and the need for always-on collaborative services for the enterprise.

(4)

(5)

Knowledge Background for Mobile TaaS Research

Cloud

Computing

& SaaS

Engineering

Mobile

Computing

Software

Testing and

Test

Automation

(6)

What Is Mobile TaaS?

No existing definition for Mobile TaaS.

Our

first definition

: (Mobile TaaS in a cloud infrastructure):

“Mobile Testing as a Service (known as Mobile TaaS) provides on-demand testing

services for mobile applications and SaaS to support software validation and

quality engineering processes by leveraging a cloud-based scalable mobile testing

environment to assure pre-defined given service-level-agreements (SLAs).

Mobile TaaS on clouds offers a new business model for diverse mobile software

validation services using the pay-as-you-test model to achieve cost-sharing and

cost-reduction in mobile computing resources, networks, cloud computing and

storage infrastructures.”

(7)

Mobile SaaS Features

Mobile

SaaS

On-Demand Function

Services with APIs

Searchable, Discoverable,

Deployable, and Executable

Multi-Tenancy and

Elastic Scalability

Online Distributed Mobile

Access at Anywhere & Anytime

Mobility and Connectivity

Over wireless Internet

Mobile Client Software

(Thin/Smart/Thick Client)

Pay-as-you-go

Billing Model

Customization and

Configuration

(8)

Mobile Testing Scope

Mobile

Testing

On-Demand Mobile

Test Run & Control

Compatibility,

Interoperation ability

and Usability Testing

Multi-Tenancy Testing

On Mobile Clouds

Internationalization

Testing on Mobile

Mobility and Connectivity

Testing

Function and

Behavior Testing

Mobile Test Simulation

and Virtualization

QoS Testing

(Performance, load,

Scalability..)

(9)

Mobile TaaS Requirements and Needs

Mobile

TaaS

Large-Scale On-demand

Mobile Test Services

On-Demand Scalable

Mobile Test Infrastructures

and Environments

Contracting,

Utilization Billing

and Reporting

Test Project-Oriented

Multi-Tenancy &

Customization Support

Control and Configuration

Of Mobile Test Environments

Test Tracking, Monitoring,

_{and Coverage Analysis}

Easy Interoperation ability,

Mobility & Connectivity

Testing at Anywhere

& Anytime (365/7/24)

for Mobile App/Web App.

(10)

Why Is MTaaS Important?

• Diverse mobile devices and HW appliances &APIs

• Diverse mobile operation environments

(platforms, connectivity, and configurations)

• Diverse wireless connectivity and configurations

High costs on mobile test infrastructures due to:

Frequent changes and upgrades on:

• Mobile devices, appliances, and their APIs

• Mobile applications, features, and service plans

• Mobile platforms, technologies, input solutions

Complex mobile user interfaces

• Diverse mobile browsers and technologies

• Input approaches, display screens

• Multi-language mobile contents

Complexity

Large-scale on:

• On-demand mobile test service requests

• Mobile test simulation and traffic loads

• Virtualization of mobile devices and environments

Large-Scale

Multi-tenancy of mobile app/web applications

• Functions, behaviors, and QoS REQs

• Data, security features, user interfaces

(11)

(12)

(13)

Service Component Unit Testing Service Component Integration Tenant-Based System Function Testing Tenant-Based System Testing Mobile Feature Testing Tenant-Based

Continuous Testing Recovery TestingFault/Disaster Live Upgrading Testing Scenario TestingLive State Live Re-Test Testing

Multi-tenancy Testing APP Installation

& Deployment Security Testing

Usability Testing Compatibility Testing Connectivity Testing System QoS Testing Mobile Security Testing Scalability

Testing Reliability Testing Performance & Load Testing Availability Testing Function Testing GUI-Based /Scenario Testing Business Intelligence Testing Behavior Testing Black-Box

Testing White-Box Testing Performance Testing Component API Testing Internationalization Testing Interoperability Testing Mobility Feature Testing

(14)

Major Players and Tools in Mobile Testing

M IT E M on ke yT al k se eT es tM ob il e N eo lo ad Sa nd St or m M ob il eC lo u d Si ku li C al ab as h eg gP la nt M on ke yR u nn er R ob ot iu mD ol lo p Se le ni um A nd ro id W eb D ri ve r Testing Focus Functional Testing R e -Testing Performance Testing Load Testing Platform Support Linux Windows Mac Tests on Mobile O.S

Android iOS Windows Testing Mobile Web Apps

Testing ative Apps Source code based testing Tests done using Emulator

Real Device Scrip languages Java

Python Jython Ruby Perl Record & Replay

Image based Service Cost

License / Subscription Open Source

(15)

Mobile Testing Environments and Infrastructures

Emulation-Based Testing

Emulation-Based Mobile GUI Mobile App server

Simulation-Based Mobile Testing

Simulation-Based Mobile GUI Mobile App server

Device-Based Mobile Testing

Mobile App server

Wireless Internet

Mobile Device in a Lab.

Remote Device-Based Mobile Testing

Mobile App server

Wireless Internet

Remote Mobile Device in a third-party mobile Lab.

(16)

Conventional Mobile Testing Approaches

Perspectives/ Different Approaches Emulation-Based Testing Simulation-Based Testing Mobile Device Based Testing in a Lab. Remote Device Based Mobile Testing in a Lab.

System Function Testing Limitations on device-specific functions Limitations on device-specific functions Support Support End-to-End System Behavior Testing

w/o real mobile device behaviors

Support Support

Mobile Compatibility & Interoperability Testing

Very limited Very limited Require diverse mobile devices and platforms

Require diverse mobile devices and platforms

Mobile GUI and Usability Testing

Limited w/o devices

Good for real device experience

Load Testing Require server-oriented load testing solutions Easy to generate simulation-based system loads Costly to generate real mobile traffic loads

Costly to generate real mobile traffic loads

Wireless Connectivity Testing

Limited coverage Limited coverage Good coverage but require local

wireless network connection control

Good coverage, but require remote wireless network connection control

Operation Costs Low Low High High

Download & deployment Not support for Mobile Native App

Not support for Mobile Native App

(17)

Conventional Mobile Testing Approaches

Perspectives/ Different Approaches Emulation-Based Testing Simulation-Based Testing Mobile Device Based Testing Remote Device Based Mobile Testing

Scalability Testing Limited Support Limited Support Not Support Not Support Performance Testing Check

emulation-based mobile system performance

Check simulation-based mobile system performance

Check mobile device user and system performance

Check remote mobile device and system performance Multi-Tenancy

Testing

Limited coverage on multi-tenanted GUI features for specific devices

Supported with very high costs

Supported with very high costs Mobility and Location-based Testing Limited testing based on configured locations Limited testing based on simulated locations Limited to signed device testing Limited to signed device testing Mobile Security Testing Limited coverage on real mobile OS/ platforms and mobile clients Limited coverage on mobile OS/ platforms and mobile clients Support testing on very limited scale

Support testing on very limited scale

Test Automation Easy for GUI testing Easy for GUI testing and load testing

Required integrated mobile test beds with wireless connectivity

(18)

Different Mobile TaaS Infrastructure

Emulation-Based

Mobile Testing on Clouds

Emulation-Based

Mobile clients Mobile App Instances Wireless Internet

Simulation-Based Mobile

Testing on Clouds

Simulation-Based Mobile clients Wireless Internet Mobile App Instances

Device-Based Mobile

Testing on Clouds

Mobile App

Instances

Wireless Internet

(19)

Mobile TaaS Infrastructure for Native Mobile App

Mobile TaaS Server

Internet

WAN

G4

G3

App. Store Server

Private/Public Cloud

Intranet

Mobile Test

(20)

Mobile TaaS Infrastructures for Mobile Web App.

Public/Private Cloud

2G,3G,4G,5G

Internet

Remote Mobile Device Cloud

Remote Mobile Emulation Cloud

Mobile TaaS Server

Mobile Test

Platform

(21)

Expected Features & Benefits of Mobile TaaS

Mobile Test Automation Mobile Test Efficiency Large-Scale Test/Simulation Mobile Test Environment Costs

Mobile Device Costs

Mobile Testing Operation Costs

Manual Test Operations

Test Service Process Tim

Pay-As-You-Test Test Anywhere Crossing Diverse Wireless Networks Crossing Platforms

(22)

Different Mobile TaaS Approaches

Perspectives/ Different Approaches Emulation-Based Testing on Clouds Simulation-Based Testing On Clouds Mobile Device Based Testing on Clouds Remote Device Based Mobile Testing On Clouds

Testing Service Model Emulation-based Mobile TaaS Service Model

Simulation-based Mobile TaaS Service Model

Private Mobile TaaS Service Model

Remote Mobile TaaS Service Model

Business and Billing Model

Pay-as-you-use on device emulation could and TaaS server cloud

Pay-as-you-use on device simulation could and TaaS server cloud

Pay-as-you-use on private device could and TaaS server cloud

Pay-as-you-use on remote device could and TaaS server cloud

Testing Environment Emulation-Based Mobile TaaS Infrastructure on Clouds Simulation-Based Mobile TaaS Infrastructure on Clouds Private Mobile TaaS Infrastructure on Clouds

Remote Mobile TaaS infrastructure on Clouds

Costs No costs on mobile

devices No costs on mobile devices High-costs for mobile devices Pay-as-you-use for third-party mobile devices Mobile Usability Testing

No Support No Support Complete

Support

Complete Support Mobile End-To-End

Transaction Testing

Complete Support Complete Support Complete Support

Complete Support Mobile Function and

Behavior Testing

No coverage on device-specific GUI function & behaviors

No coverage on device-specific GUI function and behavior

Complete Support

(23)

Different Mobile TaaS Approaches

Perspectives/ Different Approaches Emulation-Based Testing on Clouds Simulation-Based Testing On Clouds Mobile Device Based Testing on Clouds Remote Device Based Mobile Testing On Clouds Scalable Function Testing

Supported Supported Supported Supported

Scalable GUI Testing Not Supported for device-specific features

Not Supported for device-specific features

Supported Supported

Scalable Behavior Testing

Lack test coverage on device-specific behaviors

Supported Supported

Scalable Transaction Testing

Supported Supported Supported Supported

Scalable Compatibility & Interoperability Testing limited support on mobile Very limited support on mobile

Good support Good Support

Scalable QoS Testing Supported Supported Supported Supported Scalable Mobility and

Location-based Testing

Limited Support Limited Support Supported Supported Scalable Testing

Simulation and Virtualization

(24)

Proposed Testing Infrastructure for

Mobile App. and SaaS Systems

Mobile TaaS Server

Internet

WAN (G2-G5)

Under-Test

Mobile APP Server

routers

Mobile device cloud

Back-end

Test Environment

Agent

Major Objectives:

• Reduce mobile testing costs and complexity

• Easy to set up an integrated mobile test environment for projects

(25)

Proposed Mobile Test Stack for Mobile App. and MSaaS

Mobile TaaS Server

Mobile

Test Platform Server (s)

Mobile Device Test Bed Mobile Test Platform

Mobile Test Network Mobile Test Engine Deployed Mobile APP.

Mobile Test-Ware Agent Mobile Test Control Agent

Mobile Test Comm. Agent Mobile Test Migration Agent MTaaS Contract and Billing

MTaaS User Management Mobile TaaS Manager Mobile TaaS Big Repository

Mobile Test Instance

Mobile Test Instance

Mobile TaaS Cloud

Mobile Test Bed Cloud

Mobile TaaS Agent

(26)

Proposed Mobile Test Stack for Mobile App. and SaaS

Mobile

Test Server(s)

Mobile Test Instance

Mobile Test Instance

Mobile Test Bed Cloud

Defined Mobile Device Test Bed Defined Mobile Test Platform

Defined Mobile Test Network Defined Mobile Test Intelligence Defined Mobile App. Deployment.

Mobile Test Instance

Mobile Test Stack

(27)

Test Models and Test Criteria

Mobility Test Model & Criteria Connectivity Test Model & Criteria Mobile Envir. Test Model &

Criteria

Multi-tenancy Test Model &

Criteria Interoperability

Test Model & Criteria Mobile Security

Test Model & Criteria

Mobile Usability Test Model &

Criteria End-To-End

Transaction Test Model & Criteria

Mobile App QoS Test Model &

Criteria Mobile User Interface Test Model & Criteria

Compatibility Test

Model & Criteria _{Mobile Function} Test Model &

Criteria

Mobile Behavior Test Model &

(28)

Mobile Apps and Mobile SaaS Test Coverage

Mobility Functions Behaviors

QoS REQs Mobile Environments

And Compatibility Connectivity &_{Interoperability} Usability &Languages Multi-Tenancy Mobile GUIs

Mobile GUI Events GUI Forms/Contents Operation Flows Device Features Tenanted Data Tenanted Behaviors Tenanted Functions Tenanted GUIs State Paths State Transactions Dynamic States Scenarios Partitions Boundaries Decisions Throughputs Loads Performance

Scalability Appliances & APIs OS/Platforms

Mobile Devices Diverse Browsers

Mobility Features

Mobility Transactions

Mobility Data External Connections Wireless Connections

SaaS Connections Device API Connections

Look & Feel Languages Operation Flows

(29)

Modeling for Mobile Test Environments

Smartphone Tablet Web phone Select-1 Mobile device Wireless PAN (such as NFC/BlueTooth) Network Connectivity AND Platform/OS Select-1 Android iOS S40 Window RT BlackBerry Select-1 Version #1 Version #k Appliance APIs Select-1 Wireless LAN (such as WiFi..) WAN(2G/3G/4G..) Wireless internet Celluar Network (such as GSM/PCS) Smartphone A API-A API-B OR Mobile Device Mobile Environment Set-up Function Fi Dj _Ek Fi (Dj, Ek)

(30)

Scalability & Performance Modeling in Clouds

i.e. System dynamics Dynamic Scalability

and Performance Models for Testing

Adaptive Dynamic Resource-Oriented Scalability Models Dynamic Models for

Scalability Costs and QoS & SLAs

Modeling dynamic behaviors of allocating computing resources at the cloud level (such as cloud infrastructures) Modeling dynamic behaviors of

MSaaS/application performance in scalable cloud environment.

Modeling scalability-based economic costs, QoS Performance and Scalability based SLA agreements for MSaaS

Above Clouds SaaS Level Cloud Level Resource Measurement Performance

Measurement Cost Measurement

Scalability Measurement

(31)

Multi-Tenancy of SaaS and Mobile SaaS

Multi-Tenants Sharing Forms or Frames GUI Events Operation Flows Mobile GUI Structure

MSaaS Business Logics App

Workflows FunctionsService DB Schema

& Service DB Tables SaaS DBMobile Mobile SaaS App Logics Mobile GUI Single Tenant Forms or Frames GUI Events Operation Flows Mobile GUI Structure

MSaaS Business Logics MSaaS App

Workflows FunctionsService DB Schema

& Service DB Tables Isolating

(32)

W1 W2 W3 W4 e12 e24 e43 e13 _e21 e31 e42 e34 S0 e1 e2 e24

G

_WFM Loop node Switch node

Mobile GUI forms Mobile Style & Profile

Mobile GUI Flow Model

Mobile Client Structure Model

G

_CSM

SQL DB

Tenant-A

G

_DT

G

_WF

scheme

G

_GFM

Service Workflow

Decision Table

NoSQL DB

Multi-Tenancy Test Models

State diagram

G

_SD

API flow diagram

G

_API

Data test model

(33)

Multi-Tenancy Testing for MSaaS

Mobile GUI Flow Model Mobile Client Structure Model

G

B CSM SQL DB Tenant-B

_G

B DT

G

B WF

scheme

G

B GFM Workflow Decision Table NoSQL DB State diagram

G

B SD

G

B API

G

B DM Data test model Mobile GUI Flow Model Mobile Client Structure Model

G

A CSM SQL DB Tenant-A

_G

A DT

G

A WF

scheme

G

A GFM Workflow Decision Table NoSQL DB State diagram

G

A SD

G

A API

G

A DM Data test model

(34)

Cloud Computing as New Computing

• SaaS (software-as-a-service), PaaS (platform-as-a-service),

and IaaS (infrastructure-as-a-service) as three principal

components of cloud

• SaaS ≠ software, PaaS ≠ platform and IaaS ≠ infrastructure.

• In fact, SaaS

_⊃

software, PaaS

_⊃

platform, IaaS

_⊃

infrastructure.

• From

user point of view

: XaaS means online, rentable,

composable, searchable, scalable, virtualized, seemingly

unlimited computing, storage and networking

(35)

XaaS Internal Point of View

• Decentralized operations: Parallel and distributed computing, P2P

operations, and MapReduce operations

• Metadata-based computing: Separating metadata from data, and use

metadata to control almost all execution

• Data-centric computing: Metadata and data will be retrieved often,

significant computation requires data retrieval from the associated

databases, new database model such as eventual consistency, writers over

readers

• Automated redundancy and recovery: Automated triplicate writes,

redundant components, data, processors, storage, and communication

• Automated scalability: Multi-level scalability architecture with

redundant load balancers, stateless service design, automated migration,

automated data partitioning, automated workload detection

(36)

SaaS Example: Salesforce.com

[3]

36

Poly-Morphic

Application

• Distributed operations

• Metadata-driven computing

• Data-centric computing: runtime

application generator dynamically

builds applications in response to

specific user requests

• Automated redundancy and recovery

management

(37)

Example Scalability Architecture (Salesforce.com)

Network Services Storage Services Backup Services Monitoring Services NA0 Pod NA1 Pod NA2 Pod NA3 Pod NA4 Pod NA5 Pod EMEA Pod APAC Pod Sandbox Pod EMEA2 Pod NA6 Pod NA7 Pod “N” Pod Threshold User Capacity = Add a POD

(38)

Redundancy and Recovery Example

(Salesforce.com)

• PaaS level

: A PaaS system is often a

black box and provides chunk-level

redundancy and recovery.

• SaaS-oriented PaaS

: This divides the

PaaS resources to different tanants,

such as partitioning storage and

network resources (Cisco, VmWare,

NetApp for tenants.

• SaaS

-

on-PaaS

: Design SaaS R&R

mechanisms on top of a PaaS. R&R

can be at the scheduler level,

ontology

,

data

,

metadata

levels, and

other levels.

38

Si Si SiSi

(39)

TaaS from User Point of View

TaaS means testing software will be online, composable, Web-based, on demand,

searchable, scalable, virtualized and secure testing environment, unlimited

computing, storage and networking

–

Test case generation

: Users or crowd can develop, debug, and evaluate

test cases online using automated test case generation tools online or

TaaS. One set of users can develop test cases, a different set can evaluate

test cases, all participants can be ranked.

–

Test script development

: Users or crowd can develop, debug and

evaluate test scripts using TaaS tools in a collaborative manner, or use

SaaS approach where software is constructed by reusing components

–

Test script compilation and deployment

: can be done automatically by

TaaS or as user directed

–

Test script execution

: Parallel, distributed, autonomous, or MapReduce

–

Test result evaluation

: Database operations such as automated saving,

intelligent retrieval, concurrent, parallel, distributed and MapReduce

evaluation execution

(40)

Concurrent Testing and Analysis in clouds

40

TA Analysis

Combinatorial

Testing

Combinatorial

Testing

Interaction (X, F, P, N, U)

Configuration (X, F, P, N, U)

Eliminate configurations from

testing considerations

(1) Identify pass/fault

configurations

(2) Identify fault location

Automated detection of X or F

configurations using existing test

results

AR

AETG

IPO

(41)

TaaS from Internal Point of View

• Decentralized operations

: Testing tasks may be

• Executed in a parallel, distributed, autonomously,

on-demand, or triggered by policies;

• Testing tasks may be migrated, performed in a redundant

manner, or embedded within cloud operations.

• Metadata-based computing

: Use metadata to

• Specify and control testing tasks such as time, frequency,

multi-tasking, redundant testing, parallel testing

• Specify test cases, scripts, and environment

• Specify test results such as indexing, location,

organization, profiles

(42)

Built-in Continuous Testing

… Workflows Services … Databases … Workflows Services

(a) SOA model _{(b) SaaS model}

…

Databases …

… Workflows

Services …

Continuous Testing by Trigger Rules

triggers Continuous Testing by Obtaining Input/Output

triggers

(c) Continuous Testing in SaaS

(43)

Test Cases Generation from Metadata

• Test cases can be generated by examining metadata:

–

For example, length of customer must be 64 bits, One can

generate a collection of customers of 64 bits, another

collection with 128 bits or any other bits.

• Random number from 0

~

2

64 _-

₁

• Another set is negative numbers

• Greater than 2

64 _-

₁

–

The boundary value test cases can be generated from {-2,

-1, 0, +1, +2} around boundary of the constraints,

specifies by the metadata.

–

The need for International standard on test metadata for

(44)

TaaS from Internal Point of View

(continued)

• Data-centric testing

: Use data in the SaaS, PaaS, or TaaS

database for testing. This may include

• Automated test results saving in test databases like

in-memory databases where data are classified as hot, warm,

or cold

• Storing monitoring data intelligently with automated index

generation

• Analysis of test data to optimize test strategies

• Multi-tenancy test script composition. i.e., one test code for

all testing needs, and use different components to

customize different testing needs

• Automated logging like LogStash or Splunk manner

(45)

TaaS from Internal Point of View (continued)

Automated test redundancy management and recovery

:

–

Testing tasks can be partitioned and sent to different

processors for parallel and redundant processing

–

Test results can be merged following a mathematical

principle to increase the confidence of test execution

–

Test and test results can be recovered in case of failures

in a processor or in a cluster due to automated

redundancy management

–

Recovery follows the metadata-based approach

(46)

TaaS from Internal Point of View (continued)

Automated test scalability: Need to deal with scalability of 1) applications,

2) cloud infrastructure; 3) TaaS.

–

Testing scalability of applications and cloud infrastructure: Need to

follow cloud scalability mechanisms (such as 2-level scalability

architecture, stateless service design, and automated migration),

testing needs to set up an environment including test input, relevant

SaaS and PaaS services, execute test, and evaluate the performance.

–

TaaS scalability: as the SUT scales up at runtime (with more

resources and more data generated), TaaS needs to scale up

proportionally to track and monitor the execution. TaaS scalability

mechanism may include:

• Multi-level TaaS load balancers used to dispatch (stateless) testing

tasks to different processors for execution

• Automated testing tasks migration to optimize test execution

(47)

Combinatorial Testing TaaS

• Combinatorial Testing (CT) can detect failures triggered by

interactions of parameters with a covering array test suite

generated. Many outstanding algorithms are available, and

they focus on

test coverage

.

• But cloud computing need to identify faults, not just

failures. Also need a algebra to allow to Inference of the

test results of a subset to the test results of its superset.

• Allowing asynchronous merging of test results.

• Allow test problem space to be decomposed.

• Allow concurrent and asynchronous testing

(48)

Tenant Application Testing with Fault Detection

• 10 components and 10 values in each components. Thus the total number of

configurations is 10 billion.

• Faults scale ( only for two-way faults ): the number of 2-way interactions is

4500.

• Simulated different number of faults: 5, 50, 100, 200, 300, 500, that represent

0.1%, 1.1%, 2.2%, 3.3%, 6.67%, and 11.1% of 4,500 2-way interactions

respectively.

48 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 200 1200 2200 3200 4200 5200 6200

final faults/faults

5 faults 50 faults 100 faults 200 faults 300 faults

(49)

Five Status Indicators

• Infeasible (X):

For certain interactions, it may happen that no feasible

test is permitted to contain this interaction. For example, it may be

infeasible to select two GUI components in one configuration such that

one says the wall is GREEN but the other says RED.

• Faulty (F): If the interaction has been found to be faulty.

• Operational (P): Among the rest, if an interaction has appeared in a test

whose execution gave an operational result, the interaction cannot be

faulty.

• Irrelevant (N): For some feasible interactions, it may be the case that

certain interactions are not expected to arise, so while it is possible to run

a test containing the interaction, there is no requirement to do so.

• Unknown (U): If neither of these occurs then the status of the interaction

is required but not currently known.

(50)

Multiplication Rule

• The binary operation has the following properties of commutatively and

associativity. Superset of X or F are still or F.

(51)

Concurrent Testing Framework

(52)

Merging Concurrent Testing Results

The testing results of a interaction T in different servers should satisfy the following

constraints.

–

If V (T ) = U in one cluster, then in other clusters the V (T ) can be either F, P,

N, or U.

–

If V (T ) = N in one cluster, then in other clusters, the V (T ) can be either F, P,

N, or U.

–

If V (T ) = P in one cluster, then the V (T ) can be either P, N, or U in all

clusters;

–

If V (T ) = F in one cluster, then in other clusters, the V (T ) can be F, N, or U.

–

If V (T ) = X in one cluster, then in other clusters, the V (T ) can be X only.

• If these constraints are satisfied, then the testing results can be merged. Otherwise,

there must be an error in the testing results. Addition operations

(53)

Mobile Testing-as-a-Service (Mobile TaaS or MTaaS)