Perfect Your Mobile App with Load Testing and Test Automation

(1)

June 2015

Wipro & Experitest Co-webinar:

(2)

Guy Arieli

CTO

Experitest

Sudheer Mohan

Director - Mobility

Certification &

Automation

Wipro Technologies

Speakers

June 2015

(3)

Agenda

1

2

3

4 Non Functional Testing for Mobile

Demo- See Test Integration with LoadRunner

Case Studies- Major Bank and US Retailer

Challenges in Mobile Load Testing

6

5 Mobility Automation

(4)

Non Functional Testing

Performance issues

(Slow , freeze, crash)

Heavy mobile resource utilization

Uses heavy network bandwidth

Accessibility compliance

Low application ratings

Unsatisfied customers &

Potential loss of Business

E

nterprise Challenges

Platforms &

platform versions

Device

Manufacturers

Real world

conditions

Service

Providers

(5)

Wipro Non-Functional Test Solution

Mobile Apps

Mobile

Platform

Mobile network



App profiling



Interruptions



App response time



Exploratory testing



Accessibility testing



Network condition



Handover situations



Data transfer



Sensitive data validation



Simulate mobile user load*



Load testing*



Stress testing*



Security testing



Load testing *



Stress testing *



Security testing

Enterprise

Services

Focused non-functional testing

* Covered as part of SeeTest load testing solutions

3rd Party Integration  Social media  Crash analytic tools  Analytic tools

(6)

Mobile generates

more load

(7)

Mobile users use your application

more extensively

Mobile generates more load

The number of mobile users

already exceeds the number of

desktop users.

(8)

The market shifts from

web to mobile

Mobile generates

more load

(9)

Thin-Thick Client

Mobile

applications are

replacing Web

Application

“The mobile

enterprise business

application market is

projected to double

by 2018”

Gartner

(10)

Mobile Apps are

relatively more complex

The Market shifts from

web to Mobile

Mobile generate

more load

(11)

Mobile

Web

JSon over http, JSon over TCP,

XML-RPC, Soap, REST, RMI …

Single Protocol- HTTP

Protocol

Json, XML, Free Text …

HTML

Data

Both synchronized and

non-synchronized

Synchronized –

request / response

Communication

Multiple threads

Single threaded app

Concurrency

Unlimited methods

Unified security –

browser based

Security

multiple requests

Single request / single

response

-Submit / next page

Transaction

Thin-Thick Client

(12)

Mobile load creates a

larger effect on your

servers

Mobile App is relatively

more complex

The Market shifts

from web to Mobile

Mobile generate

more load

(13)

Mobile load creates a bigger effect on your servers

One of the main

causes of load on

your server is the

number of

concurrent sessions

Mobile sessions take

longer to be

processed due to

mobile network

conditions. It results

in an overload of

your servers



The same number of

users from mobile

compared to desktop

can result in over

20 %

more load on

your servers.

(14)

Traditional Load

Testing is Synthetic

Mobile load create bigger

effect on your servers

Mobile App is relatively

more complex

The Market shifts

from web to Mobile

Mobile generate

more load

(15)

Traditional Load Testing is Synthetic

In traditional load testing the application under test is being removed and a synthetic network traffic is being generated

Typical requests can have dozen or even hundreds of fields. One of the big challenges of a load engineer is to

understand the nature of all of those fields.

How does your application traffic

look like?

Your application open TCP connection to

the server (SYN-> SYN,ACK<- ACK->)

Your application sends a request.

The server sends a response.

Your application is closing the connection.

What would you find in your

application request?

Key value property data structure includes

information like:

Request information

Request parameters

Session ID

(16)

syntactic load is Risky

Traditional Load

Testing is Synthetic

Mobile load create bigger

effect on your servers

Mobile App is relatively

more complex

The Market shifts from

web to Mobile

Mobile generate

more load

(17)

Syntactic Load is Risky

2 type

of correlations:

Why

do I need to

modify the

request

fields?

User parameters to request fields

Server response to user request

To simulate different user inputs i.e different user name / password or different UI

form population

To support internal server protocol, a very common server protocol scenario is that after

the login request, the server will return a session ID. The client needs to use this session ID

in any request that HAs been performed.

(18)

Syntactic Load is Risky

It will create no effect on the server

The server will ignore this field

The server will ignore the entire request

The request will cause a server internal error

With Synthetic load you don't really know what will happen, it is a trial and error process.

Main

Risk

What will

happen if I will

modify the

request field?

(19)

Hard to Validate a

Syntactic Load

Syntactic

Load is Risky

Traditional Load

Testing is Synthetic

Mobile load create bigger

effect on your servers

Mobile App is relatively

more complex

The Market shifts

from web to Mobile

Mobile generate

more load

(20)

Validate Syntactic Load

Performance teams need to adopt their methodologies to validate that the generated load is valid.

A few approaches can be:

Database / UI validation to verify that the transaction generated has the expected

effect

Looking at the server log files to find transaction completion log messages.

Monitor the server vitals to verify the load effect on the servers CPU, memory,

number of threads…

(21)

Hard to validate

syntactic load

syntactic

load is Risky

Traditional Load

Testing is Synthetic

Mobile load create bigger

effect on your servers

Mobile App is relatively

more complex

The Market shifts

from web to Mobile

Mobile generate

more load

(22)

As you generate the synthetic load, you would like to generate few interactions with the servers based on real

devices running your real application.

Monitoring your load using a real device transaction

Provide real

transaction

response time.

Provide validation

to the syntactic

load effect.

Benefits

Provide response

time reference

(when executed

without the

syntactic load)

(23)

End-to-End Product Suite -

Manual, Automation, Monitoring, Load, Network Virtualization, Manager

• Productivity tool for Manual testers

• Report with screenshots of each step

• Used in conjunction with SeeTestCloud

• Allow remote debugging

• Device Lab located onsite (in the customer VPN)

• Devices can be reserved and accessed by any remote user

• Used in conjunction with SeeTestAutomation/Manual for testing of remote devices

SeeTestManual

SeeTestCloud

• Automation tool

• Same test runs on different devices

• Plugs into all testing & CI environments:

SeeTestAutomation

SeeTestNetworkVirtualization

• Device performance under varying network conditions

• Simulating mobile network bandwidth latency and reception while on the move

SeeTestLoad

• Real device generated mobile load

• Full integration to LoadRunner, Jmeter, etc.

Mobile Add-On for UFT

• Full & complete integration into HP UFT:

• Record a test, edit it, run it, and view results

• Integration of object repository and more…

SeeTestManager

Free

Tool!

N

L

G

U

Mobile Add-On for LoadRunner

• Full integration to LoadRunner through

SeeTestAutomation

(24)

(25)

(26)

More releases in a year

Parallel Tracks- Phone/ Tablet/ Web

Less time to test

Numerous Devices/ Platforms/ Versions

Not just UI Testing- Services/ Backend

End to End Testing

W

hy is automation a must?

(27)

1. Early Automation



Support Agile & BDD projects



Faster Testing by in- sprint automation



Optimize testing cost by expanding

automation coverage

2. Platform Agnostic



Multi platform and device support



Reusable libraries



Jump start automated testing in new

projects



Reduce testing costs

3. End to End Automation



Services & Back end validation



Test in various environments by

pointing to different systems

behind the scenes



High quality apps

4. Unattended Testing

Parallel Execution



DevOps adaptation



Fully Automated



Integration with Test Management

& CI solutions



Faster test execution



Shorter test cycles & more release support

Wipro Automation

Framework

Wipro Automation Solutions Overview

(28)

5 Early Automation Approach

6

4

2

 Application Flow

 Objects in each screen

 Automatable scenarios

Analyze

Wireframes

 Develop Framework

 Build reusable functions/ pseudo scripts

 Object repository mapping table ( Key->Value pair)

Develop

Automation Scripts

 Map actual Objects to Object repository

Execution Ready

Dry Run

 Ensure modified scripts are running fine

Re Engineer

 Do modifications to the scripts to rectify the issues found in dry run and make it stable

Execution

 Start test case execution on test devices



Dev-ops or CI/CD



In-sprint automation



Accelerated Testing

(29)

Automated

Build

Checks

in Code

Automated

Deployment

Automated

Reporting

Automated

Testing

Un-Attended Testing



DevOps support



Quick testing

(30)

Demo



Build on Java, leveraging SeeTest



Supports keyword driven and data driven approach



Supports parallel execution for faster testing



Easy to configure and use even by a manual tester



Supports early automation approach



Supports integration with 3

rd

_{party test management}

tools



Supports un-attended testing

Framework Highlights

• Configure the environments via environment sheet

• Write the test cases using Excel based template

using keywords

• Demonstrate un-attended test execution using

Jenkins by automatic deployment of build

• Demonstrate parallel execution using SeeTest on

one Android and iOS device

• Integration with HP ALM – Updating results

• Demonstrate standalone reports

(31)

Case in Point: Complete Mobile Automation- Leading Bank in US

 Testing of Bank’s mobile application on iOS and Android devices

 A flexible and highly reusable

automation framework which supports unattended testing and integration with existing test management tool (HP ALM)

 Backend mainframe validation for transactions from mobile devices

 Reduce testing cycles and improved ROI for mobile automation

 Up to 40%* productivity gains with automation in comparison with manual testing

 Continuous integration using Jenkins and

enhanced reporting capability by automated results being updated in HP ALM

 Reduce testing cycles using early automation and parallel execution

 Shift-left automation script development by improving ROI for mobile automation

 Automated backend validation for each transaction

 Easily extendable to other OS platforms if needed

 Leveraged and customized Wipro mobile automation framework which supports:

 Platform agonistic scripting

 Data driven

 Highly reusable

 Parallel automation execution

 Leveraged SeeTest partnership to build a Mainframe validation component and incorporated in to Wipro framework

 Integrated Wipro automation framework with Jenkins and HP ALM for un attended testing with SeeTest as the automation tool

 Implemented in-sprint automation using Wipro automation framework

 Implemented Parallel automation test execution using SeeTest for faster testing cycles

Problem Statement

Wipro Solution

Benefits

!

(32)

Case in Point: Automation Productivity Enhancement- US Retailer

 Testing on iOS and Android devices

 High automation script reusability across customer mobile applications

 Build an automation framework which is easy to use and allows non-technical testers to do automation

 Faster testing cycles and improved ROI for mobile automation

 Enhanced automated test reports based on customer requirement

 Reduce overall testing cost by reducing as much as commercial tools

 Flexibility to run automation across various test environments

 Jump start automation by enabling non technical testers to do automation using a simple excel spreadsheet

 Improved automation development productivity using excel based framework

 Flexible and highly reusable automation framework which is completely configurable via excel sheets

 Shift-left automation script development by improving ROI for mobile automation

 Developed a Java based automation framework which allows writing of test cases using Excel sheets which enables even non-technical testers to do automation

 Core technical team writes new keyword functions

 Framework supports parallel execution by configuring the devices in the configuration file

 Custom made reports for holistic view of test execution across all devices in a single report

 Integrated with Jenkins for Continuous Integration support

 Implemented Parallel automation test execution using SeeTest for faster testing cycles

Problem Statement

Wipro Solution

Benefits

(33)

Case in Point: Enhancing Customer Experience for a Leading US

Retailer

 Reduce production defects reported by end users ( Performance issues, app freezing , no-product found)

 Improve app rating of existing mobile applications

 Ensure seamless customer satisfaction

 Ensure latest/upcoming features & technologies like iBeacon, barcode scanning , iwatch, etc.