Mobile Testing Automation

(1)

Elevating conﬁdence

3200 Patrick Henry Drive, Suite 250 Santa Clara, CA 95054 408.727.1100 | [email protected] | www.infostretch.com

HARNESS THE MAGIC OF MOBILE TESTING AUTOMATION

An InfoStretch White Paper August 2014

(2)

TO SUMMARIZE

The need for automation is imminent. Why automation? Release cycles need to be shorter and faster to meet market demands. There has to be structured method in dealing with fragmentation, considering the proliferation of devices – phones, tablets, phablets, wearables. In this paper, we cover the parts of testing that can be automated, what tools can be used to do so and our approach Need a cost-benefit analysis of automation ROI? We’ve got you covered there, too. Read on.

TO SUMMARIZE

The need for automation is imminent. Why automation? Release cycles need to be shorter and faster to meet market demands. There has to be structured method in dealing with fragmentation, considering the proliferation of devices – phones, tablets, phablets, wearables. In this paper, we cover the parts of testing that can be automated, what tools can be used to do so and our approach Need a cost-benefit analysis of automation ROI? We’ve got you covered there, too. Read on.

(3)

3

Challenges with Manual Testing and

(4)

Software testing has evolved in a new dimension with the entry of mobile devices and is changing the status quo of the testing game. Mobile applications have to be made available quickly in the market, and tested effectively with shorter QA cycles. Companies today want to rapidly push applications in the marketplace. A traditional QA cycle which would take 3-6 weeks is now being condensed to just a few days. If the application does not meet the quality standards end users expect, more cycles will be spent on fixing those issues, negatively impacting the budget and the brand.

Mobile testing is challenging and what makes it so are the seemingly endless combinations of devices, OS’s, carriers, and networks. As it is an immature market, there is a lack of established best practices or industry standards around it. Mobile applications have become increasingly complex and to test the performance of the app on the server, device and network can be quite inundating. There are very few tools in the market to perform automation, and additionally, the automation strategy needs to be driven by specialists who can develop custom scripts for associated business need.

Traditional manual testing cannot feasibly cover all of the bases– especially when the app to market life cycle has to be short. There has to be some mechanism to test an app quickly and effectively, without wasting cycles. As a result, mobile automation has become a popular choice in overcoming the challenges in traditional mobile testing in order to maintain coverage.

(5)

5

Tenets of Automation

(6)

Regression tests can be easily automated to ensure that the basic functionality is maintained from the previous release. OS specific test cases can be defined and that case can be executed in multiple environments for that OS. Automation works best with features that are stable and are not undergoing changes. Not all features can be automated and require manual testing. So, there needs to be a balance between manual and automated testing.

QA managers must choose the right time for automation. Automating very early in the applications’ lifecycle might require too many changes in future and can significantly raise automation costs and effort. Only after manual testing has established with repeatable processes and stable test cases, should automation be considered.

What

When

There are many automation tools in the market which provide the automation engine through a public or a private cloud platform. Some very well-known automation tools are Perfecto Mobile, Device Anywhere/Keynote, ZapFix, Eggplant, Selenium, Robotium and Jamo. Here’s a list of automation tools that can be used for depending on the platform. Note: This is a basic list, and new tools get added frequently.

How

Automation is essential and it needs to be approached with a plan. Evaluating the “what”, “when” and “how” to automate becomes important.

• Write and Execute Automated Acceptance Tests • For Hybrid Apps, Android and iOS Native Apps

• Open Source Cross-Platform Functional Testing Tool for SOA and Web Service Testing

• Enterprise-Grade, Cloud-Based Mobile Application Quality Management Platform, with a Suite of Mobile Testing and Monitoring Products

• For Mobile Web, Native Apps, and Hybrid Apps

• Write Functional, System, and Acceptance Test Scenarios

• For Android Native and Hybrid Applications

• Functional and Regression Test Automation for Software Applications and Environments • Mobile Test Automation by Integrating Perfecto Mobile

• InfoStretch’s Test Automation Framework • For Mobile Web, Native Apps, and Hybrid Apps • Easy Integration with TestNG, Selenium, Autoit, and ReportNG

InfoStretch Selenium Test

Automation Framework

(7)

7

Methodology, Costs and Benefits

(8)

It is necessary to understand the overall scope and strategy in order to identify what to automate and justify efforts and ROI. Next is to identify the right tools and technologies for automation and define reusable test assets. Parameterizing select test cases by leveraging reusable assets to enable data driven test automation is essential. Execution of regression test cases and enhancements based upon the objectives and goals in addition to maintenance of reusable frameworks assets and automated tests with changes in application is critical.

(9)

9 Costs

ACQUISITION COSTS DEVICE COSTS LABOR COSTS TRAINING COSTS

Test automation tools can be expensive, and are usually employed in combination with manual testing. In the long term, it can become cost-effective, particularly when used repeatedly in regression testing.

Depending on which test automation product is selected, the user may need considerable training to become proficient at writing scripts for automated testing. The cost of mobile handsets

required for testing. With the Android fragmentation, there are many variations in the devices with manufacturers such HTC, Samsung, Motorola, Sony embedding Android into their devices.

Factor in the cost of trained personnel to script test cases or build manual tests. Be sure to include the costs of recruiting, hiring, paying, and retaining skilled QA engineers.

Prior to quantifying the benefits of test automation, it is important to understand the cost of automation. It is key to begin the conversation by discussing the different variables affecting the cost of automation, granted that the environment for automation has already been set up.

The cost of automating test processes is tangible however the net present value also includes many intangible factors. The best approach is to determine, with as much precision as possible, what the hard-dollar costs are and then compare them to the hard and soft dollar benefits of automating the test efforts.

Hard dollar costs to consider in an ROI calculation include:

Alternatively, companies may elect to hire professional services firms to handle the initial automated test creation. When weighing the potential rewards of automation, it is important to consider intangible benefits, such as higher morale and job satisfaction among testers, improved customer satisfaction and loyalty, and an enhanced reputation for reliable software among end users.

(10)

Among many other intangible benefits, automation can help to improve employee morale and productivity, because it can take on the repeated tests, allowing manual testers to focus on more challenging tasks. Discovering bugs early in the testing cycle is another intangible benefit that cannot easily be quantified. The later in the development cycle defects are found, the more expensive they are. A defect identified in a production environment can be greater than 100 times more expensive to fix than the same defect identified earlier in the design phase. Automation is the key to improving the speed, accuracy, and flexibility of the software testing process, enabling companies to find and fix more defects earlier.

Detailed Benefits of Automation:

Hours to create automated scripts Hours to maintain and enhance script

Reduction in Engineer's time requirement due to automation for test execution

Hours for 1st cycle testing

Hours for subsequent cycles testing when automation is available First year person-hours required

Subsequent year person-hours required First year cost of testing

Subsequent cost of testing per year

REDUCED COST OF TESTING DUE TO EFFORT REDUCTION

Elapsed testing time

Automation time reduction (1st year and subsequent years) Production rate

Total production recaptured Margin per unit

REDUCED TIME TO PRODUCTION

Average number of defects in product Defect-finding efficiency

Average cost of fixing defects prior to and after release Cost of fixing defects pre-production

Cost of fixing defects post-production Annual cost of fixing defects

IMPROVED PRODUCT QUALITY

Increased consumer confidence

Increased ability to support the change in application functionality Higher level of product quality and reduction in support costs

(11)

11

Approaches and Best Practices

(12)

(13)

13 Selecting the Right Test Cases

System/Peripheral Interaction

Interaction between multiple apps / OS-App / Multi-Domain

Location aware, peripheral, battery life, system alerts, etc.

TOOL AND RESOURCE CAPABILITY CAPABILITY/COMPLEXITY/PRIORITY

Less graphically intensive, more data and business scenarios

Medium complexity test cases: High → Low Smoke: High → Medium → Low

STABILITY/CHANGE

Best Practices for other GUI Automation apply UI components that will change less

Clear understood business processes

The Automation Architecture

MAKE PAYMENT TEST CASE

QTP/SELENIUM

BROWSER

OPEN BROWSER GO TO SIGN ON MAKE PAYMENT TEXT VERIFY CONFIRM PAYMENT SIGN OFF

DEVICE IMPLEMENTATION FOR SIGN ON (10+)

DEVICES

iOS ANDROID _{HTC ARIA} _{HTC DROID}ANDROID BB TORCH BB 8520 LG CU620 LG VX9000

COMPONENT LIBRARY

FRAMEWORK

ADDRESS

ATM SEARCH BROWSER OPEN ACCT VIEWSELECT SCHEDULE PAYMENT SYNC TEXT QUICK LINKS NICKNAME

BROWSER

(14)

Quantifying the benefits of test automation is a complex task. In organizations where decisions

are made purely on the expected ROI, managers must keep several criteria in mind while making

computations. In many cases, the intangible benefits of automation will supersede the tangible

benefits. Hence, ROI computations might not reveal the real value of automation. The best results

are obtained when costs and benefits are computed in terms of opportunity costs related to

test automation and manual testing. This will provide decision makers the visibility into some of

the intangible costs and benefits of automation. Test Automation has worked its magic for many

enterprises. Contact us to see how it can for you.

(15)

All registration marks are the property of their respective owners.

ABOUT INFOSTRETCH

InfoStretch is a leading provider of mobile and enterprise QA services and solutions. Our offerings range from enterprise QA, mobile application development, testing, and automation to certification and sustenance. We’ve been providing expert solutions and services to various large and strategic players in the mobile ecosystem including enterprises, carriers, app stores, VAS, and ISV’s to actualize their mobile and QA strategy very successfully.

We offer the most cost effective, secure, and scalable models to service our clients across various verticals such as banking, healthcare, education, and entertainment to name a few. Our innovations have allowed us to provide an SLA of 24 hours and the ability to scale and test over 2,000 apps a day.

CONTACT US

SANTA CLARA OFFICE InfoStretch Corporation 3200 Patrick Henry Drive Suite 250 Santa Clara, CA 95054 GENERAL INQUIRIES e: [email protected] p: (408) 727-1100 f: (408) 716-2461