Insights and Best Practices for Software Product Developers
Authors: Khimanand Upreti, Rajesh Popli
SIX TIPS TO IMPROVE MOBILE TESTING
FOR YOUR PRODUCTS
The Challenges With Mobile Testing Are Daunting…
Challenge
#1
It’s Critical to “Get Mobile Right”
Page 3
Challenge
#2
Mobile Adds Vast Complexity to Testing Process”
Page 4
Challenge
#3
Testing For Real World Conditions Becomes Expensive & Time Consuming for Mobile
Page 5
Challenge
#4
Mobile Testing Tools Still in Early Stage of Development
3 |
Challenge #1:
It’s Critical to “Get Mobile Right”
- Mobile apps have exploded with significance, now found in almost all enterprise solutions, and fundamental to the modern consumer’s lifestyle.
- User expectations for mobile apps are typically higher than for desktop apps. Mobile apps are closest to the user’s life and always available, hence coming under close scrutiny.
- Tolerance for poor performance with mobile products is low, and a bad customer experience is extremely damaging. Customers tend to leave brands with poor mobile performance.
- Mobile is the best way to reach the largest number of prospective users. But it’s imperative that the user experience with mobile be consistently delightful.
- Facebook now has all its developers create apps in mobile FIRST, before subsequent versions for the desktop. (5/28/14 video, softwaretestingmagazine.com)
The testing function emerges as a crucial step in the process of delivering mobile products to market. This is the key stage
for catching and correcting poor performance, before the product is released to unforgiving users.
*
Gartner projection, www.eurostarconferences.com, 10/30/2013.
**
ABI Research projection,
www.ministryoftesting.com
, 1/14/14.
And growing…
The stakes are high with mobile product development. Testing is more critical than before.
$27 billion mobile app market, 2013** 102 billion mobile apps downloaded in 2013*
The most important drivers of this added, unique complexity are the wide varieties of…
- target devices (with different operating systems, handset manufacturers, form factors, parallel applications running, etc.)
- network infrastructure conditions (including server load and geography issues, network type and quality, network load and geography, etc.)
- usability considerations (user-interface design differences, touchscreens, etc.)
- “Mobile apps may need to run on 18,000 different device platforms, plus variations when it comes to carrier networks” (Software Development Times, 9/5/13).
- Not to be forgotten are the various older versions of operating systems that must be tested for backwards compatibility.
- With time, market consolidation will ease the enormous array of conditions requiring testing. But even with fewer options among target devices, the task faced by mobile testing will still be vast.
The number of possible inter-dependencies to test with mobile can be overwhelming.
Challenge #2:
Mobile Adds Vast Complexity to Testing Process
Beyond standard testing issues, mobile apps introduce countless permutations and
interdependent conditions to confound the testing process.
5 |
End-to-end testing helps validate the actual sequence of events we expect to unfold with a mobile application. This calls for recreating “real world” conditions as they affect servers, networks, and devices , and finally, assessing the performance of the mobile application on each of these components.
How does the application respond to different locations and connectivity fluctuations? Dropped connections? Real bandwidth speeds that should be tested include EDGE, HSDPA, GPRS, LTE, UMTS, and HSPA+.
How does a mobile app respond to interruptions, like push notifications, text messages, and incoming calls?
Other conditions that might be included in mobile testing plans: - 3G and 4G networks;
- Network fluctuations while switching to and from Wi-Fi; - Varying signal strength;
- App usage offline; - Low battery conditions; - Sudden memory impacts; - Chipset-specific constraints; - Data encryption;
- Buffering issues;
- Usage of low-level resources.
Simulating real world conditions requires controlling a wide variety of equipment, located around the world. Testing these
conditions is time intensive. Both activities are costly.
Challenge #3:
Testing For Real World Conditions Becomes
Expensive & Time Consuming for Mobile
Performance tests should simulate traffic from mobile apps on real mobile devices, from
different parts of the world, under many conditions.
Mobile test design needs to make the most of what’s available in tool selection, especially given such diverse conditions that require testing.
With so many possible permutations to explore, automated testing offers greater efficiency, and is generally preferred. But in many cases manual testing is still required.
Selecting mobile testing tools is
difficult, full of trade-offs. Few ideal
testing tools are available to meet the
unique challenges of mobile.
Challenge #4:
Mobile Testing Tools Still in Early Stage of
Development
Compared to traditional testing, the tools for mobile testing are limited and complex, adding to
the challenge.
Before selecting tools for a test plan, Nagarro recommends starting with a proof-of-concept on critical platforms, and checking how prospective tools integrate with test management environments and application lifecycle management.
One starting point is to identify reusable scenarios across platforms and group them accordingly. This facilitates the creation of an appropriate, time-saving framework.
The choice of which automated tool can make a big difference in terms of coming in on time and on budget with well-tested mobile apps .
Developers can choose between open source and proprietary automated tools for mobile testing, as well as a third option: cloud-based tools that provide both manual and automated testing on real devices within their cloud
7 |
Tip
#1
Choose Mobile Testing Targets Strategically,
Rather Than Attempting Complete
Coverage
Page 8
Tips for Mobile Testing Should Help With Best Practices…
Tip
#2
Choose Tool & Test Plans that Support
Combinations …
Page 9
Tip
#3
Pay Attention to Usability Testing for Mobile
(MUX)
Page 10
Tip
#4
Include Security Testing in Mobile Test Plans
Page 11
Tip
#5
Closely Monitor Customer Statistics, for
Advantage …
Page 12
Tip
#6
Adopt New Mindset: Testing Mobile Saves Costs
and Brands
With such broad and diverse testing needs, no mobile app maker can possibly test everything, so don’t try. Instead, prioritize your efforts for those targets offering the greatest likely return.
- It’s important to test the mobile app on all popular components if your application supports a transaction, or supports high traffic, or serves as a shopping portal.
- But if the app suffers from a bug in a non-critical area, manage your resources wisely and moderate your response .
- End-to-end testing will still be an important part of your testing plan, but should be applied to the most strategic test cases first, then, as resources allow.
- As usage evolves, it will be helpful to look at site analytics, to help determine which aspects of usage are a priority, and which are not.
A pragmatic approach is essential to steering mobile testing activities down a feasible and
strategic path, to optimize the available resources.
Tip #1:
Choose Mobile Testing Targets Strategically, Rather
Than Attempting Complete Coverage
Mobile app makers need a strategy that offers the best possible solution for comprehensive
testing, instead of complete coverage.
- Select platforms with the largest market share, devices with the highest penetration, & performance attributes with the highest “trending”. (Web server logs can be helpful sources of market statistics.)
- For example, out of 18,000 devices that might be tested, you should narrow your list to 8 or 10 devices, to cover the majority of the market.
9 | Schedule mixed staging:
- Start with the emulators, to reveal bugs and expose functional issues. - Next, proceed to testing with real devices;
- Finish with the remotely accessed devices, to test geographic influences.
In similar fashion, tool selection should reflect an eye for mixed use. Hybrid tools offer you the flexibility of choosing which type of test to pursue at a given time, based on the nature of the mobile application and your project needs.
For example, there are hybrid tools to deal with object recognition from two different approaches (jail-breaking vs. compile time). Both approaches can be valuable when testing Apple’s closed iOS.
There are also tools that allow cloud-based access to remote devices for both manual and automated testing.
Maintaining flexible tools and processes is a priority for mobile testing, given the enormous range
of needs.
Tip #2:
Choose
Tool & Test Plans that Support Hybrid
Combinations
To cover myriad conditions, mobile product makers should lean towards tools and test plans
that support mixed needs, approached in stages.
The optimal testing framework should contain a combination of real devices, emulators, and remote devices accessed through a cloud service. This will make the most of your resources, and prepare for strategic coverage in your test plan.
10 | It’s often said that if a customer can’t figure out how to use a mobile app within 30 seconds, he or she is likely to uninstall, and move on to another competitor’s product. MUX matters
Nothing could be more frustrating to a mobile app maker than to see its product languish with low customer adoption rates, while a competing product succeeds, offering fewer functions yet commanding a higher price. Usability is often the key difference.
So, while all forms of testing will be altered by mobile challenges, MUX testing is particularly important to “get right”.. The top challenges for MUX include these (unique from traditional apps):
- Keypads and screens are smaller;
- Rapidly shifting user environments require mobile apps to respond (such as, adjusting screen brightness, etc.)
- Personalization is higher with mobile apps (affecting data usage and settings, for starters). The key areas that MUX testing must scrutinize include:
- Application effectiveness; - User customization options; - Application aesthetics
On a related note, don’t forget to prioritize testing for NFRs with large impact on the user experience (such as memory leakage, battery cons-umption, data caching, CPU use.)
While MUX standards have yet to mature, look to ISO 25010 as a starting point for quality benchmarks.
Ensure that your MUX testing is
well executed, to keep your
mobile product relevant to the
market. MUX matters.
Tip #3:
Pay Attention to Usability Testing for Mobile (MUX)
The mobile user experience (MUX) is fundamental to the ultimate market success of any
mobile application.
11 | 2) Threat Modelling – Identify threats, rate all risks, and develop countermeasures.
- One common threat is unauthorized reading of user data on the network, while communicating with the server.
- Counter measures often involve implementing SSL (Secure Sockets Layer) or TLS (Transport Layer Security).
- 3) Vulnerability Analysis & Assessment – Identify vulnerabilities, then test with static, dynamic, and forensic methods.
Lax security on a mobile solution not only diminishes its value, but also contributes to potential
liability. Make it a priority to test for mobile security.
* Stahl and Stroeher, Security Testing Guidelines for Mobile, 2013.
Tip #4:
Include Security Testing in Mobile Test Plans
Security challenges must be addressed, including authorized access, cached data, and
encrypted data transfer.
Mobile security testing* includes:
1) Intelligence Gathering - Evaluate the company’s business case with the solution, its relevance to company stakeholders, and lastly, the technical architecture of the solution.
- Architectural evaluation should include identifying backend services (such as the application server, databases, firewall), as well as analyzing the runtime environment.
The ubiquitous nature of mobile solutions introduces a greater security threat than found with traditional desktop applications. Security measures must be considered in development, and assessed during testing, before mobile solutions are launched.
Given the vast range of possible tests to conduct, it can be difficult to narrow down the options to a feasible selection. Ultimately, your choices must prioritize testing those areas with the greatest market significance for your application.
As a result, keeping on top of customer statistics is extremely important for effective testing plans. Market conditions change quickly within the mobile space, given the explosion of new products and their numerous interdependencies. The rapidly shifting tides of operating systems, devices, and network.
Conditions creates a competitive advantage for mobile testers who are the most adept at following customer trends and forecasting their future direction accurately
Mobile testing leaders must understand nuances between different vertical markets for their products, the number and location of consumer vs. enterprise users, and even the impact and size of different distribution channels.
- For instance, recent data suggests that iOS is the preferred operating system being tested (by mobile companies in North America and Europe) for retail and airline verticals (roughly 70%), whereas Android is the clear favorite (80%) for testing in the mobile advertising segment (eWeek 3/13/14, Darryl K. Taft).
Competitive advantage falls to those mobile testers who are the most adept at following customer behavior & trends, and
modifying their testing plans accordingly.
Tip #5:
Closely Monitor Customer Statistics, for Ongoing
Mobile Testing Advantages
Mobile test plans are built to optimize app performance for market leading customer segments,
which can be a moving target.
13 | Although mobile testing clearly involves more complication, time, and expense than desktop applications, the corporate payback for doing a good job with mobile testing is significant.
Catching bugs and problems before the product is released ensures the mobile application will more likely delight its users, build revenue streams and strengthen the corporate image.
Failing to invest properly in the mobile testing process threatens to harm the company deeply, with damage to the brand as well as revenue losses.
In the case of mobile apps for the enterprise, the cost of poor mobile testing includes diminished employee productivity.
In all cases, failure to “test mobile well” implies additional expense and demoralization, as the product team scrambles to make post-launch repairs.
The mandate to “get mobile right” supports renewed investment in the testing process for mobile.
Tip #6:
Adopt New Mindset:
Testing Mobile Saves Costs and Brands
Build adequate time and funding into the mobile testing process, for big picture benefits to the
company
Conclusion: Support Mobile Testing Expertise
The unique challenges of mobile testing place a high demand on its leaders to prioritize complex issues adeptly, and remain deeply knowledgeable about customer needs, as the basis for making those prioritized testing decisions.
Also, mobile testing leaders continue to need savvy technology insights, to lay the foundation for clever, creative innovations in their testing plans.
Mobile product companies which haven’t yet achieved “critical mass” with these skills in their testing organization face the possibility of either missing their market opportunity, or failing to catch key performance problems with their mobile products, both of which could have negative corporate impact.
Mobile product companies are encouraged to move quickly with support for their mobile testing process.; the alternatives are extremely costly.
Outside assistance is available, from strategic to tactical levels, to help bring mobile testing teams to full competitive
advantage.
In order to thrive, mobile testing teams will need strategic, well-informed, and technically creative
direction.
15 |
Case Study:
Testing for Pursers’ Mobile Solution for a
Leading European Airline
Learn how Nagarro performed 7
types of mobile testing for this
solution for airline pursers,
including cross-platform and
multilingual support
Mobile access to real-time data for airline pursers
improves customer service
One of the largest airlines in Europe is currently using a mobile solution for pursers, as a result of its partnership with Nagarro. The solution is cross-platform and multi-lingual. It improves the quality and efficiency of customer service for the airline, providing pursers with real-time data (combining customer and operational information).
Major functionalities include: - Access to purser roster data;
- Access to operational & customer data (seat map, position list, passenger information); - Reporting module to update data and generate real time reports;
- Document library to view, search, and synchronize documents; - Support for iPad and Android tablets, with multilingual capability.
Nagarro combines innovative technical expertise with a thorough understanding of its
clients’ business, for top results.
A cross-platform mobile solution for airline pursers, developed and tested by Nagarro,
improves customer service with real-time data.
The types of tests conducted by Nagarro on this mobile solution include: - Functional - Compatability; - Localization; - Security - Usability; - Performance; - Integration;
17 |
About Nagarro
Nagarro specializes in building complex, business-critical software and IT projects for leading companies across multiple industries. Headquartered in Silicon Valley and a member of the Allgeier Group in Germany, Nagarro is recognized worldwide for its technical expertise, agile deployment, and bedrock commitment to customer service.
Testing for software quality is one area of particular competence; Nagarro runs dedicated testing teams for leading product companies and enterprises globally. Clients benefit from Nagarro’s experience with test strategy planning, test automation, UAT support, and testing for mobile, cloud, regression, functional, and non-functional aspects of software products. Nagarro covers the spectrum of testing needs, consistently delivering top results for demanding corporate customers with the most exacting standards. With every engagement, innovation and strategic excellence remain hallmarks of Nagarro’s involvement.
Nagarro has also been named to the Global Outsourcing 100® List by the IAOP® (International Association of Outsourcing Professionals®), earning the highest possible scores in three categories – Customer References, Employee Management and Executive Leadership – and winning recognition as one of 25 “Rising Stars”.
Get the expertise and resources you need to stay ahead of the competition.
To manage risk more effectively, industry leaders partner with Nagarro to ensure greater
consistency in software quality.
USA - 3 Monterrey Jaipur New Delhi
Timisoara London Stockholm Munich Frankfurt Zurich Vienna [email protected] Twitter: @nagarro Facebook: Nagarro Inc.. LinkedIn: Nagarro.
Locations Around the World:
USA Offices:
San Jose, CA New York, NY Atlanta, GA
226 Airport Pkwy
5 Penn Plaza
400 Galleria Pkwy
Suite 390
23
rdFloor
Suite 1500
San Jose, CA 95110
New York, NY 10001
Atlanta, GA 30339
(408) 436-6170
(212) 799-2899
(678) 401-3131
www.nagarro.com
