Mobile Software Testing
– Facing the Future
Challenges
Ian Smith & Robert Campbell,
Sopra Group Ltd.,
UK
Europe’s Premier Software Testing Event
World Forum Convention Centre, The Hague, Netherlands
Mobile Software Testing
Facing The Future Challenges
Ian F. SmithPrincipal Consultant ismith@uk.sopragroup.com Sopra Group Ltd. Tel: +44(0)131 332 3311
Robert Campbell
Senior Developer rcampbell@uk.sopragroup.com Sopra Group Ltd. Tel: +44(0)1698 574 200
Overview
Introductions
Mobile Systems
Trends
Challenges/Problems
Approaches
Testing Mobile Systems
Example/Case Study
Introductions
Ian F. Smith
Principal Consultant – Testing Division
Over 28 years experience in software and system development and testing
Strong background in telecoms and real-time/embedded systems
Robert Campbell
Senior Developer – Testing Division
28 years experience in embedded development, manufacturing and test
Focussed on testing telecoms monitoring equipment for the last 6 years
Sopra Group
Sopra Group, a major European IT Services Company
Total of 11,000 staff and group 2007 Revenue of 1 Billion €
Offices throughout UK and Europe + and a fully integrated offshore Development and Testing Centre in Noida, Delhi
Services:
Application Outsourcing, Systems Integration, Testing + Specialist services
(Consulting, Information Security, Business Intelligence)
Key markets:
Finance, government, retail / manufacturing, energy & utilities, health,
Mobile Systems
What’s a typical mobile system?
Various types of mobile system User operates mobile devices
Supports access/operation in various locations and can often connect by various methods
Key Characteristics of Mobile Systems
Elements:
Mobile Devices
Wireless/Wired connection Server + Network Infrastructure
Mobile applications
Can run on either device or server, depending on architecture and device capability More internet based apps/services are being now provided for mobile use
Increasingly, core business systems are providing mobile application interfaces
Mobile Handset Mobile Application Server Mobile Network Mobile Device ISP User User Corporate Application Server Content/Data Server Internet
Mobile Systems - Examples
Navigation/Route Planning
Mobile Application Server: Navigation/Route Planner
Handset software: Navigation Handset Client
Content: Route information
Access Methods: GSM/3G
Mobile Application Server: Email Server
Handset software: Email Client Software
Content: Email information
Access Methods: Desktop, Laptop, GSM, WiFi, …
Voip Service
Mobile Application Server: Voip Network System
Handset software: Handset Voip Features + Agent software
Content: Voip calls data + SMS
Trends
Some key stats
1.28 billion mobile handsets to be sold in 2008 [Source: Gartner]
32.2 million smartphones sold in Q1 2008 [Source: Gartner, May 2008]
Better support for data (networks and devices)
3G data services
WiFi
Increasing power and sophistication of mobile devices
More powerful processor and storage Software capabilities
Connectivity
Audio/Camera/Video Sensors/Accelerometers
GPS
Application trends
Web 2.0 applications Increasing expectations
Desktop applications on mobile devices
Device capabilities
e.g. LBS
1980s Future 2010+ 2000s 1990s LANs Connectivity Fixed Access 10k 100K 1M 10M 100M 1G 10G Mobile Access GSM 3G 1980s Future 2010+ 2000s 1990s Desktop Application Capability Laptop Mobile Devices Converged Devices
Trends (Cont.)
Handsets gaining laptop/PDA capabilities
Processing, storage and software
Mobile OSs
Established players …
Symbian, Windows Mobile, RIM BREW, Linux Mobile, …
New Players
Apple, OHA/Android, LiMo, …
Landscape is changing
Symbian Foundation Open source Possible consolidation??
Development Environments
Varied environments
Open source Proprietary
IDEs/SDKs
Improving (but still with limitations)
Challenges/Problems
Cellular network services can vary significantly
Data offerings
Performance
Mobile device platforms
Many different software models + SDKs/IDEs/Toolsets
Changing platform landscape (Symbian Foundation, Android, iPhone, …) Device specific firmware issues
Application complexity
Meeting expectations for current desktop based applications on mobile
Application/content access
Access Diversity
Ubiquitous access …
GSM/UMTS WiFi/WiMax Wired
Security
Trust models and signing
Approaches
Manage Complexity
Application and system architecture
Component partitioning
Divide and conquer
Multi-level testing (unit, component, integration, system levels)
Get value from unit testing
Layer UI to simplify testing and support automation
Maximise code reuse
Important for multi-platform developments
Different platform models can limit this
Consider signing testing or manufacturer specific testing
Approaches (Cont.)
Using emulation
Doesn’t address all testing needs (telephony, performance) Combine with GUI based test automation tools
Device Testing
Confirm compatibility (device firmware issues)
Cover device specific features
Security Testing
Threat models change constantly, so review security testing regularly
Consider full operational lifecycle
Building Mobile Systems
Choosing platforms
Implications
System Architecture
DfT – Design for Test
Create test support features
Handset software
Engine (portable) + (thin) UI layer
Different software models
Application signing
Consider application efficiency
Power efficiency/management is a major factor for mobile systems
Development environments
SDKs are getting better, but building and testing will still be challenging
Design for offline use
Connectivity is improving – but devices won’t always be connected Also, some OSs don’t support application concurrency, so consider this
Testing Mobile Systems
Planning testing
Server and core system tests
Mobile device software testing
Using “devices” in system testing
Design testing to cover different handset platform needs
Aim for reuse, where practical
Server testing
Existing web or other test tools may be used as mobile device client capabilities approach those of desktop client systems.
Network & Infrastructure testing
Understand provider capabilities and limitations
Tools
Consider whether additional support tools can be procured/created to assist in testing (set-up, reporting, diagnosis, etc).
Emulators
Can support application level testing
Not good at emulating telephony or real-time operations
Automating Testing
Quick wins
Unit test can be well automated
Continuous Integration tools
Even some handset code!!
Automating device testing
This can be tough – different environments, user interfaces, etc.
Design UI to support automation (allow virtual user actions driven by tools)
Test Automation Tools
Performance Testing
Automation is essential to effective performance testing
Consider this in design-for-test activities.
Example
Webraska Navigation System
Handset based navigation system
Route information from server GPRS connection
Bluetooth GPS
Handset software testing project Evaluate handset software
Quality Performance Usability
Case Study
Mobile Call Generator Project
Develop proof of concept test call generation system Key project objectives:
Automated control of voice calls, messaging and data functions Scalable solution
Remote control RemCall System
Architecture Software
Handset Server(s)
Development Challenges Achievements Limitations GSM/UMTS Test Operator (1) Test Handset Test Operator (M) Test Handset Test Control Computer Mobile Handset Controller Quality Center Server
Network Under Test
Radio Access Network Billing System Core Mobile Network Test Control Computer
Case Study (Cont.)
Multi-level Testing
Unit, Component and System level test
Handset Agent Software
Piloted automation for unit testing
Introduced CI with CruiseControl
Handset Controller PC Software
Summary
Mobile systems offer many major opportunities but developing and testing
them pose significant challenges.
The pressure to supply mobile access to existing business applications and
systems is growing as device capabilities improve to support these.
New and existing businesses are bring innovative mobile systems to market.
The handset landscape is changing rapidly, with established players evolving
and new entrants joining. This change will continue.
